Neighbot, our web crawler

Who we are

Neighbo is a search engine for an independent and human-curated directory of places in our local area that we are building by hand. Right now the area we cover is just around Brunswick, Australia, but it will grow across Melbourne over time. You can read more on the our story page.

What Neighbot is

Neighbot is how we keep the Neighbo place accurate. We start by manually adding your website to our list of local places, then later the Neighbot visits the websites of those places to confirm that the website is still published, that it really belongs to the place, and that details like contact information are up to date.

We believe in being transparent about this. Neighbot identifies itself on every request and respects the wishes of the sites it visits.

The user-agent we send

Every request Neighbot makes carries this exact user-agent string. Use it to can recognise and target Neighbot in your logs or robots.txt rules:

Mozilla/5.0 (compatible; Neighbot/1.0; +https://neighbo.au/neighbot)

We may increment that version number in future.

What we crawl, and how often

Neighbot is not a general web crawler. It only visits the websites of places that are already in our directory, and only to verify our listings. Neighbot does not follow links out across the open web, and it skips images, video, document, and other binary files.

We try to keep Neighbot crawls to a minimum so it fetches a site only when one of these happen:

• One of our data sources suggests a change that needs to be verified against the official website, or
• It has been a while since Neighbot verified that the website is still reachable and relevant, or
• One of our staff manually requests a fresh look at a specific site.

When it does crawl, Neighbot reads only a small number of pages per site and paces itself to avoid putting load on your server. Most data downloaded in the crawl is automatically deleted 24-48 hours later, with only information that we intend to publish retained. In other words if you don't see particular information published on Neighbo then we are probably not retaining it^†, even if Neighbot downloaded it.

^† We do retain some information that we have chosen not to publish. That information, for obvious reasons, is not visible on the Neighbo site. It's not a lot that we hold, mostly small things like old phone numbers and email addresses which are no longer valid. Retaining these helps us avoid publishing outdated information because we have a record that they are no longer valid.

User-agent: Neighbot Disallow: /