Crawler traffic is not an audience
Assistant and training crawlers can account for a substantial share of requests to a content-heavy site. Counted as visits they make a dashboard meaningless. Counted separately they are the clearest available signal of which pages machines are actually reading.
Two things are true at once. Crawler requests are not an audience and must be kept out of any figure described as visits. And crawler requests are the most direct observation available of which of your pages machines are reading, which makes them far too useful to discard.
Why it distorts more than it used to
Conventional analytics runs in the browser, so most crawlers never appeared in it. Anything measured server-side — request counts, bandwidth, response times, and increasingly the server-side event collection that replaced browser tracking — sees all of it. A knowledge site can see a meaningful proportion of its requests coming from automated readers, and a dashboard that mixes them reports growth that no human produced.
How to separate them properly
Tag every request as human, crawler or unknown when it arrives, and store the tag. Filtering afterwards in a reporting tool is lossy and gets forgotten.
Major crawlers publish their address ranges or support reverse lookup. A user agent alone is trivially spoofed; for anything you will report on, verify.
Do not force ambiguous traffic into one bucket. An honest third category is better than a confident misclassification, and its size is itself worth watching.
Two lines on the same chart. They answer different questions and the sum answers neither.
Two lines on one chart. Never one line.
What the crawler line actually tells you
- Which pages machines consider worth re-reading, by fetch frequency. This is a reasonable stand-in for perceived importance.
- Whether a newly published page has been discovered at all, usually within days if your sitemap and linking are right.
- Whether fetches are succeeding. A pattern of errors or timeouts on your best pages is a visibility problem with a technical cause.
- Whether a robots change did what you intended, visible within about a week.
- Which crawlers are present at all, which is the only way to know whether a provider you care about is reading you.
What it does not tell you
Whether anything was used. A fetch is a read, not a citation, and a page fetched daily may never be quoted. Resist the temptation to build a score out of fetch counts; it will correlate with page count and nothing else.
Questions this settles
Cache and serve efficiently first. Blocking a retrieval crawler to save bandwidth removes you from answers about your own subject, which is an expensive way to save a small amount.
Usually not, because it runs in the browser. Server-side collection and log-based reporting are where the distortion appears.
Check the published address ranges or use reverse DNS where the provider supports it. User agent strings are trivially copied and frequently are.