Bot filtering
Also known as bot traffic filtering, spider filtering
4 min read · updated 2026-09-09
Definition
Bot filtering is separating automated requests from human ones in analytics. The important design decision is where it happens: filtering at ingest discards the request permanently, while filtering at read time keeps it and decides what to show — which is the difference between a number you can revise and one you cannot.
Ingest versus read time
Filtering at ingest means the request is classified as a bot and thrown away. It is cheaper, and it is permanent: when the classifier turns out to have been wrong, or when a crawler that was unknown last month gets a name this month, the history cannot be recovered.
Filtering at read time means everything is stored with a label and the label decides what a given view counts. It costs storage and a little query complexity, and in exchange every past decision stays revisable. A crawler recognised today can be counted correctly in last year's data.
For a product whose whole subject is AI crawler traffic, discarding bot requests at ingest would throw away the feature.
The categories worth keeping apart
"Bot" is too coarse to be useful. A training crawler, an assistant fetching an answer, a search indexer, an uptime monitor and a link-preview fetcher are five different things, and folding them together produces a number that means nothing.
The split that matters most in practice is preview bots. `facebookexternalhit`, `Twitterbot` and `Slackbot` fetch a page to draw a link card, and every share produces one — so a popular post can generate more preview fetches than a site has real AI visits. Counting them as AI inflates the figure visibly.
What a good bot report looks like
Four properties, and most analytics products have none of them because bot traffic is treated as a nuisance to be removed rather than a subject.
**Separate axes.** Human traffic and bot traffic on the same chart makes a training sweep look like a launch. They answer different questions and belong in different series.
**Named operators, not a lump.** "Bots: 4,312" tells a reader nothing they can act on. "OpenAI 900, Meta 2,100, unidentified 1,312" tells them where the bandwidth went and which of it was worth having.
**Confidence shown, not implied.** A verified count and an unverified count are different claims, and merging them means the number cannot survive the question of how you know.
**Reversible classification.** The list of known crawlers grows monthly. A product that labels at read time can apply today's knowledge to last year's traffic; one that filtered at ingest cannot, and its history is permanently as good as its classifier was on the day.
What goes wrong
- Filtering at ingest. It is the one decision here that cannot be undone.
- Counting link previews as AI traffic.
- Missing your own infrastructure. A favicon fetcher, an uptime monitor or a preview service you run yourself will show up as crawler traffic on your customers' sites.
- Assuming an unrecognised bot is a human. It is the safer default, because it understates the AI figure rather than inflating it, and for a number sold on honesty that is the correct direction to be wrong — but it is still a known error, and the size of it is unknowable by definition.
Worth doing
- Store the classification alongside the event rather than instead of it.
- Keep a separate axis for bot traffic in every chart. A crawl is not a visit and should not share a line with one.
Questions
- Does Google Analytics filter bots?
- It filters known bots from a list it maintains, and it never sees the crawlers discussed here at all — they do not run JavaScript, so no event is ever sent. The filtering question only arises for bots that execute scripts.
- Should bot traffic count toward my plan usage?
- That is a policy decision rather than a technical one, and it deserves to be stated plainly wherever the number is shown. Charging for traffic a customer did not ask for and cannot see is a way to lose them.
- Can I tell a headless browser from a real one?
- Sometimes, and never reliably. `HeadlessChrome` in the user agent is an admission rather than a detection, and anything determined to hide will remove it. Behavioural signals exist but they misclassify real people — screen readers, old devices, privacy browsers — which is a worse error than missing a bot.
See which AI assistants are reading your site.
AI crawlers take your HTML and leave without running a line of JavaScript, so a browser tag records nothing. TrueStat reads them server-side, checks each address against what the operator publishes, and shows you which pages were fetched — including the ones that returned a 404.