robots.txt
Also known as robots exclusion protocol, robots file
5 min read · updated 2026-09-09
Definition
robots.txt is a plain text file at a site's root that tells crawlers which paths they may request. It is a request rather than an enforcement — well-behaved operators honour it, and nothing in the protocol makes anyone.
One rule per crawler, because the crawlers differ
The file matches on user-agent tokens, and the useful consequence is that an operator running several crawlers can be answered differently for each. OpenAI runs five; blocking `GPTBot` and allowing `OAI-SearchBot` is two lines and it is usually what somebody means when they say they want out of training.
A blanket `User-agent: *` rule is where this goes wrong. It catches the search crawlers that were bringing traffic along with the training crawlers somebody wanted to refuse.
Two tokens that never make a request
`Google-Extended` and `Applebot-Extended` appear in robots.txt and are not crawlers. They are permission tokens: they control whether content already crawled may be used for model training. Google's own documentation states that Google-Extended "doesn't have a separate HTTP request user agent string".
So neither will ever appear in a log, and a directory listing them as crawlers with traffic types is publishing a fiction. If you blocked Google-Extended and want to check it worked, no analytics product can tell you — nothing would have appeared either way. The crawler to look at instead is `GoogleOther`.
What it cannot do
It cannot enforce anything. A crawler that ignores the file simply requests the path, and Bytespider is widely measured doing exactly that. Where a block has to hold, it belongs at the firewall or CDN.
It also cannot hide anything. A disallowed URL is still a URL, and listing sensitive paths in a public file is a way of advertising them.
How it is actually parsed
The details that surprise people, all of which follow from the file being older and simpler than the uses now made of it.
**Most specific user-agent group wins, and only one group applies.** A crawler matching both `User-agent: *` and `User-agent: GPTBot` obeys the GPTBot block and ignores the wildcard entirely — including any `Disallow` lines the wildcard carried. A rule added to the wildcard block does not apply to any crawler that has its own group.
**Allow beats Disallow when it is more specific.** `Disallow: /docs/` with `Allow: /docs/public/` opens the subtree, which is how a site exposes part of a blocked section.
**It is per host and per protocol.** `https://example.com/robots.txt` does not govern `https://cdn.example.com/`, and a subdomain serving no robots.txt is open regardless of what the parent says.
**Crawl-delay is not part of the standard.** Google ignores it; some others honour it. If rate is the problem, the reliable control is at the server.
**It is cached.** A change can take a day or more to take effect, so a rule that appears to be ignored may simply not have been read yet.
What goes wrong
- Blocking with a wildcard when you meant one operator's training crawler.
- Expecting a rule to stop a crawler documented as ignoring rules.
- Blocking `Applebot` to opt out of Apple's training — that costs Siri and Spotlight placement and stops no training; the token you want is `Applebot-Extended`.
- Assuming the file was read. Crawlers cache it, sometimes for a long time.
Worth doing
- Write one block per operator token rather than one for everything.
- Check the fetch. A crawler requesting robots.txt is the evidence it is paying attention.
Questions
- Does robots.txt block AI training?
- It asks. Operators that honour it will stop; operators that do not will not, and nothing in the file distinguishes the two in advance. For training specifically, some operators offer a dedicated token — Google-Extended and Applebot-Extended — which controls use rather than access.
- Will blocking AI crawlers hurt my search rankings?
- Not if you block the right tokens. `Googlebot` is search and `GoogleOther` is research; the two are separate rules. Getting them the wrong way round is the expensive mistake.
- What happens if I have no robots.txt at all?
- Everything is allowed. An absent file is not a refusal — crawlers treat a 404 as permission to fetch anything they can reach. A 500, on the other hand, is treated by some crawlers as a temporary instruction to stay away entirely, which is why a broken robots.txt can be worse than a missing one.
Related
See which AI assistants are reading your site.
AI crawlers take your HTML and leave without running a line of JavaScript, so a browser tag records nothing. TrueStat reads them server-side, checks each address against what the operator publishes, and shows you which pages were fetched — including the ones that returned a 404.