# robots.txt > robots.txt is a plain text file at a site's root that tells crawlers which paths they may request. It is a request rather than an enforcement — well-behaved operators honour it, and nothing in the protocol makes anyone. **Category:** Verification **Also known as:** robots exclusion protocol, robots file **Updated:** 2026-09-09 **Source:** https://truestat.io/glossary/robots-txt --- ## One rule per crawler, because the crawlers differ The file matches on user-agent tokens, and the useful consequence is that an operator running several crawlers can be answered differently for each. OpenAI runs five; blocking `GPTBot` and allowing `OAI-SearchBot` is two lines and it is usually what somebody means when they say they want out of training. A blanket `User-agent: *` rule is where this goes wrong. It catches the search crawlers that were bringing traffic along with the training crawlers somebody wanted to refuse. ## Two tokens that never make a request `Google-Extended` and `Applebot-Extended` appear in robots.txt and are not crawlers. They are permission tokens: they control whether content already crawled may be used for model training. Google's own documentation states that Google-Extended "doesn't have a separate HTTP request user agent string". So neither will ever appear in a log, and a directory listing them as crawlers with traffic types is publishing a fiction. If you blocked Google-Extended and want to check it worked, no analytics product can tell you — nothing would have appeared either way. The crawler to look at instead is `GoogleOther`. ## What it cannot do It cannot enforce anything. A crawler that ignores the file simply requests the path, and Bytespider is widely measured doing exactly that. Where a block has to hold, it belongs at the firewall or CDN. It also cannot hide anything. A disallowed URL is still a URL, and listing sensitive paths in a public file is a way of advertising them. ## How it is actually parsed The details that surprise people, all of which follow from the file being older and simpler than the uses now made of it. **Most specific user-agent group wins, and only one group applies.** A crawler matching both `User-agent: *` and `User-agent: GPTBot` obeys the GPTBot block and ignores the wildcard entirely — including any `Disallow` lines the wildcard carried. A rule added to the wildcard block does not apply to any crawler that has its own group. **Allow beats Disallow when it is more specific.** `Disallow: /docs/` with `Allow: /docs/public/` opens the subtree, which is how a site exposes part of a blocked section. **It is per host and per protocol.** `https://example.com/robots.txt` does not govern `https://cdn.example.com/`, and a subdomain serving no robots.txt is open regardless of what the parent says. **Crawl-delay is not part of the standard.** Google ignores it; some others honour it. If rate is the problem, the reliable control is at the server. **It is cached.** A change can take a day or more to take effect, so a rule that appears to be ignored may simply not have been read yet. ## What goes wrong - Blocking with a wildcard when you meant one operator's training crawler. - Expecting a rule to stop a crawler documented as ignoring rules. - Blocking `Applebot` to opt out of Apple's training — that costs Siri and Spotlight placement and stops no training; the token you want is `Applebot-Extended`. - Assuming the file was read. Crawlers cache it, sometimes for a long time. ## Worth doing - Write one block per operator token rather than one for everything. - Check the fetch. A crawler requesting robots.txt is the evidence it is paying attention. ## Questions ### Does robots.txt block AI training? It asks. Operators that honour it will stop; operators that do not will not, and nothing in the file distinguishes the two in advance. For training specifically, some operators offer a dedicated token — Google-Extended and Applebot-Extended — which controls use rather than access. ### Will blocking AI crawlers hurt my search rankings? Not if you block the right tokens. `Googlebot` is search and `GoogleOther` is research; the two are separate rules. Getting them the wrong way round is the expensive mistake. ### What happens if I have no robots.txt at all? Everything is allowed. An absent file is not a refusal — crawlers treat a 404 as permission to fetch anything they can reach. A 500, on the other hand, is treated by some crawlers as a temporary instruction to stay away entirely, which is why a broken robots.txt can be worse than a missing one. ## Related - https://truestat.io/glossary/llms-txt - https://truestat.io/glossary/ai-training-crawler - https://truestat.io/glossary/user-agent --- From the TrueStat glossary — https://truestat.io/glossary. Privacy-first web analytics that also shows you which AI assistants are reading your site.