# AI training crawler > An AI training crawler collects public pages as material that may be used to train a future model. It is not answering anyone's question and not building a search index — nothing about how you appear in search or in live answers changes when one visits, which is also why blocking it changes nothing you can measure this quarter. **Category:** AI crawlers **Also known as:** training bot, corpus crawler, LLM training crawler **Updated:** 2026-09-09 **Source:** https://truestat.io/glossary/ai-training-crawler --- ## How the traffic behaves Broad and slow. Training crawls want everything once rather than the same pages repeatedly, so they reach parts of a site search engines rarely bother with — old posts, deep archive pages, anything a link points at. Volume spreads over days or weeks instead of arriving as spikes. For most sites the reason anyone notices is bandwidth. `meta-externalagent` in particular has been reported by Cloudflare as second only to Googlebot across the web, and it brings no search traffic back. ## The decision, honestly Blocking a training crawler changes nothing you will see. Training feeds models that ship later, so there is no traffic effect to measure and no before-and-after to point at. The argument for allowing it is that being in the training data is part of how a model comes to mention you at all — an assistant that has never seen your product cannot recommend it. The argument against is that your work becomes part of a commercial model you have no stake in. There is no measurement that settles this, and anyone who tells you there is has not looked. It is a decision about your terms rather than about your numbers. ## Common Crawl is the multiplier `CCBot` deserves separate attention because its reach exceeds its request count. Common Crawl is a non-profit that archives the public web and publishes the result as a free dataset, and most model builders start there rather than crawling themselves. So allowing CCBot is an indirect allow for training sets belonging to operators who never visit your site, and blocking it is one of the few decisions here that reaches more than one company at once. It also removes you from a public research dataset, which some site owners count as a cost. ## What the operators actually promise Less than most people assume, and the differences between them are worth knowing before deciding one policy for all. OpenAI, Anthropic and Google document their training crawlers, publish address lists, honour robots.txt and give training its own token — which means a site can refuse training and keep search. That combination is the most a site owner can currently ask for, and only a handful of operators offer it. Meta publishes crawler documentation but no address list, so its traffic cannot be verified. ByteDance publishes neither and is widely reported ignoring robots.txt. The newer entrants — Cohere, DeepSeek, Zhipu, Alibaba, xAI — mostly publish nothing at all, which means their tokens are known from observed traffic rather than from any statement the company has made. None of them promises anything about what happens to content already collected. A block stops future crawls; it does not reach back into a dataset, and no operator offers a mechanism that does. ## What goes wrong - Blocking the wrong token. `GPTBot` is training; `OAI-SearchBot` is the index that decides whether you exist inside ChatGPT search. A rule meant for one that lands on the other costs visibility and prevents nothing. - Assuming a robots rule stops it. Bytespider is widely measured fetching disallowed paths; where that matters, the block has to happen at the network layer. - Expecting a traffic change. There is none to see, which is not evidence the block failed. ## Worth doing - Decide operator by operator rather than as one policy — the operators differ in what they publish, what they honour, and what they give back. - If bandwidth is the concern, measure it first: for most sites training crawls are a rounding error, and for a few they are the largest line. ## Questions ### Does blocking a training crawler remove my content from a model? No. It prevents future collection; it does nothing about what has already been collected, including copies held in datasets like Common Crawl's that were gathered before the block. ### Will blocking training crawlers hurt my AI visibility? Nobody can answer this with evidence. Training and live retrieval are separate systems with separate crawlers, so the mechanical answer is no — but whether a model recommends products it was never trained on is not something anyone outside the labs can measure. ## Related - https://truestat.io/glossary/ai-answers - https://truestat.io/glossary/robots-txt - https://truestat.io/glossary/openai-gptbot - https://truestat.io/glossary/commoncrawl-ccbot --- From the TrueStat glossary — https://truestat.io/glossary. Privacy-first web analytics that also shows you which AI assistants are reading your site.