CCBot
Also known as Common Crawl CCBot
4 min read · updated 2026-09-09
Definition
CCBot is Common Crawl's crawler for collecting training data. Builds the open web dataset most model builders start from. Common Crawl trains nothing itself, but its archive is a training input for most of the models on this page — so it belongs here.
For the reference card — CCBot in the crawler directory.
What CCBot is
CCBot is worth understanding even if you never see much of it, because it has more reach than its request count suggests. Common Crawl's archive is free to download, and most model builders start there rather than crawling the web themselves. Allowing CCBot is therefore an indirect allow for training sets belonging to operators who never visit your site at all; blocking it is one of the few decisions here that reaches more than one company.
Common Crawl is a non-profit that has been archiving the public web since 2008 and publishes the result as a free dataset. It trains nothing itself, which is exactly why it matters: its archive is the starting corpus for most of the models named elsewhere on this site.
On this site CCBot is counted as training, which is the row it appears in on your dashboard. The category matters because it decides which number moves when this crawler visits, and the four are reported separately for exactly that reason.
What its traffic looks like
Broad and slow: many pages, few repeats, spread over days or weeks rather than concentrated. Training crawls tend to want everything once, so the long tail of a site sees traffic it never gets from search — old posts, deep archive pages, anything reachable by a link. Bandwidth is the usual reason people notice these at all.
How it is verified
Checked as published ip ranges. The user agent has to carry the token, and then the address is compared against what Common Crawl publishes — refetched every six hours, so a newly announced range is picked up without a deploy.
A request that matches neither is recorded as unverified rather than as a forgery: a published list can be incomplete, and an accusation needs evidence. Only a reverse DNS record resolving to somebody else produces a spoofed verdict, which is proof rather than the absence of it.
If you block it
You leave the open crawl archive that most model builders start from — so it is an indirect block on many training sets at once, including ones from operators who never visit you themselves. It also removes you from a public research dataset, which some site owners consider a cost rather than a benefit.
Nothing you see today changes if you block this. Training crawls feed models that ship months later, so the decision is about whether your work becomes part of a commercial model, not about this quarter's traffic.
The honest counter-argument is that being in the training data is part of how a model comes to mention you at all. There is no measurement that settles this either way — anyone who tells you there is has not looked — so it is a decision about your terms rather than about your numbers.
One real request
CCBot/2.0 (https://commoncrawl.org/faq/)Questions
- Is CCBot in my logs really Common Crawl?
- Not necessarily, and that is why it is worth checking. Any script can send CCBot as its user agent. Common Crawl publishes the addresses its crawler uses, so the claim can be tested against them — a request from outside those ranges is recorded as unverified rather than accepted.
- How do I match CCBot in my own logs?
- Match the token as a substring of the user-agent header, case-insensitively. Common Crawl adds and changes the version and product text around it without notice, so a rule pinned to the whole header stops working the first time they bump a version number, while a rule matching CCBot keeps working.
- Does CCBot run JavaScript?
- No. It requests the HTML and leaves. That is why a browser-based analytics tag never records these visits: the tag is JavaScript, and nothing runs it. Seeing this crawler at all requires reading it server-side.
- Why is CCBot counted as Training?
- Because that is what the fetch is for. The four categories separate what a crawler came to do rather than who sent it, so a visit from Common Crawl moves this row and not the others. Where an operator publishes nothing to settle the question, we say so on the page rather than choosing quietly.
- Why does CCBot not show up in Google Analytics?
- Google Analytics runs in the browser. This crawler never opens a browser — it requests the HTML, reads it, and leaves, so the tracking script is never executed and no event is ever sent. Every browser-based analytics tool has the same blind spot, which is why crawler traffic has to be read from the server side to be seen at all.
- What happens if a CCBot visit cannot be verified?
- It is recorded as unverified, not as a forgery. A published address list can be incomplete or behind — an operator can start using a range before it appears in the file — so a miss is not evidence of anyone pretending. A visit is only marked as spoofed when the address's own reverse DNS record resolves to somebody else, which is proof rather than absence of proof.
Related
See whether CCBot is reaching your pages.
AI crawlers take your HTML and leave without running a line of JavaScript, so a browser tag records nothing. TrueStat reads them server-side, checks each address against what the operator publishes, and shows you which pages were fetched — including the ones that returned a 404.