ClaudeBot

Anthropic splits its crawling the same way OpenAI does — training, search and live answers each get a token — and publishes one combined address list covering all of them. The list is kept current; it was three weeks old when this page was last checked.

On this site ClaudeBot is counted as training, which is the row it appears in on your dashboard. The category matters because it decides which number moves when this crawler visits, and the four are reported separately for exactly that reason.

Broad and slow: many pages, few repeats, spread over days or weeks rather than concentrated. Training crawls tend to want everything once, so the long tail of a site sees traffic it never gets from search — old posts, deep archive pages, anything reachable by a link. Bandwidth is the usual reason people notice these at all.

Checked as published ip ranges. The user agent has to carry the token, and then the address is compared against what Anthropic publishes — refetched every six hours, so a newly announced range is picked up without a deploy.

A request that matches neither is recorded as unverified rather than as a forgery: a published list can be incomplete, and an accusation needs evidence. Only a reverse DNS record resolving to somebody else produces a spoofed verdict, which is proof rather than the absence of it.

Nothing today — it collects training material only. Claude will still fetch and cite your page when a person asks about it, because that is Claude-User's job, not this one's.

Nothing you see today changes if you block this. Training crawls feed models that ship months later, so the decision is about whether your work becomes part of a commercial model, not about this quarter's traffic.

The honest counter-argument is that being in the training data is part of how a model comes to mention you at all. There is no measurement that settles this either way — anyone who tells you there is has not looked — so it is a decision about your terms rather than about your numbers.

Anthropic sends 4 crawlers and gives each its own token, so they can be allowed and refused separately. The tokens differ by a few characters and what the fetch means does not:

Claude-User — ai assistant. Fetches a page while Claude is answering someone's question. Claude-Code — ai assistant. Anthropic's coding agent reading a page a developer pointed it at. Claude-SearchBot — search indexing. Discovers and refreshes content for Claude's search.

Confusing two of these is the most common mistake with this operator, and it is expensive in one direction: a rule meant for the training crawler that lands on the search crawler removes you from results without stopping any training.