Claude-SearchBot

Anthropic splits its crawling the same way OpenAI does — training, search and live answers each get a token — and publishes one combined address list covering all of them. The list is kept current; it was three weeks old when this page was last checked.

On this site Claude-SearchBot is counted as indexing, which is the row it appears in on your dashboard. The category matters because it decides which number moves when this crawler visits, and the four are reported separately for exactly that reason.

A steady crawl over time rather than a burst: robots.txt first, then pages in waves, revisiting the ones that change often. Volume tracks how large and how frequently updated your site is. A sudden increase usually follows a sitemap change or a burst of new pages, and a sudden stop is worth investigating — it often means a robots rule or a server error rather than a decision by the operator.

Checked as published ip ranges. The user agent has to carry the token, and then the address is compared against what Anthropic publishes — refetched every six hours, so a newly announced range is picked up without a deploy.

A request that matches neither is recorded as unverified rather than as a forgery: a published list can be incomplete, and an accusation needs evidence. Only a reverse DNS record resolving to somebody else produces a spoofed verdict, which is proof rather than the absence of it.

You stop appearing in Claude's search results. Claude-User can still open your page if someone pastes the URL, but it will no longer find you on its own.

Treat this as a search-engine decision rather than an AI one, because that is what it is. The question is whether the people you want are looking through this engine, and your own analytics answer it better than any general advice can.

It is worth checking what you actually lose before writing the rule. Blocking an index crawler is one of the few changes here with an effect that compounds quietly: nothing breaks on the day, and traffic simply stops arriving over the following weeks.

Anthropic sends 4 crawlers and gives each its own token, so they can be allowed and refused separately. The tokens differ by a few characters and what the fetch means does not:

Claude-User — ai assistant. Fetches a page while Claude is answering someone's question. Claude-Code — ai assistant. Anthropic's coding agent reading a page a developer pointed it at. ClaudeBot — ai training. Collects public content that may be used to improve Claude.

Confusing two of these is the most common mistake with this operator, and it is expensive in one direction: a rule meant for the training crawler that lands on the search crawler removes you from results without stopping any training.