GrokBot

GrokBot signs its requests with a link to x.ai, which is how the fleet was identified despite xAI publishing no crawler page. It is the training side of that fleet: separate from the fetches Grok makes while answering somebody, and separate again from the crawl that builds its index.

xAI publishes no crawler documentation at all, which is unusual for an operator of its size — it does not appear even in the community-maintained agent lists. What is known comes from logs: several of its crawlers sign themselves with an x.ai URL, which is how the fleet was identified.

On this site GrokBot is counted as training, which is the row it appears in on your dashboard. The category matters because it decides which number moves when this crawler visits, and the four are reported separately for exactly that reason.

Broad and slow: many pages, few repeats, spread over days or weeks rather than concentrated. Training crawls tend to want everything once, so the long tail of a site sees traffic it never gets from search — old posts, deep archive pages, anything reachable by a link. Bandwidth is the usual reason people notice these at all.

xAI publishes neither a machine-readable list of the addresses this crawler uses nor a reverse DNS record pointing back at itself. The user-agent header is therefore the only evidence, and anyone can send it — so a visit is recorded as an unverified claim rather than as a confirmed fetch.

That is not a gap in the measurement so much as an honest ceiling. A report that showed these as verified would be inventing a confidence nobody can support, and the number it produced would be the first thing to fall apart when somebody asked where it came from.

xAI stops collecting your pages as training material. Nothing about how you appear in search or in live answers changes — those are separate crawlers with separate rules.

Nothing you see today changes if you block this. Training crawls feed models that ship months later, so the decision is about whether your work becomes part of a commercial model, not about this quarter's traffic.

The honest counter-argument is that being in the training data is part of how a model comes to mention you at all. There is no measurement that settles this either way — anyone who tells you there is has not looked — so it is a decision about your terms rather than about your numbers.

xAI sends 5 crawlers and gives each its own token, so they can be allowed and refused separately. The tokens differ by a few characters and what the fetch means does not:

xAI-SearchBot — ai assistant. Grok fetching a page while answering. Grok-DeepSearch — ai assistant. Grok's deep research mode gathering sources for a long answer. xAI-Grok — search indexing. Retrieves and indexes pages so Grok can find them later. xAI-Bot — other bots. The family name xAI's own robots.txt examples use.

Confusing two of these is the most common mistake with this operator, and it is expensive in one direction: a rule meant for the training crawler that lands on the search crawler removes you from results without stopping any training.