Is Googlebot in my logs really Google?

Not necessarily, and that is why it is worth checking. Any script can send Googlebot as its user agent. Google publishes the addresses its crawler uses, so the claim can be tested against them — a request from outside those ranges is recorded as unverified rather than accepted.

How do I tell Googlebot apart from Google's other crawlers?

By the token, and only by the token. Google also sends Google-NotebookLM, Google-Agent, Gemini-Deep-Research, and the strings differ by a few characters while the jobs differ completely — one may be answering somebody's question while another collects training material. Match on the exact token rather than on the operator's name appearing anywhere in the header, or the four end up counted as one.

Does Googlebot run JavaScript?

No. It requests the HTML and leaves. That is why a browser-based analytics tag never records these visits: the tag is JavaScript, and nothing runs it. Seeing this crawler at all requires reading it server-side.

Why is Googlebot counted as Indexing and not something else?

Because that is the job this particular crawler does. Google runs more than one — Google-NotebookLM, Google-Agent, Gemini-Deep-Research — and they are counted separately because the decisions about them are separate. Filing them together under one operator name would let ordinary indexing look like AI interest, which is the single most misleading thing an AI traffic report can do.

Why does Googlebot not show up in Google Analytics?

Google Analytics runs in the browser. This crawler never opens a browser — it requests the HTML, reads it, and leaves, so the tracking script is never executed and no event is ever sent. Every browser-based analytics tool has the same blind spot, which is why crawler traffic has to be read from the server side to be seen at all.

What happens if a Googlebot visit cannot be verified?

It is recorded as unverified, not as a forgery. A published address list can be incomplete or behind — an operator can start using a range before it appears in the file — so a miss is not evidence of anyone pretending. A visit is only marked as spoofed when the address's own reverse DNS record resolves to somebody else, which is proof rather than absence of proof.

AI crawlers

Googlebot

Also known as Google Googlebot

5 min read · updated 2026-09-09

Definition

Googlebot is Google's crawler for search indexing. Google's search crawler, and the input to AI Overviews. Ordinary Google indexing. Blocking it removes you from Google Search, not just from AI answers.

For the reference card — Googlebot in the crawler directory.

What Googlebot is

Googlebot is the only crawler on this site where blocking is almost never the right answer. It indexes for Google Search, and Google has confirmed that the same index feeds AI Overviews — so there is no rule that keeps you in Search while removing you from the AI summary above it. Site owners who want that separation are usually looking for Google-Extended, which controls training and grounding, or GoogleOther, which is the research crawl.

Google runs more crawlers than any other operator here, and the important thing is that they are genuinely separate. Search indexing, AI training, research fetches and the agents that act for a person all carry different tokens and appear in different published address files. Treating them as one 'Google' decision is the most expensive mistake available on this page.

On this site Googlebot is counted as indexing, which is the row it appears in on your dashboard. The category matters because it decides which number moves when this crawler visits, and the four are reported separately for exactly that reason.

What its traffic looks like

Continuous, and the most predictable pattern on this page. Googlebot revisits according to how often a page changes, so a news site sees it hourly and a static page monthly. Search Console shows the same requests from Google's side, which makes this the one crawler you can reconcile against an independent record.

How it is verified

Checked as ip ranges + reverse dns. The user agent has to carry the token, and then the address is compared against what Google publishes — refetched every six hours, so a newly announced range is picked up without a deploy.

A request that matches neither is recorded as unverified rather than as a forgery: a published list can be incomplete, and an accusation needs evidence. Only a reverse DNS record resolving to somebody else produces a spoofed verdict, which is proof rather than the absence of it.

If you block it

You disappear from Google Search — not only from AI Overviews. This is the one row on this page where blocking is almost never the right answer, and the rule is one character away from the rule that blocks Google's AI training crawler instead.

Treat this as a search-engine decision rather than an AI one, because that is what it is. The question is whether the people you want are looking through this engine, and your own analytics answer it better than any general advice can.

It is worth checking what you actually lose before writing the rule. Blocking an index crawler is one of the few changes here with an effect that compounds quietly: nothing breaks on the day, and traffic simply stops arriving over the following weeks.

How it differs from Google's other crawlers

Google sends 7 crawlers and gives each its own token, so they can be allowed and refused separately. The tokens differ by a few characters and what the fetch means does not:

Google-NotebookLM — ai assistant. Fetches a page a person added to a NotebookLM notebook. Google-Agent — ai assistant. A Google agent fetching on a person's behalf. Gemini-Deep-Research — ai assistant. Gemini's deep research mode gathering sources. Google-Read-Aloud — ai assistant. Fetches a page so it can be read out loud. GoogleOther — ai training. Google product teams fetching public content, including research. Google-CloudVertexBot — ai training. Crawls sites for owners building Vertex AI agents.

Confusing two of these is the most common mistake with this operator, and it is expensive in one direction: a rule meant for the training crawler that lands on the search crawler removes you from results without stopping any training.

One real request

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Questions

Is Googlebot in my logs really Google?: Not necessarily, and that is why it is worth checking. Any script can send Googlebot as its user agent. Google publishes the addresses its crawler uses, so the claim can be tested against them — a request from outside those ranges is recorded as unverified rather than accepted.
How do I tell Googlebot apart from Google's other crawlers?: By the token, and only by the token. Google also sends Google-NotebookLM, Google-Agent, Gemini-Deep-Research, and the strings differ by a few characters while the jobs differ completely — one may be answering somebody's question while another collects training material. Match on the exact token rather than on the operator's name appearing anywhere in the header, or the four end up counted as one.
Does Googlebot run JavaScript?: No. It requests the HTML and leaves. That is why a browser-based analytics tag never records these visits: the tag is JavaScript, and nothing runs it. Seeing this crawler at all requires reading it server-side.
Why is Googlebot counted as Indexing and not something else?: Because that is the job this particular crawler does. Google runs more than one — Google-NotebookLM, Google-Agent, Gemini-Deep-Research — and they are counted separately because the decisions about them are separate. Filing them together under one operator name would let ordinary indexing look like AI interest, which is the single most misleading thing an AI traffic report can do.
Why does Googlebot not show up in Google Analytics?: Google Analytics runs in the browser. This crawler never opens a browser — it requests the HTML, reads it, and leaves, so the tracking script is never executed and no event is ever sent. Every browser-based analytics tool has the same blind spot, which is why crawler traffic has to be read from the server side to be seen at all.
What happens if a Googlebot visit cannot be verified?: It is recorded as unverified, not as a forgery. A published address list can be incomplete or behind — an operator can start using a range before it appears in the file — so a miss is not evidence of anyone pretending. A visit is only marked as spoofed when the address's own reverse DNS record resolves to somebody else, which is proof rather than absence of proof.

See whether Googlebot is reaching your pages.

AI crawlers take your HTML and leave without running a line of JavaScript, so a browser tag records nothing. TrueStat reads them server-side, checks each address against what the operator publishes, and shows you which pages were fetched — including the ones that returned a 404.

Start tracking All terms