Can I trust the user agent at all?

For identification, yes — most traffic is not trying to deceive you, and the token tells you what the client is. As evidence, no. If a number depends on it being true, verify it.

Why do all browsers claim to be Mozilla?

A compatibility arms race from the 1990s that nobody has managed to unwind. Servers checked for `Mozilla` before sending the modern version of a page, so every browser started claiming to be it, and thirty years later they all still do. The part identifying the actual browser is a fragment near the end of the string.

Should I use the user agent to decide what to serve?

For crawlers, sometimes — serving a lighter page to a bot is legitimate as long as the content matches, since serving different content to a crawler than to a person is cloaking. For browsers, no: the strings are unreliable, increasingly frozen by privacy work, and feature detection answers the real question better.

Verification

User agent

Also known as user-agent string, UA string

4 min read · updated 2026-09-09

Definition

A user agent is a string a client sends with every HTTP request to say what it is. It is self-reported and unverified by the protocol, which makes it useful for identification and worthless as proof — anyone can send any string.

Match the token, not the string

A crawler's full header changes constantly — version numbers, product suffixes, contact URLs. What stays stable is the token: `GPTBot`, `ClaudeBot`, `PerplexityBot`.

So a filter pinned to a whole header breaks the first time an operator bumps a version, while one matching the token keeps working. Every crawler entry in this glossary names its token for exactly that reason.

Where a loose match goes wrong

Short tokens are the hazard. Matching a bare `Grok` catches any user agent containing the word — a browser extension, a research script, an app with Grok in its name — and reports all of them as xAI traffic.

The same problem hits `Copilot`, which appears in GitHub Copilot's and Edge's user agents without being Microsoft's crawler. The fix is anchoring: require the token to be followed by a version or a delimiter, so `Copilot/1.0` matches and `Edg/121 Copilot Mode` does not.

This is why a directory that lists every plausible token is worse than one that lists only the attested ones. A pattern that never matches costs nothing; one that over-matches quietly corrupts a number.

The empty user agent

A request with no user-agent header at all is not a browser — every real one sends it. Treating that as a human visitor, which is the naive default, opens an escape route no amount of pattern-writing can close.

Why the strings look the way they do

Every browser user agent begins `Mozilla/5.0` — including browsers with no connection to Mozilla — and the reason is a compatibility arms race from the 1990s that nobody has been able to unwind. Servers sniffed for `Mozilla` to decide whether to send the good version of a page, so every browser claimed to be Mozilla, and thirty years later they all still do.

The practical residue is that a browser user agent is mostly archaeology, and the part identifying the actual browser is a fragment near the end. Crawler user agents are the opposite: short, deliberate, and usually signed with a URL explaining what the crawler is — `+https://openai.com/gptbot`, `+http://www.google.com/bot.html`.

That signature is worth noticing. An operator that wanted to be hard to identify would not put its own documentation URL in the header, so the presence of one is weak evidence of good faith — and its absence, on a token nobody has documented, is a reason to be careful about what you conclude from the string alone.

The direction of travel is away from the header entirely. Browsers have been freezing and reducing what they send for years on privacy grounds, and agentic browsers send an ordinary browser string precisely because there is a real person driving. Neither trend makes crawler tokens less useful, but both are reasons not to build anything important on user-agent parsing for human traffic.

What goes wrong

Trusting it. It is a claim; verification is a separate step with a separate answer.
Matching a short token unanchored.
Case-sensitive matching. Operators are inconsistent about capitalisation — `DoubaoBot` and `Doubaobot` are the same crawler.
Counting an absent header as a person.

Worth doing

Keep a written list of tokens with their operator and purpose; a regex without that context becomes unmaintainable within a year.
Where a token is too short to match safely, say so rather than matching it anyway.

Questions

Can I trust the user agent at all?: For identification, yes — most traffic is not trying to deceive you, and the token tells you what the client is. As evidence, no. If a number depends on it being true, verify it.
Why do all browsers claim to be Mozilla?: A compatibility arms race from the 1990s that nobody has managed to unwind. Servers checked for `Mozilla` before sending the modern version of a page, so every browser started claiming to be it, and thirty years later they all still do. The part identifying the actual browser is a fragment near the end of the string.
Should I use the user agent to decide what to serve?: For crawlers, sometimes — serving a lighter page to a bot is legitimate as long as the content matches, since serving different content to a crawler than to a person is cloaking. For browsers, no: the strings are unreliable, increasingly frozen by privacy work, and feature detection answers the real question better.

See which AI assistants are reading your site.

AI crawlers take your HTML and leave without running a line of JavaScript, so a browser tag records nothing. TrueStat reads them server-side, checks each address against what the operator publishes, and shows you which pages were fetched — including the ones that returned a 404.

Start tracking All terms