Glossary
The vocabulary of AI traffic, written out.
59 crawlers explained at length — how each behaves, how it is verified, and what allowing it actually costs — plus 12 entries on the ideas underneath: forward-confirmed reverse DNS, spoofing, zero-click, cookieless measurement.
Looking up a token you found in a log? The crawler directory is the short answer.
- AI crawlers
The bots themselves — who sends them and what each one fetches for.
- Verification
How you tell a real crawler from something wearing its name.
- AI visibility
Being read, cited and found by assistants rather than by search.
- Analytics
Measurement terms — sessions, attribution, and what they leave out.
- Privacy
Cookies, consent, and what a measurement product should not collect.
A
- AI answer trafficAI visibilityAI answer traffic is a fetch made while an assistant is answering somebody's question — the model decided a page was worth reading and went and read it, with a person waiting.
- AI training crawlerAI crawlersAn AI training crawler collects public pages as material that may be used to train a future model.
- AI2BotAI crawlersAI2Bot is Allen AI's crawler for collecting training data.
- AliyunBotAI crawlersAliyunBot is Alibaba's crawler for traffic that is neither answering nor training.
- Amzn-SearchBotAI crawlersAmzn-SearchBot is Amazon's crawler for search indexing.
- Amzn-UserAI crawlersAmzn-User is Amazon's crawler for answering questions.
- ApplebotAI crawlersApplebot is Apple's crawler for search indexing.
B
- BaiduspiderAI crawlersBaiduspider is Baidu's crawler for search indexing.
- bedrockbotAI crawlersbedrockbot is Amazon's crawler for collecting training data.
- bingbotAI crawlersbingbot is Microsoft Bing's crawler for search indexing.
- BingPreviewAI crawlersBingPreview is Microsoft Bing's crawler for answering questions.
- Bot filteringAnalyticsBot filtering is separating automated requests from human ones in analytics.
- BytespiderAI crawlersBytespider is ByteDance's crawler for collecting training data.
C
- CCBotAI crawlersCCBot is Common Crawl's crawler for collecting training data.
- ChatGLM-SpiderAI crawlersChatGLM-Spider is Zhipu AI's crawler for collecting training data.
- ChatGPT-AgentAI crawlersChatGPT-Agent is OpenAI's crawler for answering questions.
- ChatGPT-UserAI crawlersChatGPT-User is OpenAI's crawler for answering questions.
- Claude-CodeAI crawlersClaude-Code is Anthropic's crawler for answering questions.
- Claude-SearchBotAI crawlersClaude-SearchBot is Anthropic's crawler for search indexing.
- Claude-UserAI crawlersClaude-User is Anthropic's crawler for answering questions.
- ClaudeBotAI crawlersClaudeBot is Anthropic's crawler for collecting training data.
- cohere-aiAI crawlerscohere-ai is Cohere's crawler for collecting training data.
- Cookieless analyticsPrivacyCookieless analytics measures site usage without storing an identifier on the visitor's device.
- CopilotAI crawlersCopilot is Microsoft Bing's crawler for answering questions.
D
- DeepSeekBotAI crawlersDeepSeekBot is DeepSeek's crawler for collecting training data.
- DoubaobotAI crawlersDoubaobot is ByteDance's crawler for traffic that is neither answering nor training.
- DuckAssistBotAI crawlersDuckAssistBot is DuckDuckGo's crawler for answering questions.
- DuckDuckBotAI crawlersDuckDuckBot is DuckDuckGo's crawler for search indexing.
G
- Gemini-Deep-ResearchAI crawlersGemini-Deep-Research is Google's crawler for answering questions.
- Generative engine optimizationAI visibilityGenerative engine optimization is the practice of writing and structuring content so that AI assistants cite it when answering questions.
- Google-AgentAI crawlersGoogle-Agent is Google's crawler for answering questions.
- Google-CloudVertexBotAI crawlersGoogle-CloudVertexBot is Google's crawler for collecting training data.
- Google-NotebookLMAI crawlersGoogle-NotebookLM is Google's crawler for answering questions.
- Google-Read-AloudAI crawlersGoogle-Read-Aloud is Google's crawler for answering questions.
- GooglebotAI crawlersGooglebot is Google's crawler for search indexing.
- GoogleOtherAI crawlersGoogleOther is Google's crawler for collecting training data.
- GPTBotAI crawlersGPTBot is OpenAI's crawler for collecting training data.
- Grok-DeepSearchAI crawlersGrok-DeepSearch is xAI's crawler for answering questions.
- GrokBotAI crawlersGrokBot is xAI's crawler for collecting training data.
K
M
- meta-externaladsAI crawlersmeta-externalads is Meta's crawler for traffic that is neither answering nor training.
- meta-externalagentAI crawlersmeta-externalagent is Meta's crawler for collecting training data.
- meta-externalfetcherAI crawlersmeta-externalfetcher is Meta's crawler for answering questions.
- meta-webindexerAI crawlersmeta-webindexer is Meta's crawler for search indexing.
- MistralAI-IndexAI crawlersMistralAI-Index is Mistral's crawler for search indexing.
- MistralAI-TrainingAI crawlersMistralAI-Training is Mistral's crawler for collecting training data.
- MistralAI-UserAI crawlersMistralAI-User is Mistral's crawler for answering questions.
O
P
- PanguBotAI crawlersPanguBot is Huawei's crawler for collecting training data.
- Perplexity-UserAI crawlersPerplexity-User is Perplexity's crawler for answering questions.
- PerplexityBotAI crawlersPerplexityBot is Perplexity's crawler for search indexing.
- PetalBotAI crawlersPetalBot is Huawei's crawler for search indexing.
Q
S
- Server-side trackingAnalyticsServer-side tracking records a request where it arrives — at the server or edge — rather than from a script running in the visitor's browser.
- Spoofed crawlerVerificationA spoofed crawler is any request that claims a crawler's identity in its user-agent header without belonging to that operator.