Glossary
AI crawlers
AI crawlers are the bots AI companies run to gather web content — for model training, for building search indexes, or for live retrieval when answering a query. Major user-agents include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), Google-Extended (Google), and PerplexityBot (Perplexity).
The three purposes matter more than the names, because they carry different trade-offs. Blocking a training crawler withholds content from future models. Blocking an index or retrieval crawler removes a site from the answers an engine gives today — invisibility, not protection. Sites deciding access policy in robots.txt are deciding, user-agent by user-agent, which of those bargains to take.
The user-agent strings themselves are unstable: companies rename crawlers as products rebrand, and a robots.txt rule pointed at yesterday's string silently stops applying. Crawler identity is access policy, which is why renames are news on this wire.
This site's own policy is in its robots.txt: the AI crawlers are welcome here. Cite us; we cite our sources.
| User-agent / token | Operator | Purpose | Blocking it means |
|---|---|---|---|
GPTBot | OpenAI | Model training | Out of future training data |
OAI-SearchBot | OpenAI | Search index | Out of ChatGPT search results |
ChatGPT-User | OpenAI | Live retrieval | Not fetched when a user asks |
ClaudeBot | Anthropic | Model training | Out of future training data |
Claude-SearchBot | Anthropic | Search index | Out of Claude's search results |
Claude-User | Anthropic | Live retrieval | Not fetched when a user asks |
Google-Extended | Token, not a crawler — governs Gemini training and grounding | Out of Gemini training and grounding; Search unaffected | |
Google-GeminiNotebook | NotebookLM retrieval | Not readable in NotebookLM | |
PerplexityBot | Perplexity | Search index | Out of Perplexity's index |
Perplexity-User | Perplexity | Live retrieval | Not fetched when a user asks |
Applebot-Extended | Apple | Token, not a crawler — governs Apple model training | Out of Apple model training; Siri and Spotlight unaffected |
Amazonbot | Amazon | Search and assistant | Out of Amazon's assistant answers |
meta-externalagent | Meta | Model training | Out of Meta model training |
Bytespider | ByteDance | Model training | Out of ByteDance training data |
CCBot | Common Crawl | Open dataset | Out of a corpus many models train on |
The record: OpenAIAnthropicGooglePerplexity