# AI crawlers

AI crawlers are the bots AI companies run to gather web content — for model training, for building search indexes, or for live retrieval when answering a query. Major user-agents include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), Google-Extended (Google), and PerplexityBot (Perplexity).

The three purposes matter more than the names, because they carry different trade-offs. Blocking a training crawler withholds content from future models. Blocking an index or retrieval crawler removes a site from the answers an engine gives today — invisibility, not protection. Sites deciding access policy in robots.txt are deciding, user-agent by user-agent, which of those bargains to take.

The user-agent strings themselves are unstable: companies rename crawlers as products rebrand, and a robots.txt rule pointed at yesterday's string silently stops applying. Crawler identity is access policy, which is why renames are news on this wire.

This site's own policy is in its robots.txt: the AI crawlers are welcome here. Cite us; we cite our sources.

| User-agent / token | Operator | Purpose | Blocking it means |
| --- | --- | --- | --- |
| GPTBot | OpenAI | Model training | Out of future training data |
| OAI-SearchBot | OpenAI | Search index | Out of ChatGPT search results |
| ChatGPT-User | OpenAI | Live retrieval | Not fetched when a user asks |
| ClaudeBot | Anthropic | Model training | Out of future training data |
| Claude-SearchBot | Anthropic | Search index | Out of Claude's search results |
| Claude-User | Anthropic | Live retrieval | Not fetched when a user asks |
| Google-Extended | Google | Token, not a crawler — governs Gemini training and grounding | Out of Gemini training and grounding; Search unaffected |
| Google-GeminiNotebook | Google | NotebookLM retrieval | Not readable in NotebookLM |
| PerplexityBot | Perplexity | Search index | Out of Perplexity's index |
| Perplexity-User | Perplexity | Live retrieval | Not fetched when a user asks |
| Applebot-Extended | Apple | Token, not a crawler — governs Apple model training | Out of Apple model training; Siri and Spotlight unaffected |
| Amazonbot | Amazon | Search and assistant | Out of Amazon's assistant answers |
| meta-externalagent | Meta | Model training | Out of Meta model training |
| Bytespider | ByteDance | Model training | Out of ByteDance training data |
| CCBot | Common Crawl | Open dataset | Out of a corpus many models train on |

Named user-agents and robots.txt tokens, by what blocking each one actually costs. Two of these are not crawlers at all — Google-Extended and Applebot-Extended are directives about use, so blocking them withholds content from training without removing you from any answer.

**What is an AI crawler?** A bot an AI company runs to gather web content — for model training, for building a search index, or for live retrieval when answering a query. GPTBot, ClaudeBot, Google-Extended, and PerplexityBot are among the major user-agents.

**Should I block AI crawlers?** The three crawler purposes carry different trade-offs: blocking a training bot withholds content from future models, while blocking an index or retrieval bot removes you from today's answers. The decision is per-bot, not all-or-nothing.

Canonical: https://anythingengineoptimization.com/glossary/ai-crawlers/
From the Anything Engine Optimization glossary — https://anythingengineoptimization.com/glossary/