Glossary
AI crawlers
AI crawlers are the bots AI companies run to gather web content — for model training, for building search indexes, or for live retrieval when answering a query. Major user-agents include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), Google-Extended (Google), and PerplexityBot (Perplexity).
The three purposes matter more than the names, because they carry different trade-offs. Blocking a training crawler withholds content from future models. Blocking an index or retrieval crawler removes a site from the answers an engine gives today — invisibility, not protection. Sites deciding access policy in robots.txt are deciding, user-agent by user-agent, which of those bargains to take.
The user-agent strings themselves are unstable: companies rename crawlers as products rebrand, and a robots.txt rule pointed at yesterday's string silently stops applying. Crawler identity is access policy, which is why renames are news on this wire.
This site's own policy is in its robots.txt: the AI crawlers are welcome here. Cite us; we cite our sources.
The record: OpenAIAnthropicGooglePerplexity