ContextFetch finds bigger sites block AI crawlers more often
What this answers
- What percentage of websites block OpenAI's GPTBot crawler?
- 13.6% of the 7,897 top-10,000 sites with a readable robots.txt block GPTBot entirely, per a ContextFetch scan run October 19, 2026 — the most-blocked of the 17 AI crawlers it checked was ByteDance's Bytespider (14.4%), the least-blocked was Anthropic's Claude-SearchBot (4.6%).
- Do bigger websites block AI crawlers more than smaller ones?
- Yes, per ContextFetch: sites ranked 1-1,000 by Majestic block GPTBot 17.4% of the time and at least one of 17 AI crawlers 27.1% of the time, versus 12.1% and 17.9% for sites ranked 5,001-10,000.
ContextFetch, a web-retrieval API vendor, fetched robots.txt as “ContextFetchBot” from the 10,000 most-linked-to domains in the Majestic Million on October 19, 2026, and could read rules on 7,897 of them. Among the 17 AI crawlers it checked, ByteDance’s Bytespider was blocked most often (14.4%), followed closely by GPTBot (13.6%); Anthropic’s Claude-SearchBot was blocked least (4.6%). Overall, 20.3% of sites block at least one of the 17 crawlers and 2.3% block all of them.
Splitting the same sites by Majestic rank, blocking tracks size: sites ranked 1-1,000 block GPTBot 17.4% of the time and at least one AI crawler 27.1% of the time, falling to 12.1% and 17.9% for sites ranked 5,001-10,000. Separately, 11.6% of the 7,462 sites that answered a request for /llms.txt serve one, and the Markdown ContextFetch’s own engine extracts from a homepage runs a median 0.4% the size of the page’s raw HTML, across 6,472 homepages it could fetch.
Why it matters: The size gradient suggests AI-crawler blocking tracks which sites have the resources to run bot-management infrastructure, not a uniform industry stance on AI access.
The record: OpenAIAnthropicllms.txt
Glossary: AI crawlersllms.txt
Via ContextFetch ↗
Posted to the wire October 25, 2026. Edited by Joe Balewski.