# SearchDock scan finds top sites block AI crawlers at the server, not just robots.txt

Published: 2026-11-02T00:16:31.885Z · Source: SearchDock (https://searchdock.io/research/ai-crawler-access-2026/)
Source date: 2026-10-20
Flags: SearchDock study — vendor-published, catch-up (Oct 20)
Entities: openai, anthropic, llms-txt

SearchDock, an SEO platform vendor, requested the homepages of the Tranco top 1,000 domains on October 20, 2026 as a browser and as GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Googlebot, and read each site's robots.txt. Of the 555 that served a homepage, 27.2% blocked at least one AI crawler: 20.5% in robots.txt and 10.3% at the server or CDN. In robots.txt, Bytespider was disallowed most often (16.9%), then CCBot (16.6%), GPTBot (14.4%) and ClaudeBot (13.7%); Anthropic's Claude-SearchBot was disallowed on 8.1% and OpenAI's OAI-SearchBot on 7.7%, against 1.8% for Googlebot.

SearchDock counted a server block only where the same site served its Googlebot request normally. On that test, 49 of 542 sites tested (9%) turned GPTBot away and 47 of 545 (8.6%) turned ClaudeBot away; 35 sites (6.5% of those tested) did so to GPTBot while their robots.txt allowed it. It also found 22.5% of the 555 serve an llms.txt file. The company's requests came from one location using crawler user-agent strings rather than the vendors' own IP ranges, so sites that admit only verified crawler IPs can look blocked, and it fetched each site once. Per-site results are published as a CSV under CC BY 4.0.

Why it matters: A robots.txt read alone misses server-level refusals: on this panel 9% of sites turned GPTBot away at the server against 14.4% that disallowed it in robots.txt, and on most of the server blockers the file looked open.

## What this answers

**How many top websites block AI crawlers?**

In SearchDock's October 20, 2026 scan of the Tranco top 1,000 domains, 27.2% of the 555 sites that served a homepage blocked at least one AI crawler, either in robots.txt (20.5%) or at the server or CDN (10.3%).

**Can a site block GPTBot even if its robots.txt allows it?**

Yes. SearchDock found 6.5% of the sites it tested turned GPTBot away at the server while serving Googlebot normally, even though their robots.txt allowed GPTBot; 6.4% did the same to ClaudeBot.


Canonical: https://anythingengineoptimization.com/item/2026-11-02-searchdock-scan-finds-top-sites-block-ai-crawlers-at-the-server-not/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/