SearchDock scan finds top sites block AI crawlers at the server, not just robots.txt

SearchDock, an SEO platform vendor, requested the homepages of the Tranco top 1,000 domains on October 20, 2026 as a browser and as GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Googlebot, and read each site’s robots.txt. Of the 555 that served a homepage, 27.2% blocked at least one AI crawler: 20.5% in robots.txt and 10.3% at the server or CDN. In robots.txt, Bytespider was disallowed most often (16.9%), then CCBot (16.6%), GPTBot (14.4%) and ClaudeBot (13.7%); Anthropic’s Claude-SearchBot was disallowed on 8.1% and OpenAI’s OAI-SearchBot on 7.7%, against 1.8% for Googlebot.

SearchDock counted a server block only where the same site served its Googlebot request normally. On that test, 49 of 542 sites tested (9%) turned GPTBot away and 47 of 545 (8.6%) turned ClaudeBot away; 35 sites (6.5% of those tested) did so to GPTBot while their robots.txt allowed it. It also found 22.5% of the 555 serve an llms.txt file. The company’s requests came from one location using crawler user-agent strings rather than the vendors’ own IP ranges, so sites that admit only verified crawler IPs can look blocked, and it fetched each site once. Per-site results are published as a CSV under CC BY 4.0.