# StackScan: 6.88% of sites with a robots.txt block GPTBot, most behind Cloudflare

Published: 2026-11-09T00:17:36.568Z · Source: StackScan (https://www.stackscan.com/blog/ai-crawler-statistics)
Source date: 2026-09-10
Flags: StackScan study — vendor-published, catch-up (Sep 10)
Entities: openai, cloudflare

StackScan, a web-technology data vendor, published a crawl of robots.txt files on September 10, 2026 (updated October 31), per its own report. Of 151,649,745 live websites it reached, 90,797,323 returned a robots.txt it could parse, and 6,250,586 of those (6.88%) block OpenAI's GPTBot training crawler. Among the 10.5 million sites that name GPTBot in a rule, 59.4% block it; across all live sites the share is 4.1%. StackScan says the three figures describe one fact on three bases.

Sites served through Cloudflare's network block GPTBot at 23.0% of those with a robots.txt, against 1.5% for every other site, so 84.0% of all GPTBot blocks sit behind Cloudflare. StackScan found 8,621,742 sites carrying the same Content-Signals block that Cloudflare's managed robots.txt writes, with eight crawlers listed together, and calls the pattern "consistent with" that setting; it did not test the cause. It does not state the crawl's exact dates, so the report cannot show whether the crawl fell before or after Cloudflare's September 15 retirement of that feature, which this wire logged on October 8 from HasData's index.

The top tiers block GPTBot at 13.0% (60 of 463 sites with a robots.txt in the Tranco top 1,000) and 13.1% (693 of 5,271 in the top 10,000), within about a point of the 13.6% ContextFetch and 14.1% ProxyPicker reported on different site lists. StackScan counts 5,861,965 sites that block GPTBot while allowing both OAI-SearchBot and ChatGPT-User; OAI-SearchBot is blocked on 130,684 sites (0.14%) and ChatGPT-User on 391,829 (0.43%). It also reports 54,110 sites that block Anthropic's retired anthropic-ai and Claude-Web agents but not the current ClaudeBot by name.

Limits StackScan states: the 60.9 million live sites without a parseable robots.txt, among them 10.2 million that refused the request with a 401 or 403, sit outside every percentage; it is one three-week window, not a trend; and rules are read as declared, so a block here is a robots.txt line, not proof of what a server enforces.

Why it matters: Read alone, a headline GPTBot block rate mostly measures one provider's default setting rather than thousands of separate publisher decisions, so a rate without its denominator and hosting split says little about intent.

## What this answers

**How many websites block GPTBot in robots.txt?**

In StackScan's September 2026 crawl, 6.88% of the 90.8 million sites with a readable robots.txt (6,250,586 sites) block GPTBot. Across all 151.6 million live websites StackScan counted, the share is 4.1%.

**Are most GPTBot blocks set through Cloudflare?**

StackScan found 84.0% of GPTBot blocks sit on sites behind Cloudflare's network, which block it at 23.0% against 1.5% for other sites. StackScan calls that consistent with Cloudflare's managed robots.txt setting; it did not test the cause.


Canonical: https://anythingengineoptimization.com/item/2026-11-09-stackscan-6-88-of-sites-with-a-robots-txt-block-gptbot-most-behind/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/