Forbes blocks PerplexityBot; El País reopens to GPTBot and ClaudeBot
The wire's crawler access panel caught shifts at three publishers between its October 18 and October 25 captures.
Via Crawler access panel ↗ · Oct 25 · · Tools & Vendors, AEO
Original reporting
Measurements you can check
Our own crawler observations and comparisons of primary documents. Each item states what was measured, when, and what the result cannot establish.
Publisher crawler policies · Crawl Ledger · AEO Index historical readings (paused)
Source audit · November 2026
An AI search percentage needs a denominator
We checked 20 claims in our own Numbers reference. Read the corrections, qualifications, unresolved questions and downloadable review ledger.
Original reporting, newest first
wire's own measurement — robots.txt rules on 52 of a fixed 55-site panel, read October 19; policy as stated, not what each server enforces
Sites with OpenAI deals let ChatGPT's search in and keep Claude's out
Corrected — read the correction
Which AI search bot a site lets in depends, on AEO Wire's crawler panel, on whether the site has a licensing deal with OpenAI.
Via AEO Wire crawler panel ↗ · Oct 19 · · AEO, Engines, Law & Money
Common Crawl, Mistral, Cohere bots visit, then vanish from logs
The wire's own Cloudflare logs recorded four registry crawlers for the first time in the week of September 21, 2026: Common Crawl's CCBot, Mistral's…
Via Crawl Ledger: crawler roster ↗ · Oct 9 · · Tools & Vendors, AEO
The Atlantic's robots.txt allows GPTBot, but Cloudflare still blocks it
The wire's crawler access panel caught four verified shifts between its September 20 and 27 captures.
Via Crawler access panel ↗ · Sep 27 · · Tools & Vendors, AEO
El País reopens to ClaudeBot; The Atlantic ends Perplexity's paid crawl access
The wire's crawler access panel caught four shifts between its September 16 and 20 captures.
Via Crawler access panel ↗ · Sep 20 · · Tools & Vendors, AEO
El País blocks GPTBot, ClaudeBot and PerplexityBot
The wire's crawler access panel caught El País's homepage flip from a 200 to a 403 for GPTBot, ClaudeBot and PerplexityBot alike between the September 13…
Via Crawler access panel ↗ · Sep 16 · · Tools & Vendors, AEO
People Inc. sites and The Atlantic block GPTBot, still bill Claude, Perplexity
The wire's crawler access panel — 55 publisher domains probed daily as GPTBot, ClaudeBot, PerplexityBot and a browser — caught three People Inc.…
Via Crawler access panel ↗ · Sep 13 · · Tools & Vendors, AEO
wire analysis of published documentation — Cloudflare has published no compliance list, and a company could satisfy a condition privately
No major AI crawler publicly meets all four of Cloudflare's conditions
When Cloudflare launched Bot Preference Sync on August 21 it set four conditions a crawler doing both search and training must meet to keep reaching sites…
Via Cloudflare Blog ↗ · Aug 27 · · Engines, Tools & Vendors, AEO