# El País reopens to ClaudeBot; The Atlantic ends Perplexity's paid crawl access

Published: 2026-09-20T21:49:16.665Z · Source: Crawler access panel (https://anythingengineoptimization.com/crawl-ledger/)
Source date: 2026-09-20
Entities: cloudflare, anthropic, perplexity

The wire's crawler access panel caught four shifts between its September 16 and 20 captures. El País's homepage returned to a 200 for both a browser request and ClaudeBot, reversing the 403 block the wire reported four days earlier — even though El País's robots.txt still disallows ClaudeBot by name; GPTBot and PerplexityBot there remain blocked. The Atlantic moved PerplexityBot off Cloudflare's Pay Per Crawl track, the 402 response offering paid access, onto a flat 403 block, while keeping Anthropic's ClaudeBot on the paid track unchanged. Politico's robots.txt became readable to the panel for the first time, disclosing disallow-all rules for GPTBot, ClaudeBot and PerplexityBot by name, and its homepage dropped Cloudflare's browser challenge (403 to 200) while continuing to challenge all three bots. The New York Times' homepage began returning a 403 to the panel's browser probe, now served by the bot-detection vendor DataDome in place of the Envoy server it ran on September 16; its existing blocks on GPTBot, ClaudeBot and PerplexityBot were unchanged.

Why it matters: It shows crawler enforcement moving independently of stated policy in both directions the same week — El País reopened to a bot its own robots.txt still disallows, while The Atlantic ended one AI crawler's paid-access option and kept another's, shifts visible only in the wire's own probe data, not in either site's robots.txt.

## What this answers

**Does El País block ClaudeBot?**

Not as of the wire's September 20 capture — the block reported on September 16 reversed for both a browser user agent and ClaudeBot, which now return a 200, though El País's robots.txt still disallows ClaudeBot by name and GPTBot and PerplexityBot remain blocked at 403.

**Does The Atlantic charge AI crawlers to access its site?**

For ClaudeBot, yes — Anthropic's crawler still draws Cloudflare's 402 Pay Per Crawl response as of September 20. PerplexityBot lost that paid option the same week, moving to a flat 403 block instead.


Canonical: https://anythingengineoptimization.com/item/2026-09-20-el-pa-s-reopens-to-claudebot-the-atlantic-ends-perplexity-s-paid/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/