El País reopens to ClaudeBot; The Atlantic ends Perplexity's paid crawl access

The wire’s crawler access panel caught four shifts between its September 16 and 20 captures. El País’s homepage returned to a 200 for both a browser request and ClaudeBot, reversing the 403 block the wire reported four days earlier — even though El País’s robots.txt still disallows ClaudeBot by name; GPTBot and PerplexityBot there remain blocked. The Atlantic moved PerplexityBot off Cloudflare’s Pay Per Crawl track, the 402 response offering paid access, onto a flat 403 block, while keeping Anthropic’s ClaudeBot on the paid track unchanged. Politico’s robots.txt became readable to the panel for the first time, disclosing disallow-all rules for GPTBot, ClaudeBot and PerplexityBot by name, and its homepage dropped Cloudflare’s browser challenge (403 to 200) while continuing to challenge all three bots. The New York Times’ homepage began returning a 403 to the panel’s browser probe, now served by the bot-detection vendor DataDome in place of the Envoy server it ran on September 16; its existing blocks on GPTBot, ClaudeBot and PerplexityBot were unchanged.