The Atlantic's robots.txt allows GPTBot, but Cloudflare still blocks it

The wire’s crawler access panel caught four verified shifts between its September 20 and 27 captures. The Atlantic’s robots.txt became readable again for the first time since the wire reported its GPTBot block on September 13 — the file had returned a 403 at every capture in between — and its directives now name GPTBot, OAI-SearchBot and ChatGPT-User as allowed, while Claude-SearchBot, anthropic-ai and PerplexityBot remain disallowed outright. The Atlantic’s live homepage still returns Cloudflare’s 403 to GPTBot itself, unchanged since September 13, so the published policy and the enforced one now disagree. Search Engine Land’s robots.txt, readable throughout the wire’s tracking, started returning the same Cloudflare challenge on September 27, closing off the one part of the site the panel could still read after its homepage went behind an identical wall on September 16. NPR’s robots.txt added a disallow-all rule for Anthropic’s Claude-SearchBot, a token distinct from the ClaudeBot and anthropic-ai rules already there. Press Gazette’s homepage began returning a 403 to GPTBot and ClaudeBot while continuing to let PerplexityBot through unblocked.