# The Atlantic's robots.txt allows GPTBot, but Cloudflare still blocks it

Published: 2026-09-27T21:18:26.515Z · Source: Crawler access panel (https://anythingengineoptimization.com/crawl-ledger/)
Source date: 2026-09-27
Entities: cloudflare, openai, anthropic

The wire's crawler access panel caught four verified shifts between its September 20 and 27 captures. The Atlantic's robots.txt became readable again for the first time since the wire reported its GPTBot block on September 13 — the file had returned a 403 at every capture in between — and its directives now name GPTBot, OAI-SearchBot and ChatGPT-User as allowed, while Claude-SearchBot, anthropic-ai and PerplexityBot remain disallowed outright. The Atlantic's live homepage still returns Cloudflare's 403 to GPTBot itself, unchanged since September 13, so the published policy and the enforced one now disagree. Search Engine Land's robots.txt, readable throughout the wire's tracking, started returning the same Cloudflare challenge on September 27, closing off the one part of the site the panel could still read after its homepage went behind an identical wall on September 16. NPR's robots.txt added a disallow-all rule for Anthropic's Claude-SearchBot, a token distinct from the ClaudeBot and anthropic-ai rules already there. Press Gazette's homepage began returning a 403 to GPTBot and ClaudeBot while continuing to let PerplexityBot through unblocked.

Why it matters: It shows two publishers' stated crawler policy and their live enforcement pulling in different directions the same week — The Atlantic's robots.txt says GPTBot is welcome while Cloudflare still turns it away at the door — and shows a trade-press outlet on this beat losing machine-readable access to its own policy.

## What this answers

**Does The Atlantic block GPTBot?**

Its live homepage still returns a Cloudflare 403 to GPTBot as of the wire's September 27 capture, the same block reported on September 13 — but The Atlantic's own robots.txt, readable again for the first time since that block began, lists GPTBot, OAI-SearchBot and ChatGPT-User as allowed.

**Can AI crawlers read Search Engine Land's robots.txt?**

Not as of September 27 — the file started returning a Cloudflare challenge (403) for the first time in the wire's tracking, joining a homepage that has required the same challenge since September 16.


Canonical: https://anythingengineoptimization.com/item/2026-09-27-the-atlantic-s-robots-txt-allows-gptbot-but-cloudflare-still-blocks-it/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/