# The Atlantic's robots.txt now restricts GPTBot, OAI-SearchBot and ChatGPT-User

Published: 2026-11-01T22:18:12.203Z · Source: Crawler access panel (https://anythingengineoptimization.com/crawl-ledger/)
Source date: 2026-11-01
Flags: single-probe — panel capture
Entities: openai, meta

The wire's crawler access panel found ten changes between its October 25 and November 1 captures. The Atlantic's robots.txt moved GPTBot, OAI-SearchBot and ChatGPT-User from "allow" to "disallow some": each now has at least one Disallow rule, but not a block on the whole site. The panel records that a rule exists, not which paths it covers, so what is newly off limits is not known from this capture. The wire's September 27 item had logged those three as allowed. The Atlantic's homepage still returns a 403 to GPTBot and PerplexityBot and a 402 to ClaudeBot, the same codes as a week earlier, and its file still disallows ClaudeBot, Claude-SearchBot and PerplexityBot outright.

NPR's robots.txt now disallows Meta's meta-externalagent crawler across the whole site; the panel had no entry for it a week earlier. NPR already blocked every other tracked AI crawler, and the wire logged its Claude-SearchBot rule on September 27.

At the homepage level, elpais.com returned a 403, up from a 200, to a browser-style probe, GPTBot and PerplexityBot, reversing the October 25 reopening the wire reported. Forbes moved from a 402 to a 403 for GPTBot and ClaudeBot, and Stack Overflow moved from a 403 to a 402 for ClaudeBot. Each reading is a single probe with no retries, and a flip that does not hold on the next capture counts as transient, not a policy change.

Why it matters: A publisher's stated crawler policy can change between weekly reads without any announcement, and the wire's panel is one of the few places where each change gets a date.

## What this answers

**Does The Atlantic still allow GPTBot in robots.txt?**

Not in full. The wire's panel read The Atlantic's robots.txt on October 25 as allowing GPTBot, OAI-SearchBot and ChatGPT-User, and on November 1 as carrying at least one Disallow rule for each. The panel records that a rule exists, not which paths it covers.

**Has NPR blocked Meta's AI crawler?**

As of the wire's November 1 capture, NPR's robots.txt disallows meta-externalagent site-wide. The panel had no entry for that crawler in NPR's file a week earlier.


Canonical: https://anythingengineoptimization.com/item/2026-11-01-the-atlantic-s-robots-txt-now-restricts-gptbot-oai-searchbot-and/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/