Anything Engine Optimization The rolling record of the AI-search industry.
Send a tip
A white two-way arrow glyph on a deep slate field.
Illustration: AEO Wire
wire's own measurement — robots.txt rules on 52 of a fixed 55-site panel, read October 19; policy as stated, not what each server enforces
AEO Wire crawler panel · Oct 19 AEOEnginesLaw & Money

Sites with OpenAI deals let ChatGPT's search in and keep Claude's out

Which AI search bot a site lets in depends, on AEO Wire’s crawler panel, on whether the site has a licensing deal with OpenAI. On October 19 the wire read the robots.txt files of the panel’s 55 sites (27 national and international news outlets, 9 trade publications and 16 reference, review and community sites) and worked out which rule applies to each AI bot. 52 files were readable.

Across all 52, OpenAI’s OAI-SearchBot, which feeds ChatGPT search, was disallowed on 16 sites. Anthropic’s Claude-SearchBot was disallowed on 28 and PerplexityBot on 30. Googlebot and Bingbot were each disallowed on one site, Reddit, whose robots.txt disallows every crawler.

The difference comes from 17 sites with public OpenAI deals: Vox Media’s The Verge, Condé Nast’s Wired and Ars Technica, The Wall Street Journal, The Guardian, Reuters, AP, The Atlantic, Le Monde, El País, People Inc’s People, Investopedia and Allrecipes, the Financial Times, Axios, The Washington Post and Reddit. Of those 17, 16 let OAI-SearchBot in and three let Claude-SearchBot in. Among the 35 sites without an OpenAI deal, the three search bots are treated alike: 15 disallow OAI-SearchBot, 14 Claude-SearchBot and 15 PerplexityBot. Thirteen sites let ChatGPT’s search bot in and keep Claude’s out, and all 13 are on OpenAI’s partner lists. Only the BBC does the reverse. The Claude blocks don’t look like leftover blanket rules: 11 of the 13 sites name Claude-SearchBot explicitly, and sites without OpenAI deals disallow Claude-SearchBot no more often than OAI-SearchBot (14 and 15 of 35). Training bots split the same way: the 17 OpenAI partners disallow GPTBot on 3 sites and ClaudeBot on 16. Google-Extended, Google’s AI-training control, is disallowed on 12 sites with an OpenAI deal and 12 without.

None of these publishers has an announced licensing deal with Anthropic. Perplexity has revenue-share deals with several of these publishers through its Publishers’ Program and Comet Plus; of the five checked here, only Le Monde lets PerplexityBot in.

A robots.txt rule is a request, not a lock, and the companies treat requests differently. OpenAI says sites that disallow OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links,” and that for ChatGPT-User, which fetches pages a user asks about, “robots.txt rules may not apply.” Perplexity says Perplexity-User “generally ignores robots.txt rules” for the same reason. Anthropic says all three of its bots honor robots.txt. Anthropic’s and Perplexity’s search tools may also draw on third-party search indexes, so a block on their own crawler stops that crawler from fetching the page, not every route to the reporting. Google says blocking Google-Extended, its AI-training control, “does not impact a site’s inclusion in Google Search,” which is where AI Overviews and AI Mode appear. Licensing can also grant access a robots.txt refuses: Reddit’s file disallows everyone, and Reddit licenses its content to Google and OpenAI.

The panel was fixed on September 11 for the wire’s crawler probe and is not a sample of the web. Three sites refused every automated fetch (Search Engine Land, Politico, Stack Overflow). Verdicts are for each site’s homepage path; several sites block specific sections only. Deal status comes from each company’s announcement or Reuters, AP, Axios, The Verge or Digiday reporting; Reuters appears on OpenAI’s partner list without announced terms. The data shows which sites’ rules line up with which deals, not why they were written. Raw verdicts: ops/crawler-probes/who-can-read-2026-10-19.json.

What Claude’s own tools could reach, October 19. The wire ran Claude’s web search and web fetch, as they run in Claude Code, against twelve sites. Site-restricted searches for recent AI coverage returned an error, “The following domains are not accessible to our user agent,” for The New York Times, The Wall Street Journal, Reuters, AP, The Guardian, The Atlantic, The Verge and the BBC. They returned results for CNN, The Washington Post, Axios and TechCrunch. Fetch was refused by the tool itself for The Verge, Reuters and the BBC. CNN’s server answered a cnn.com article request with HTTP 451. The Washington Post’s server answered with 403, though its robots.txt allows Claude’s bots. TechCrunch returned the article. Unfiltered searches about two scoops returned retellings but not the original: nine other sites for The Verge’s October 17 report on sites blocking AI agents, and Law360 plus aggregators for Reuters’ report on Reddit and Perplexity’s law firm. The tool’s refusals don’t map exactly onto the robots.txt rules read the same day. The BBC disallows ClaudeBot and anthropic-ai but not Claude-SearchBot, and was refused. CNN disallows all of Anthropic’s agents by name, and its pages still came back in search. Anthropic doesn’t publish how its tools decide. Claude’s consumer app wasn’t tested, and the same test couldn’t be run on ChatGPT.

SiteOpenAI dealOAI-SearchBotClaude-SearchBotPerplexityBotGooglebotGoogle-Extended
allrecipes.comyesopenblockedblockedopenblocked
apnews.comyesopenblockedblockedopenopen
arstechnica.comyesopenblockedblockedopenblocked
axios.comyesopenopenopenopenopen
elpais.comyesopenblockedblockedopenopen
ft.comyesopenopenblockedopenblocked
investopedia.comyesopenblockedblockedopenblocked
lemonde.fryesopenblockedopenopenblocked
people.comyesopenblockedblockedopenblocked
reddit.comyesblockedblockedblockedblockedblocked
reuters.comyesopenblockedblockedopenblocked
theatlantic.comyesopenblockedblockedopenblocked
theguardian.comyesopenblockedblockedopenopen
theverge.comyesopenblockedblockedopenblocked
washingtonpost.comyesopenopenblockedopenopen
wired.comyesopenblockedblockedopenblocked
wsj.comyesopenblockedblockedopenblocked
adweek.comopenopenopenopenopen
bbc.comblockedopenblockedopenblocked
bloomberg.comopenopenblockedopenblocked
businessinsider.comopenopenopenopenopen
cnet.comblockedblockedblockedopenopen
cnn.comblockedblockedblockedopenopen
dailymail.co.ukblockedblockedblockedopenblocked
digiday.comopenopenopenopenopen
forbes.comopenopenopenopenopen
healthline.comopenopenopenopenopen
imdb.comblockedblockedblockedopenblocked
independent.co.ukopenopenopenopenopen
latimes.comopenopenopenopenopen
medium.comopenopenopenopenopen
nerdwallet.comopenopenopenopenblocked
newsday.comopenopenopenopenopen
npr.orgblockedblockedblockedopenblocked
nytimes.comblockedblockedblockedopenblocked
ppc.landopenopenopenopenopen
pressgazette.co.ukopenopenopenopenopen
quora.comblockedblockedblockedopenopen
searchenginejournal.comopenopenopenopenopen
seattletimes.comblockedblockedblockedopenblocked
seroundtable.comopenopenopenopenopen
spiegel.deblockedblockedopenopenopen
substack.comopenopenopenopenopen
techcrunch.comopenopenopenopenblocked
telegraph.co.ukblockedblockedblockedopenopen
tripadvisor.comopenopenopenopenblocked
usatoday.comblockedblockedblockedopenblocked
webmd.comopenopenopenopenopen
wikihow.comblockedblockedblockedopenopen
wikipedia.orgopenopenopenopenopen
yelp.comblockedblockedblockedopenblocked
zdnet.comblockedblockedblockedopenopen

Why it matters: Publishers' stated AI-crawler rules now track who pays them: on this panel, sites with OpenAI deals allow OpenAI's search and training bots and disallow Anthropic's, while sites without deals treat the companies alike. A rule governs a company's own bot fetching pages, not every route an AI has to a publisher's reporting.

The record: OpenAIChatGPTAnthropicPerplexityGoogleReddit

Glossary: AEOAI OverviewsAI Mode

Via AEO Wire crawler panel ↗ · OpenAI crawler documentation ↗

Posted to the wire October 18, 2026. Edited by Joe Balewski.

Corrections

  • 2026-10-19: This item's summary first said which assistant someone uses "now decides which publishers it can read." robots.txt rules govern a company's own crawlers, not every route to a publisher's content: licensed partners often receive feeds or APIs, and AI search tools can draw headlines and snippets from other search indexes. The summary now says what the data shows, and the item adds the training-bot and Google-Extended counts and the check that the Claude blocks are named, not blanket.

More on the wire

← Back to the wire