# Sites with OpenAI deals let ChatGPT's search in and keep Claude's out

Published: 2026-10-19T03:28:19.743Z · Source: AEO Wire crawler panel (https://anythingengineoptimization.com/crawl-ledger/)
Source date: 2026-10-19
Flags: wire's own measurement — robots.txt rules on 52 of a fixed 55-site panel, read October 19; policy as stated, not what each server enforces
Entities: openai, chatgpt, anthropic, perplexity, google, reddit

Which AI search bot a site lets in depends, on AEO Wire's crawler panel, on whether the site has a licensing deal with OpenAI. On October 19 the wire read the robots.txt files of the panel's 55 sites (27 national and international news outlets, 9 trade publications and 16 reference, review and community sites) and worked out which rule applies to each AI bot. 52 files were readable.

Across all 52, OpenAI's OAI-SearchBot, which feeds ChatGPT search, was disallowed on 16 sites. Anthropic's Claude-SearchBot was disallowed on 28 and PerplexityBot on 30. Googlebot and Bingbot were each disallowed on one site, Reddit, whose robots.txt disallows every crawler.

The difference comes from 17 sites with public OpenAI deals: Vox Media's The Verge, Condé Nast's Wired and Ars Technica, The Wall Street Journal, The Guardian, Reuters, AP, The Atlantic, Le Monde, El País, People Inc's People, Investopedia and Allrecipes, the Financial Times, Axios, The Washington Post and Reddit. Of those 17, 16 let OAI-SearchBot in and three let Claude-SearchBot in. Among the 35 sites without an OpenAI deal, the three search bots are treated alike: 15 disallow OAI-SearchBot, 14 Claude-SearchBot and 15 PerplexityBot. Thirteen sites let ChatGPT's search bot in and keep Claude's out, and all 13 are on OpenAI's partner lists. Only the BBC does the reverse. The Claude blocks don't look like leftover blanket rules: 11 of the 13 sites name Claude-SearchBot explicitly, and sites without OpenAI deals disallow Claude-SearchBot no more often than OAI-SearchBot (14 and 15 of 35). Training bots split the same way: the 17 OpenAI partners disallow GPTBot on 3 sites and ClaudeBot on 16. Google-Extended, Google's AI-training control, is disallowed on 12 sites with an OpenAI deal and 12 without.

None of these publishers has an announced licensing deal with Anthropic. Perplexity has revenue-share deals with several of these publishers through its Publishers' Program and Comet Plus; of the five checked here, only Le Monde lets PerplexityBot in.

A robots.txt rule is a request, not a lock, and the companies treat requests differently. OpenAI says sites that disallow OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links," and that for ChatGPT-User, which fetches pages a user asks about, "robots.txt rules may not apply." Perplexity says Perplexity-User "generally ignores robots.txt rules" for the same reason. Anthropic says all three of its bots honor robots.txt. Anthropic's and Perplexity's search tools may also draw on third-party search indexes, so a block on their own crawler stops that crawler from fetching the page, not every route to the reporting. Google says blocking Google-Extended, its AI-training control, "does not impact a site's inclusion in Google Search," which is where AI Overviews and AI Mode appear. Licensing can also grant access a robots.txt refuses: Reddit's file disallows everyone, and Reddit licenses its content to Google and OpenAI.

The panel was fixed on September 11 for the wire's crawler probe and is not a sample of the web. Three sites refused every automated fetch (Search Engine Land, Politico, Stack Overflow). Verdicts are for each site's homepage path; several sites block specific sections only. Deal status comes from each company's announcement or Reuters, AP, Axios, The Verge or Digiday reporting; Reuters appears on OpenAI's partner list without announced terms. The data shows which sites' rules line up with which deals, not why they were written. Raw verdicts: ops/crawler-probes/who-can-read-2026-10-19.json.

What Claude's own tools could reach, October 19. The wire ran Claude's web search and web fetch, as they run in Claude Code, against twelve sites. Site-restricted searches for recent AI coverage returned an error, "The following domains are not accessible to our user agent," for The New York Times, The Wall Street Journal, Reuters, AP, The Guardian, The Atlantic, The Verge and the BBC. They returned results for CNN, The Washington Post, Axios and TechCrunch. Fetch was refused by the tool itself for The Verge, Reuters and the BBC. CNN's server answered a cnn.com article request with HTTP 451. The Washington Post's server answered with 403, though its robots.txt allows Claude's bots. TechCrunch returned the article. Unfiltered searches about two scoops returned retellings but not the original: nine other sites for The Verge's October 17 report on sites blocking AI agents, and Law360 plus aggregators for Reuters' report on Reddit and Perplexity's law firm. The tool's refusals don't map exactly onto the robots.txt rules read the same day. The BBC disallows ClaudeBot and anthropic-ai but not Claude-SearchBot, and was refused. CNN disallows all of Anthropic's agents by name, and its pages still came back in search. Anthropic doesn't publish how its tools decide. Claude's consumer app wasn't tested, and the same test couldn't be run on ChatGPT.

Same question in the consumer apps, October 19. The wire then asked the free ChatGPT app and claude.ai (Sonnet 5.5), each in a new chat with web search, to "find this week's AI coverage from [site]" and link the articles, for the same twelve sites. ChatGPT linked articles on the site itself for 9: The Wall Street Journal, Reuters, AP, The Guardian, The Atlantic, The Verge, The Washington Post, Axios and TechCrunch. It failed on The New York Times (it linked only nytimes.com search pages, citing a third-party archive), the BBC (secondhand links, including a lookalike domain, bbc.com.im) and CNN. Those are the three sites whose robots.txt disallows OAI-SearchBot. claude.ai linked articles on the site itself for 4: CNN, The Washington Post, Axios and TechCrunch. For The Wall Street Journal it found two articles through a news tracker. It found nothing for the others, or only syndicated copies and retellings. Asked who broke two scoops, ChatGPT linked The Verge's and Reuters' originals. claude.ai named both outlets correctly but linked neither: it gave secondary write-ups, and a Reuters wire copy on TradingView. In one chat claude.ai offered to open the page in the user's own Chrome; the wire declined, to keep the test to Claude's own search. One run each; AI answers vary from run to run.

| Site | OpenAI deal | OAI-SearchBot | Claude-SearchBot | PerplexityBot | Googlebot | Google-Extended |
|---|---|---|---|---|---|---|
| allrecipes.com | yes | open | blocked | blocked | open | blocked |
| apnews.com | yes | open | blocked | blocked | open | open |
| arstechnica.com | yes | open | blocked | blocked | open | blocked |
| axios.com | yes | open | open | open | open | open |
| elpais.com | yes | open | blocked | blocked | open | open |
| ft.com | yes | open | open | blocked | open | blocked |
| investopedia.com | yes | open | blocked | blocked | open | blocked |
| lemonde.fr | yes | open | blocked | open | open | blocked |
| people.com | yes | open | blocked | blocked | open | blocked |
| reddit.com | yes | blocked | blocked | blocked | blocked | blocked |
| reuters.com | yes | open | blocked | blocked | open | blocked |
| theatlantic.com | yes | open | blocked | blocked | open | blocked |
| theguardian.com | yes | open | blocked | blocked | open | open |
| theverge.com | yes | open | blocked | blocked | open | blocked |
| washingtonpost.com | yes | open | open | blocked | open | open |
| wired.com | yes | open | blocked | blocked | open | blocked |
| wsj.com | yes | open | blocked | blocked | open | blocked |
| adweek.com |  | open | open | open | open | open |
| bbc.com |  | blocked | open | blocked | open | blocked |
| bloomberg.com |  | open | open | blocked | open | blocked |
| businessinsider.com |  | open | open | open | open | open |
| cnet.com |  | blocked | blocked | blocked | open | open |
| cnn.com |  | blocked | blocked | blocked | open | open |
| dailymail.co.uk |  | blocked | blocked | blocked | open | blocked |
| digiday.com |  | open | open | open | open | open |
| forbes.com |  | open | open | open | open | open |
| healthline.com |  | open | open | open | open | open |
| imdb.com |  | blocked | blocked | blocked | open | blocked |
| independent.co.uk |  | open | open | open | open | open |
| latimes.com |  | open | open | open | open | open |
| medium.com |  | open | open | open | open | open |
| nerdwallet.com |  | open | open | open | open | blocked |
| newsday.com |  | open | open | open | open | open |
| npr.org |  | blocked | blocked | blocked | open | blocked |
| nytimes.com |  | blocked | blocked | blocked | open | blocked |
| ppc.land |  | open | open | open | open | open |
| pressgazette.co.uk |  | open | open | open | open | open |
| quora.com |  | blocked | blocked | blocked | open | open |
| searchenginejournal.com |  | open | open | open | open | open |
| seattletimes.com |  | blocked | blocked | blocked | open | blocked |
| seroundtable.com |  | open | open | open | open | open |
| spiegel.de |  | blocked | blocked | open | open | open |
| substack.com |  | open | open | open | open | open |
| techcrunch.com |  | open | open | open | open | blocked |
| telegraph.co.uk |  | blocked | blocked | blocked | open | open |
| tripadvisor.com |  | open | open | open | open | blocked |
| usatoday.com |  | blocked | blocked | blocked | open | blocked |
| webmd.com |  | open | open | open | open | open |
| wikihow.com |  | blocked | blocked | blocked | open | open |
| wikipedia.org |  | open | open | open | open | open |
| yelp.com |  | blocked | blocked | blocked | open | blocked |
| zdnet.com |  | blocked | blocked | blocked | open | open |

Why it matters: Publishers' stated AI-crawler rules now track who pays them: on this panel, sites with OpenAI deals allow OpenAI's search and training bots and disallow Anthropic's, while sites without deals treat the companies alike. A rule governs a company's own bot fetching pages, not every route an AI has to a publisher's reporting.

## What this answers

**Do websites block Claude more than ChatGPT?**

On AEO Wire's panel of 52 major news, trade and reference sites, yes: robots.txt rules on October 19, 2026 shut out Anthropic's Claude-SearchBot on 28 sites and OpenAI's OAI-SearchBot on 16. The gap sits almost entirely at sites with OpenAI licensing deals.

**Do OpenAI licensing deals affect which AI bots a site allows?**

On this panel they line up exactly. Of 17 sites with public OpenAI deals, 16 let OAI-SearchBot in and three let Claude-SearchBot in. Of 35 sites without one, 15 block OAI-SearchBot, 14 block Claude-SearchBot and 15 block PerplexityBot. The data shows the overlap, not the reason for it.

**Which AI search bot is blocked most often?**

PerplexityBot, on 30 of 52 panel sites, then Claude-SearchBot on 28 and OAI-SearchBot on 16. Googlebot and Bingbot were each disallowed only by Reddit, whose robots.txt disallows every crawler while it licenses access to Google and OpenAI.


Canonical: https://anythingengineoptimization.com/item/2026-10-19-sites-with-openai-deals-let-chatgpt-s-search-in-and-keep-claude-s-out/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/