Who can read which site

Method, and what this is not

• What's read. Each site's robots.txt, fetched once a week by the wire's poller, which identifies itself (see how we fetch). For each bot, its own User-agent group applies if the file names it; otherwise the catch-all group does (RFC 9309). The verdict is for the homepage; some sites close only sections.
• A rule, not a lock. robots.txt is a request. Firewalls and CDNs enforce access separately, and can block a bot the file allows or admit one it disallows. OpenAI says its user-triggered fetcher may not follow robots.txt; Perplexity says its own "generally ignores" it; Anthropic says its bots honor it.
• Deals open other doors. Licensed partners often receive feeds or API access, so a disallowed bot doesn't mean the company has no access to the content. Reddit disallows every bot in its file and licenses its posts to Google and OpenAI.
• Search indexes leak. An AI tool can see a page's headline and snippet through another company's search index even when its own crawler is disallowed.
• The panel was fixed in September 2026 for the wire's crawler probe: the publishers this wire reads, the publishers in the AI copyright suits, and the reference sites the engines cite most. It isn't a sample of the web.