BBC's robots.txt now disallows Claude-SearchBot, Anthropic's search crawler

The wire’s crawler access panel found seven changes between its November 1 and November 8 captures. The BBC’s robots.txt now carries a Disallow: / rule for Claude-SearchBot, Anthropic’s search crawler. The November 1 capture recorded no rule for that token, and the wire’s October 19 survey found the BBC disallowing ClaudeBot and anthropic-ai but not Claude-SearchBot. The BBC has disallowed ClaudeBot across the whole site in every capture since at least October 11. The November 1 capture kept only a summary of the file, not the file itself, so the day the line was added is known only to within that week.

One practical effect is visible in the wire’s own tools. The desk’s page fetcher checks robots.txt before it reads a page, and on November 8 it refused a BBC page because of the new rule. The wire’s October 23 ledger entry recorded the BBC as readable to it.

The other six changes are homepage status codes, each a single probe with no retries. The panel’s ClaudeBot probe drew a 403 from El País, up from a 200, while PerplexityBot moved from a 403 to a 200. Press Gazette returned a 403 to GPTBot and ClaudeBot, up from a 200 for both; its robots.txt itself returns a 403 to the panel, so the file cannot be compared. WikiHow returned a 403 to ClaudeBot and PerplexityBot, up from a 200, while GPTBot still drew a 200 despite robots.txt rules that disallow all three. A flip that does not hold on the next capture counts as transient rather than a change.