Forbes blocks PerplexityBot; El País reopens to GPTBot and ClaudeBot

The wire’s crawler access panel caught shifts at three publishers between its October 18 and October 25 captures. Forbes’s homepage began returning a 403 to PerplexityBot, where it had drawn a 200 a week earlier, now served by the bot-detection vendor DataDome in place of the Istio proxy that answered a week before; a plain-text probe sent with a browser’s user-agent string, not a rendered browser, drew the same new 403. GPTBot and ClaudeBot were unaffected, continuing to draw the 402 Payment Required response they got a week earlier. Forbes’s robots.txt names no AI crawler, unchanged.

El País’s homepage returned to a 200 for GPTBot and ClaudeBot, both of which the panel found at 403 a week earlier — its third flip there since mid-September, when El País first blocked all three bots, then reopened to ClaudeBot alone four days later. El País’s robots.txt still disallows ClaudeBot, its Claude-SearchBot counterpart and PerplexityBot by name; PerplexityBot’s homepage has returned a 200 throughout, despite that rule.

Nerdwallet’s robots.txt dropped the line disallowing Google’s AI-training crawler, Google-Extended, that the wire’s October 19 robots.txt survey had found there. Nothing else in its file or its homepage access changed.

Search Engine Land’s robots.txt, readable a week earlier, now returns a 403 to the panel’s fetcher; its homepage was already challenged for all four probed agents in both captures, so the panel can no longer compare what the file says. As with any single-probe reading, a flip that doesn’t hold on the next capture would count as transient rather than a change.