Common Crawl, Mistral, Cohere bots visit, then vanish from logs
The wire’s own Cloudflare logs recorded four registry crawlers for the first time in the week of September 21, 2026: Common Crawl’s CCBot, Mistral’s MistralAI-User, Cohere’s cohere-ai and Huawei’s PetalBot, alongside dozens of unregistered scanners led by ShapBot’s 872 requests. One week later, CCBot, MistralAI-User and cohere-ai logged zero requests, and Google-Extended — present in the ledger since early September — also fell to zero; only PetalBot’s traffic held. The same week’s data recorded 3,519 AI-crawler requests against zero AI-referral pageloads, down from 10 the week before, on the wire’s own Crawl Ledger.
Why it matters: It is a first-party reminder that a registry crawler's appearance in a given week can be a one-off rather than a standing presence, which weakens any single week's crawl count as a signal of real training or answer use.
The record: CloudflareGoogle
Via Crawl Ledger: crawler roster ↗
Posted to the wire October 9, 2026. Edited by Joe Balewski.