# Common Crawl, Mistral, Cohere bots visit, then vanish from logs

Published: 2026-10-09T05:38:42.367Z · Source: Crawl Ledger: crawler roster (https://anythingengineoptimization.com/crawl-ledger/)
Source date: 2026-10-09
Entities: cloudflare, google

The wire's own Cloudflare logs recorded four registry crawlers for the first time in the week of September 21, 2026: Common Crawl's CCBot, Mistral's MistralAI-User, Cohere's cohere-ai and Huawei's PetalBot, alongside dozens of unregistered scanners led by ShapBot's 872 requests. One week later, CCBot, MistralAI-User and cohere-ai logged zero requests, and Google-Extended — present in the ledger since early September — also fell to zero; only PetalBot's traffic held. The same week's data recorded 3,519 AI-crawler requests against zero AI-referral pageloads, down from 10 the week before, on the wire's own Crawl Ledger.

Why it matters: It is a first-party reminder that a registry crawler's appearance in a given week can be a one-off rather than a standing presence, which weakens any single week's crawl count as a signal of real training or answer use.

## What this answers

**Does Common Crawl's bot crawl a site every week once it shows up?**

Not necessarily — on the wire's own site, Common Crawl's CCBot logged requests in the week of September 21, 2026, then zero the following week, per the site's own Cloudflare data.

**What crawlers besides the major AI chatbots hit AI-search sites?**

The wire's Cloudflare logs caught dozens of unregistered scanners in the week of September 21, 2026, led by ShapBot at 872 requests, alongside SEO tools like AhrefsBot and SemrushBot.


Canonical: https://anythingengineoptimization.com/item/2026-10-09-common-crawl-mistral-cohere-bots-visit-then-vanish-from-logs/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/