The record

Cloudflare

What is Cloudflare?

The infrastructure layer where crawler access is actually enforced. Its defaults reach a large share of the web, so its policy changes are the industry's, whether or not anyone voted.

Cloudflare is where crawler policy stops being an opinion and becomes a setting. Its data underpins much of this beat — automated requests crossed 57.5% of web traffic in June 2026, with 52% of AI crawler requests serving model training against 2.6% for live queries — and its products increasingly set the terms: Pay Per Crawl evolved into usage-based payments for AI search, and an Agent Readiness dashboard now scores domains on crawler rules, machine-readable content and agent-callable APIs. Because its defaults reach a large share of the web, a change to them is an industry event; IAB Australia built its four-verdict crawler matrix explicitly around Cloudflare's September 15 changes. The record is thin on one thing: what those defaults do to sites that are already customers, which Cloudflare's public documentation does not clearly say.

6 items on the wire, newest first. Elsewhere: official · Wikipedia · Wikidata