# Cloudflare — the record

The infrastructure layer where crawler access is actually enforced. Its defaults reach a large share of the web, so its policy changes are the industry's, whether or not anyone voted.

Cloudflare is where crawler policy stops being an opinion and becomes a setting. Its data underpins much of this beat — automated requests crossed 57.5% of web traffic in June 2026, with 52% of AI crawler requests serving model training against 2.6% for live queries — and its products increasingly set the terms: Pay Per Crawl evolved into usage-based payments for AI search, and an Agent Readiness dashboard now scores domains on crawler rules, machine-readable content and agent-callable APIs. Because its defaults reach a large share of the web, a change to them is an industry event; IAB Australia built its four-verdict crawler matrix explicitly around Cloudflare's September 15 changes. The record is thin on one thing: what those defaults do to sites that are already customers, which Cloudflare's public documentation does not clearly say.

Also known as: https://www.cloudflare.com/ · https://en.wikipedia.org/wiki/Cloudflare · https://www.wikidata.org/wiki/Q2140714

- 2026-08-08: Donor-database site blocks Amazon's AI crawler after logging 117,000 daily requests (https://anythingengineoptimization.com/item/2026-08-08-donor-database-site-blocks-amazon-s-ai-crawler-after-logging-117-000/)
  PatronView, a database of U.S. museum and cultural-institution donor records, blocked Amazon's Amzn-SearchBot crawler in early August after founder Nick…
- 2026-08-06: Cloudflare launches dashboard tools to measure agent access and AI citations (https://anythingengineoptimization.com/item/2026-08-06-cloudflare-launches-dashboard-tools-to-measure-agent-access-and-ai/)
  Cloudflare added two tools to its dashboard on August 6: Agent Readiness, a technical audit of how well AI crawlers and agents can discover and read a…
- 2026-08-01: Bipartisan bill would force AI crawlers to identify themselves (https://anythingengineoptimization.com/item/2026-08-01-bipartisan-bill-would-force-ai-crawlers-to-identify-themselves/)
  Reps. Laurel Lee (R-FL), Valerie Foushee (D-NC), and Gus Bilirakis (R-FL) introduced the Stealth Bot Prohibition Act, H.R.
- 2026-08-01: Cloudflare evolves Pay Per Crawl into usage-based AI search payments (https://anythingengineoptimization.com/item/2026-08-01-cloudflare-evolves-pay-per-crawl-into-usage-based-ai-search-payments/)
  Cloudflare announced two initiatives on July 1, 2026 to reshape AI-era crawling economics.
- 2026-08-01: IAB Australia sorts AI crawlers into four access verdicts (https://anythingengineoptimization.com/item/2026-08-01-iab-australia-sorts-ai-crawlers-into-four-access-verdicts/)
  IAB Australia published version 1.0 of its Bots and Crawler Guidance and Decision Matrix on July 22, 2026, unveiling it at the Discovery: AI and Search…
- 2026-08-01: Time starts selling ads inside its AI-agent-readable markdown pages (https://anythingengineoptimization.com/item/2026-08-01-time-starts-selling-ads-inside-its-ai-agent-readable-markdown-pages/)
  Time has converted its web pages into markdown — stripped-down, text-only copies without design or images — to make its content easier for AI crawlers and…

Canonical: https://anythingengineoptimization.com/entity/cloudflare/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/entity/