Corrections
Every correction the wire has run, in one place. Corrections are appended to the item and dated — never silently edited. This log exists whether it's empty or not.
Cloudflare syncs robots.txt, sets four conditions for crawlers that also train
- 2026-08-27: This item originally said Cloudflare "rewrites" a site's robots.txt and that Bot Preference Sync "auto-generates" the file. It does neither. Cloudflare prepends a managed block and, in its own words, any existing Disallow directives are maintained; the sync is opt-in and can be turned off at any time. The headline has been changed from "Cloudflare rewrites robots.txt, gates access on disclosure", which also implied the disclosure conditions apply to crawling generally when they apply only to crawlers that both search and train, seeking access to sites that disallow training. The URL is unchanged.