Cloudflare rewrites robots.txt, gates access on disclosure

Cloudflare on August 21 launched Bot Preference Sync, which auto-generates a site’s robots.txt from its dashboard-configured AI crawler policy, rolling out across all plans, free through Enterprise, over the following week. Crawlers that both search and train keep access to sites disallowing training only if they meet four conditions: honoring the no-training preference, offering an AI-summary opt-out, reporting URL-level training use alongside search metrics, and publicly showing the block doesn’t hurt search visibility. Separately, new customers can now flag a domain as ad-funded, which defaults its training setting to disallow.