# Cloudflare rewrites robots.txt, gates access on disclosure

Published: 2026-08-23T13:19:28.724Z · Source: PPC Land (https://ppc.land/cloudflare-blocks-opaque-ai-crawlers-from-sites-that-disallow-training/)
Source date: 2026-08-23
Flags: single-source — confirmation pending
Entities: cloudflare

Cloudflare on August 21 launched Bot Preference Sync, which auto-generates a site's robots.txt from its dashboard-configured AI crawler policy, rolling out across all plans, free through Enterprise, over the following week. Crawlers that both search and train keep access to sites disallowing training only if they meet four conditions: honoring the no-training preference, offering an AI-summary opt-out, reporting URL-level training use alongside search metrics, and publicly showing the block doesn't hurt search visibility. Separately, new customers can now flag a domain as ad-funded, which defaults its training setting to disallow.

Why it matters: It closes the long-standing gap between what a site's robots.txt says and what Cloudflare's edge actually enforces, and makes continued crawler access to training-disallowed sites conditional on public disclosure rather than a self-declared promise.

## What this answers

**Does Cloudflare block AI crawlers that refuse to allow training?**

Cloudflare's Bot Preference Sync, launched August 21, 2026, lets sites set a no-training preference that still allows cooperating search crawlers through, but crawlers doing both search and training keep that access only if they meet four public-disclosure conditions.

**What is Cloudflare Bot Preference Sync?**

A feature announced August 21, 2026 that automatically rewrites a site's robots.txt to match its AI-crawler policy set in the Cloudflare dashboard, available on every plan from free to Enterprise.


Canonical: https://anythingengineoptimization.com/item/2026-08-23-cloudflare-rewrites-robots-txt-gates-access-on-disclosure/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/