OpenAI and Anthropic join Google, Microsoft in backing one robots.txt AI-training opt-out
What this answers
- Which AI companies will honor the shared robots.txt AI training opt-out?
- OpenAI, Anthropic, Google and Microsoft said on October 30, 2026 that they will honor a shared robots.txt policy token, __usage_ai-training, defined in an IETF Internet-Draft by Anthropic's Kevin Kelley. Timelines differ: OpenAI says in the coming months, Microsoft from February 1, 2027, Google in early 2027.
- Does the __usage_ai-training robots.txt token stop AI crawlers from fetching my pages?
- No. The draft says the token is not a directive for crawling behavior; it expresses a preference about AI training only. It is also an individual Internet-Draft, which the IETF does not endorse and which has no formal standing as a standard.
OpenAI, Anthropic, Google and Microsoft said on October 30 that they will honor a single robots.txt policy token, __usage_ai-training, so a site can opt out of AI model training across all four with one rule (per reports of the announcements, including The Verge via AI Weekly and Unite.AI on OpenAI’s post). The token is defined in an individual IETF Internet-Draft, “Machine-Readable Training Preference Signal,” authored by Anthropic’s Kevin Kelley and dated October 30; the IETF notes such drafts are not endorsed and have no formal standing. OpenAI says it aims to implement the signal “in the coming months” and that its existing robots.txt controls keep applying. Google and Microsoft had already published their own plans the same day, for 2027 and from February 1, 2027 respectively; Anthropic is reported to have started implementing it.
The draft says the token is not a directive for crawling behavior: it expresses a preference about training only, and wildcard User-agent: * groups do not match it. It also leaves content already collected, and answer-time use of pages, outside what it controls.
Why it matters: Until now a publisher wanting out of AI training had to name each company's crawler separately; four of the largest AI companies committing to one token turns an IETF draft into the first opt-out most of them say they will read, though only for training, not crawling or answers.
The record: OpenAIAnthropicGoogle
Via IETF Datatracker (draft-kelley-aipref-training-preference-signal-00) ↗ · OpenAI ↗
Posted to the wire October 30, 2026. Edited by Joe Balewski.