# IETF's AI robots.txt opt-out tags barely used, study finds

Published: 2026-10-12T00:18:55.071Z · Source: ProxyPicker (https://proxypicker.com/robots-txt-ai-preferences/)
Source date: 2026-10-08
Flags: ProxyPicker study — vendor-published

Just 13 of 6,735 working robots.txt files use the IETF's Content-Usage directive — the draft standard letting a site separately flag AI training, AI use and search crawling — per a ProxyPicker check of Chrome's top 10,000 most-visited origins, run October 8. Twelve of the 13 were country editions of one company, Vinted. By comparison, 950 of the same sites (14.1%) carry a named block on OpenAI's GPTBot alone, and smaller shares block ClaudeBot, CCBot and Google-Extended by name.

Why it matters: It shows sites are still managing AI crawler access by naming individual bots in robots.txt one by one, not through the structured preference vocabulary the IETF has been drafting since 2025.

## What this answers

**Do any websites use the IETF's new robots.txt AI-training opt-out standard?**

Almost none. A ProxyPicker check of 6,735 working robots.txt files (from Chrome's top 10,000 sites) found only 13 — 0.19% — using the IETF's Content-Usage directive, and 12 of those 13 were country editions of a single company, Vinted.


Canonical: https://anythingengineoptimization.com/item/2026-10-12-ietf-s-ai-robots-txt-opt-out-tags-barely-used-study-finds/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/