IETF's AI robots.txt opt-out tags barely used, study finds

Just 13 of 6,735 working robots.txt files use the IETF’s Content-Usage directive — the draft standard letting a site separately flag AI training, AI use and search crawling — per a ProxyPicker check of Chrome’s top 10,000 most-visited origins, run October 8. Twelve of the 13 were country editions of one company, Vinted. By comparison, 950 of the same sites (14.1%) carry a named block on OpenAI’s GPTBot alone, and smaller shares block ClaudeBot, CCBot and Google-Extended by name.