# Google: robots.txt limits AI crawling, not AI training use

Published: 2026-10-23T06:39:22.467Z · Source: Google Search Central (YouTube) (https://www.youtube.com/watch?v=-jHONLQgE-s)
Second source: Search Engine Journal (https://www.searchenginejournal.com/google-illyes-robots-txt-crawling-ai-training/593957/)
Source date: 2026-10-21
Entities: google

Robots.txt, as RFC 9309 defines it, governs which URLs a crawler may request — not what happens to the content afterward, Google's Gary Illyes said in a Search Central video published Wednesday, per Search Engine Journal. To keep one AI crawler out while allowing others, Illyes said, a site has to target that crawler's specific user-agent token in the file; a made-up directive like "No-AI-Chicken-Training" would simply be ignored by Google's parser. Separately, Google-Extended — the token that controls whether Google can use crawled content to train future Gemini models or for grounding — doesn't affect a site's inclusion in Search. Illyes is also co-author of an IETF draft that would add a standard "Content-Usage" rule for stating AI-training preferences in robots.txt; the Working Group has not yet sent it to the IETF's steering body for review.

Why it matters: The clarification matters for anyone writing AI-crawler rules: robots.txt still can't express a standalone 'crawl but don't train' preference, and the IETF draft meant to add one is a Working Group document Google's own Illyes co-authored that hasn't yet gone to the IETF's steering body for review.

## What this answers

**Does blocking an AI crawler in robots.txt stop a company from training on content it already fetched?**

No. Google's Gary Illyes says robots.txt is an access preference protocol governing which URLs a crawler may request, not what happens to the content afterward; keeping one AI crawler out while allowing others means targeting that crawler's specific user-agent token.

**Does blocking Google-Extended in robots.txt remove a site from Google Search?**

No, per Google: Google-Extended only controls whether crawled content can be used to train future Gemini models or for grounding; it doesn't affect a site's inclusion in Search or act as a ranking signal.


Canonical: https://anythingengineoptimization.com/item/2026-10-23-google-robots-txt-limits-ai-crawling-not-ai-training-use/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/