Google: robots.txt limits AI crawling, not AI training use
What this answers
- Does blocking an AI crawler in robots.txt stop a company from training on content it already fetched?
- No. Google's Gary Illyes says robots.txt is an access preference protocol governing which URLs a crawler may request, not what happens to the content afterward; keeping one AI crawler out while allowing others means targeting that crawler's specific user-agent token.
- Does blocking Google-Extended in robots.txt remove a site from Google Search?
- No, per Google: Google-Extended only controls whether crawled content can be used to train future Gemini models or for grounding; it doesn't affect a site's inclusion in Search or act as a ranking signal.
Robots.txt, as RFC 9309 defines it, governs which URLs a crawler may request — not what happens to the content afterward, Google’s Gary Illyes said in a Search Central video published Wednesday, per Search Engine Journal. To keep one AI crawler out while allowing others, Illyes said, a site has to target that crawler’s specific user-agent token in the file; a made-up directive like “No-AI-Chicken-Training” would simply be ignored by Google’s parser. Separately, Google-Extended — the token that controls whether Google can use crawled content to train future Gemini models or for grounding — doesn’t affect a site’s inclusion in Search. Illyes is also co-author of an IETF draft that would add a standard “Content-Usage” rule for stating AI-training preferences in robots.txt; the Working Group has not yet sent it to the IETF’s steering body for review.
Why it matters: The clarification matters for anyone writing AI-crawler rules: robots.txt still can't express a standalone 'crawl but don't train' preference, and the IETF draft meant to add one is a Working Group document Google's own Illyes co-authored that hasn't yet gone to the IETF's steering body for review.
The record: Google
Glossary: AI crawlersGrounding
Via Google Search Central (YouTube) ↗ · Search Engine Journal ↗
Posted to the wire October 23, 2026. Edited by Joe Balewski.