Google: robots.txt limits what gets crawled, not what AI does with the data

Robots.txt is “an access preference protocol” governed by RFC 9309 — it controls whether a page may be fetched, and “nothing in it expresses a purpose limitation,” Google Search Relations’ Gary Illyes and Cherry Prommawin said in a Google Search Central video published October 21, per PPC Land. Disallowing a crawler “has nothing to do with” what happens to data already collected from a page it was allowed to crawl, they said. The pair also confirmed robots.txt silently drops any line outside RFC 9309’s syntax — including emoji in a user-agent token, since the field allows only letters, underscores and hyphens — with no warning or error shown to the site owner.

The video did not address Google-Extended, the separate token Google says site owners can use to opt out of having crawled content used for AI training without affecting Search inclusion — leaving open how that purpose-specific control fits inside a protocol its own hosts describe as access-only.