# Google: robots.txt limits what gets crawled, not what AI does with the data

Published: 2026-10-21T21:35:22.973Z · Source: PPC Land (https://ppc.land/google-says-robots-txt-limits-ai-crawling-not-what-bots-do-with-data/)
Source date: 2026-10-21
Flags: single-source — confirmation pending
Entities: google

Robots.txt is "an access preference protocol" governed by RFC 9309 — it controls whether a page may be fetched, and "nothing in it expresses a purpose limitation," Google Search Relations' Gary Illyes and Cherry Prommawin said in a Google Search Central video published October 21, per PPC Land. Disallowing a crawler "has nothing to do with" what happens to data already collected from a page it was allowed to crawl, they said. The pair also confirmed robots.txt silently drops any line outside RFC 9309's syntax — including emoji in a user-agent token, since the field allows only letters, underscores and hyphens — with no warning or error shown to the site owner.

The video did not address Google-Extended, the separate token Google says site owners can use to opt out of having crawled content used for AI training without affecting Search inclusion — leaving open how that purpose-specific control fits inside a protocol its own hosts describe as access-only.

Why it matters: Robots.txt is a search-era protocol being read as an AI-training opt-out, and Google's own description of what it legally does under RFC 9309 — access only, no purpose limitation — doesn't match that use without a separate signal like Google-Extended.

## What this answers

**Does blocking an AI crawler in robots.txt stop it from using already-crawled data?**

No — per Google Search Relations, robots.txt is an access protocol that only controls whether a page gets crawled, not what happens to data already collected from it.

**Can you use emoji in a robots.txt user-agent line?**

No — per Google, user-agent tokens only allow letters, underscores and hyphens, so emoji and other custom characters are silently dropped rather than flagged as errors.


Canonical: https://anythingengineoptimization.com/item/2026-10-21-google-robots-txt-limits-what-gets-crawled-not-what-ai-does-with-the/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/