# AI Search Best Practices 2026 — Current, Dated, Referenced

What the record supports, practice by practice. Every entry carries the date it was last verified against new evidence and the references behind it.

Last updated 2026-09-08.

# Level 1 — Foundations

## 1. Decide your AI crawler policy deliberately — and per bot

Mechanism: robots.txt, per user-agent

AI companies run separate crawlers for model training, search indexing, and live retrieval, and the trade-offs differ: blocking a training bot withholds content from future models, while blocking an index or retrieval bot removes you from today's answers. Set robots.txt per user-agent with that distinction in mind, and re-check the strings periodically — companies rename crawlers as products rebrand, and a rule pointed at yesterday's string silently stops applying. IAB Australia's Bots and Crawler Guidance formalizes the same call into four verdicts per crawler category — allow, allow with conditions, require licensing, or block — a workable checklist for making the decision deliberately rather than defaulting to one blanket rule.

A robots.txt block is a request, not a lock. TollBit found roughly 15% of AI page fetchers it tracked in the first half of 2026 reached European publisher URLs marked disallowed, with OpenAI's ChatGPT-User among the bots most often crossing blocks aimed specifically at it — treat a block as a policy statement worth logging compliance against, not a guarantee.

For Cloudflare-proxied sites, part of that policy is now set in a dashboard rather than hand-edited into the file. Bot Preference Sync, rolling out across every plan from free to Enterprise starting August 21, 2026, keeps robots.txt in step with an AI-crawler policy configured in the dashboard. It prepends a managed block rather than replacing the file — Cloudflare says any existing Disallow directives are maintained — and the sync is opt-in and can be switched off at any time. It also changes what a no-training preference actually buys: a crawler that does both search and training keeps access to a site marked no-training only if it meets four conditions — honoring the preference, offering an AI-summary opt-out, reporting URL-level training use alongside search metrics, and publicly showing the block doesn't hurt search visibility. Continued access now turns on public disclosure, not a crawler's self-declared promise.

Checked against Google, OpenAI and Anthropic's own documentation on August 27, 2026, none of the three publicly meets all four conditions — Google clears three, OpenAI two, Anthropic none. The condition every operator fails is the third: URL-level reporting of which pages were used for training, alongside search metrics. Cloudflare has published no compliance list of its own, and an operator could satisfy a condition through a private arrangement it never discloses — but on the public record, a dual-purpose crawler reaching a Cloudflare site that disallows training is doing so without publicly clearing the bar Cloudflare set for it.

That fourth condition now has teeth on new domains. Starting September 15, 2026, Cloudflare's own default blocks Training and Agent crawlers on any page that displays ads, leaving Search allowed — but Cloudflare's post also states that "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training," because those crawlers carry search and training under one user-agent. A site that takes the default therefore loses Google's crawler on every ad-bearing page — the exact harm the fourth condition requires an operator to show doesn't happen. The default applies only to new domains onboarding to Cloudflare; an existing domain keeps its current configuration, and owners who don't want it can switch it off in Security settings any time before September 15. What "pages that display ads" resolves to in practice, and whether the change also reaches existing free-plan sites, is not addressed in Cloudflare's post.

The user-agent string this whole policy runs on is itself forgeable. GreyNoise found 824 addresses across 795 networks sending traffic with forged AI-crawler names between July 28 and August 23, 2026 — including 263,849 sessions claiming to be Google-Extended, a token Google's own documentation says has no separate user-agent string at all, meaning none of those sessions could have been genuine. The forged traffic was scanning for credential and configuration files (.env, .aws/credentials, .git/config). Verify the connecting IP against each company's published crawler ranges before trusting what a User-Agent header claims.

**Should I block AI crawlers in robots.txt?** The decision is per bot, not blanket. Blocking a training crawler withholds content from future models; blocking an index or retrieval crawler removes you from the answers engines give today.

**How do I decide which AI crawlers to allow?** IAB Australia's guidance sorts every crawler category into one of four verdicts — allow, allow with conditions, require licensing, or block. Re-check the user-agent strings periodically: companies rename crawlers as products rebrand, and a rule pointed at yesterday's string silently stops applying.

**Do AI crawlers actually respect robots.txt blocks?** Not reliably. TollBit found about 15% of AI page fetchers it tracked reached European publisher URLs marked disallowed in the first half of 2026, with OpenAI's ChatGPT-User among the bots most often crossing blocks aimed specifically at it — a block is worth monitoring, not just setting.

**Does Cloudflare change my robots.txt for AI crawlers now?** Only if you switch it on. Bot Preference Sync, rolling out from August 21, 2026 across every plan, keeps robots.txt in step with an AI-crawler policy set in the Cloudflare dashboard. It prepends a managed block and maintains any existing Disallow directives rather than replacing the file, and it is opt-in and reversible at any time. It also conditions continued access for dual-purpose search-and-training crawlers on public disclosure rather than a self-declared promise.

**Does any major AI crawler actually meet Cloudflare's four disclosure conditions?** Not on the public record as of August 27, 2026. Google meets three of four, OpenAI two, Anthropic none it documents — every operator fails the requirement to report URL-level training use alongside search metrics, which makes the no-training preference a block rather than a standard for now.

**Does Cloudflare's September 15 default also block Google's search crawler?** Yes, on pages that display ads, if the default is taken. Cloudflare says its own multi-purpose crawlers — Googlebot, Applebot and BingBot — are blocked by customers who block Training, because they carry both behaviors under one user-agent. The default applies to new domains onboarding to Cloudflare; existing domains keep their current settings, and owners can opt out in Security settings any time before September 15.

**Can I trust the User-Agent string to identify a real AI crawler?** No. GreyNoise found 824 addresses forging AI-crawler names, including sessions claiming to be Google-Extended — a token with no user-agent string at all — while scanning for credential files. Verify the connecting IP against each company's published crawler ranges instead of trusting the header.

Verified 2026-08-31. References: AI crawlers — glossary (https://anythingengineoptimization.com/glossary/ai-crawlers/) · Google robots.txt documentation (https://developers.google.com/search/docs/crawling-indexing/robots/intro) · NotebookLM crawler renamed (wire) (https://anythingengineoptimization.com/item/2026-07-20-google-renames-notebooklm-crawler-user-agent-to-google-gemininotebook/) · IAB Australia sorts AI crawlers into four access verdicts (wire) (https://anythingengineoptimization.com/item/2026-08-01-iab-australia-sorts-ai-crawlers-into-four-access-verdicts/) · ChatGPT ads expand to Europe as crawler bypasses blocks (wire) (https://anythingengineoptimization.com/item/2026-08-16-chatgpt-ads-expand-to-europe-as-crawler-bypasses-blocks/) · Cloudflare syncs robots.txt, sets four conditions for crawlers that also train (wire) (https://anythingengineoptimization.com/item/2026-08-23-cloudflare-rewrites-robots-txt-gates-access-on-disclosure/) · No major AI crawler publicly meets all four of Cloudflare's conditions (wire) (https://anythingengineoptimization.com/item/2026-08-27-no-major-ai-crawler-publicly-meets-all-four-of-cloudflare-s-conditions/) · Cloudflare's September 15 default blocks Googlebot on ad-bearing pages (wire) (https://anythingengineoptimization.com/item/2026-08-30-cloudflare-s-september-15-default-blocks-googlebot-on-ad-bearing-pages/) · GreyNoise finds forged ClaudeBot and GPTBot traffic hunting for credentials (wire) (https://anythingengineoptimization.com/item/2026-08-30-greynoise-finds-forged-claudebot-and-gptbot-traffic-hunting-for/)

## 2. Serve real HTML links — AI crawlers won't run your JavaScript

Mechanism: Server-rendered HTML anchor links, not JS-injected navigation

GPTBot, ClaudeBot and Bingbot do not execute JavaScript to discover links. A 41-day test on a 2,400-page site found all three crawlers found zero of the pages linked only through JavaScript, while even Googlebot — which does render JS — reached just 2% of them (48% went to Google's separate, JS-capable GoogleOther crawler). The fix is mechanical, not architectural: after the site's JavaScript links were converted to plain HTML anchors, GPTBot recovered 250 of the previously unreachable pages within 48 hours.

An internal-link audit judged only against Googlebot's crawl behavior is not sufficient for AI visibility — an AI crawler can be stricter than Google's own, silently excluding JavaScript-linked pages from citation eligibility even while the same pages rank fine in Search.

**Can GPTBot or ClaudeBot follow JavaScript-rendered links?** No. A 41-day test found GPTBot, ClaudeBot and Bingbot discovered zero pages that were linked only through JavaScript, while even Googlebot reached just 2% of them.

**How do I fix pages AI crawlers can't discover because of JavaScript links?** Convert JavaScript-injected links to plain HTML anchors. In the same test, GPTBot recovered 250 previously unreachable pages within 48 hours of the fix.

Verified 2026-08-20. References: 41-day test: AI crawlers miss JavaScript-only links (wire) (https://anythingengineoptimization.com/item/2026-08-19-41-day-test-finds-ai-crawlers-miss-javascript-only-links-entirely/)

## 3. Put the answer where a machine can lift it

Mechanism: Question-shaped headings, Q&A blocks

Answer engines quote passages, not pages. Content that answers a searchable question in a self-contained, quotable block — near the top, in plain declarative language — is retrievable and citable in a way that the same facts diffused across twelve paragraphs are not. Question-shaped headings and explicit Q&A blocks make the mapping from query to passage trivial. A Pillarbase analysis of 15.7 million Google AI Mode citations puts numbers to that shape: the passages Google reuses most average 117 words, answer in the first sentence about 80% of the time, and are self-contained about 85% of the time.

**How do I get my content quoted by AI search?** Answer engines quote passages, not pages: put a self-contained answer in plain declarative language near the top, under a question-shaped heading. A Pillarbase analysis of 15.7 million Google AI Mode citations found the most-reused passages answer in the first sentence about 80% of the time.

**How long should an answer passage be?** The passages Google's AI Mode reuses most average 117 words and are self-contained about 85% of the time, per Pillarbase's analysis of 15.7 million citations.

Verified 2026-07-29. References: RAG — glossary (why retrieval units matter) (https://anythingengineoptimization.com/glossary/rag/) · AEO — glossary (https://anythingengineoptimization.com/glossary/aeo/) · Pillarbase: 15.7M AI Mode citations show what Google reuses (wire) (https://anythingengineoptimization.com/item/2026-07-29-pillarbase-study-15-7-million-ai-mode-citations-show-what-google/)

## 4. Ship structured data for what the page actually is

Mechanism: Schema.org JSON-LD

Schema.org markup — NewsArticle for news, FAQPage for question-and-answer content, DefinedTerm for definitions, Organization and Person for who stands behind it — is machine-readable self-description, and it is cheap. Mark up what the page genuinely is; structured data that misdescribes a page is a trust liability, not an optimization. That framing now has a direct test behind it: an Ahrefs analysis of 1,885 pages found adding schema markup produced no measurable increase in AI citations after 30 days — treat it as descriptive infrastructure, not a lever for citation volume.

**Which schema types should I use for AI search?** Mark up what the page genuinely is: NewsArticle for news, FAQPage for question-and-answer content, DefinedTerm for definitions, and Organization and Person for who stands behind it.

**Can structured data hurt my site?** Yes. Structured data that misdescribes a page is a trust liability, not an optimization — and FAQ markup whose text a reader cannot see on the page violates Google's policy outright.

**Does adding schema markup increase AI citations?** Not measurably. An Ahrefs test of 1,885 pages found no increase in AI citations 30 days after adding schema markup — mark up what the page genuinely is for its own sake, not as a citation lever.

Verified 2026-08-18. References: Google structured data documentation (https://developers.google.com/search/docs/appearance/structured-data) · Schema.org (https://schema.org/) · Ahrefs study: llms.txt files mostly unread by AI crawlers (wire) (https://anythingengineoptimization.com/item/2026-08-18-ahrefs-study-llms-txt-files-mostly-unread-by-ai-crawlers/)

## 5. Be unambiguous about your entities

Mechanism: Canonical hub pages, sameAs

Knowledge graphs and answer engines resolve content to entities — companies, products, people. Give each entity you own a canonical hub page, keep names consistent, and declare external identities with sameAs links to official sites and reference pages. Ambiguity about which thing you mean is a retrieval failure you inflicted on yourself.

**What is entity optimization?** Knowledge graphs and answer engines resolve content to entities — companies, products, people. Entity work means giving each entity you own a canonical hub page and keeping its name consistent everywhere it appears.

**How do I tell AI engines which company my brand is?** Declare external identities with sameAs links to official sites and reference pages. Ambiguity about which thing you mean is a retrieval failure you inflicted on yourself.

Verified 2026-07-25. References: The record — this site's entity hubs (https://anythingengineoptimization.com/entity/) · schema.org/sameAs (https://schema.org/sameAs)

## 6. Give readers a one-click way to name you a Preferred Source

Mechanism: Preferred Sources embed button

Google's Preferred Sources feature lets a reader mark a site as preferred across Search, AI Overviews and AI Mode, and Google says a preferred pick makes that site surface more often in all three. As of August 2026, publishers can embed a one-click "Add to Preferred Sources" button directly on their own pages rather than relying on readers to find the pick buried in search settings, and Google has now extended the same mechanism into AI Mode's new developing-topic link carousel. Adoption is real and growing: unique sources selected as Preferred rose from over 345,000 at the feature's May 2026 launch to over 600,000 by August.

**What is Google's Preferred Sources feature?** A reader-facing pick that Google says makes a site surface more often in Top Stories, AI Overviews and AI Mode. Publishers can now embed a one-click button on their own pages that adds the site as a Preferred Source.

**Is Google's Preferred Sources feature gaining adoption?** Yes — unique sources picked as Preferred rose from over 345,000 at the feature's May 2026 launch to over 600,000 by August 2026, and Google has extended the mechanism into AI Mode's developing-topic link carousel.

Verified 2026-08-26. References: Google: personalize your search and Discover with preferred sources (https://blog.google/products-and-platforms/products/search/personalize-search-discover-news/) · Google lets publishers embed a Preferred Sources button (wire) (https://anythingengineoptimization.com/item/2026-08-20-google-lets-publishers-embed-a-preferred-sources-button/) · Google's AI Mode gets link carousels for developing topics (wire) (https://anythingengineoptimization.com/item/2026-08-25-google-s-ai-mode-gets-link-carousels-for-developing-topics/)

## 7. Date everything, visibly and in markup

Mechanism: Visible dates, honest dateModified

Answer engines weigh freshness and increasingly surface dates alongside citations. Publish visible dates, keep dateModified truthful in markup, and never launder old content as new — an engine that catches a laundered date has a durable reason to distrust the domain. Honest dating is both a trust signal and a defense. The effect is measurable: a controlled Otterly experiment adding the current year to 11 page titles lifted their AI-search citations 61% — with the obvious corollary that the year must be true, and the content actually current.

A separate Seer Interactive analysis of 47,097 citations across ChatGPT, Gemini and Perplexity found the same pattern at scale: 75% of cited pages had been updated within the past year, and among 4,124 pages with both dates recorded, 72% looked fresh by last-modified date against just 42% by original publish date — a genuinely refreshed old page outcited a newly published one more often than not.

The same signal predicts traditional Google ranking, not just AI citation: a Growth Memo analysis of nearly 60,000 search results pages found a listicle with an old or unreadable publish date had 56% lower odds of reaching the top 3 — the single strongest factor measured, ahead of author bios, schema markup and word count.

**Does updating old content help AI visibility?** Seer Interactive's analysis of 47,097 citations found 72% of cited pages looked fresh by last-modified date against just 42% by original publish date — a genuinely refreshed old page outcited a newly published one more often than not.

**Should I put the current year in my page titles?** Only when it is true. Otterly reports a controlled experiment where adding the year to 11 page titles lifted AI-search citations 61%; that is the vendor's own figure, not one this wire has verified.

**Does a stale publish date hurt traditional Google ranking too, not just AI citation?** Yes — a Growth Memo analysis of nearly 60,000 search results pages found an old or unreadable publish date was the strongest predictor of a listicle failing to reach the Google top 3, ahead of author bios, schema markup and word count.

Verified 2026-08-31. References: This site's editorial standards (https://anythingengineoptimization.com/standards/) · Otterly title-year experiment — vendor blog, not wire-verified (https://otterly.ai/blog/) · Seer: content recency's impact on AI visibility (wire) (https://anythingengineoptimization.com/item/2026-08-06-study-ties-ai-citation-likelihood-to-how-recently-a-page-was-updated/) · Growth Memo: freshness and Google listicle ranking (wire) (https://anythingengineoptimization.com/item/2026-08-31-freshness-outranks-word-count-in-google-s-listicle-rankings/)

## 8. Know the AI Overviews opt-out — and what it doesn't reach

Mechanism: Search Console → Settings → Search generative AI control

Search Console carries a Search generative AI control, under Settings, that is the only lever separating appearance in AI Overviews, AI Mode and Discover's generative features from ordinary Search — unlike noindex, nosnippet or Google-Extended, which each take something else with them. Google says the control "isn't used as a ranking or inclusion signal affecting other parts of Search" and takes one to two days to apply. It rolled out to all sites worldwide as of August 31, 2026, but Google's own AI-features guidance for site owners still does not mention it at all, naming nosnippet 18 times instead.

The control covers appearance, not training — Google's documentation sends publishers to Google-Extended for that, so it answers only one of the three uses (training, fine-tuning, grounding) the UK CMA's June 3, 2026 Publisher Conduct Requirement ordered Google to let publishers separately opt out of. There's a further caveat for news publishers specifically: Google renders Top Stories carousels inside AI Overviews on roughly 15.5% of US news searches, per an unconfirmed NewzDash analysis, so opting a news site out of AI Overviews may cost the Top Stories placement rendered within them, not just the summary.

**How do I opt a site out of Google's AI Overviews without leaving Search entirely?** Search Console's Settings carry a Search generative AI control that removes a site from AI Overviews, AI Mode and Discover's generative features while Google says it isn't used as a ranking or inclusion signal elsewhere in Search. It rolled out to all sites worldwide as of August 31, 2026, and takes one to two days to apply.

**Does opting out of AI Overviews also stop Google from training on my content?** No. The control covers appearance only; Google's documentation points to Google-Extended for training, fine-tuning and grounding — the other two uses the UK CMA's June 2026 order requires publishers be able to opt out of separately.

**Is there a downside to opting a news site out of AI Overviews?** Possibly. Google places Top Stories carousels inside AI Overviews on roughly 15.5% of US news searches per an unconfirmed NewzDash analysis, so excluding a site from AI Overviews may also remove it from that carousel placement.

Verified 2026-08-31. References: Google Search Console Help: Search generative AI control (https://support.google.com/webmasters/answer/16908024) · Google: AI features and your website (https://developers.google.com/search/docs/appearance/ai-features) · What Google's AI Overviews opt-out does, and two things it does not (wire) (https://anythingengineoptimization.com/item/2026-08-27-what-google-s-ai-overviews-opt-out-does-and-two-things-it-does-not/) · Search Console's AI Overviews opt-out control rolls out to all sites (wire) (https://anythingengineoptimization.com/item/2026-08-31-search-console-s-ai-overviews-opt-out-control-rolls-out-to-all-sites/)

# Level 2 — Measurement

## 9. Wire the freshness signals: sitemaps, IndexNow, WebSub

Mechanism: Sitemaps, IndexNow, WebSub

Engines can only be as current as their discovery. Maintain an XML sitemap (a news sitemap if you publish news), ping IndexNow on every publish for the engines that accept it, and declare a WebSub hub on your feeds so subscribers are pushed rather than left to poll. None of this improves content; all of it shortens the gap between publishing and being retrievable.

**How do I get new pages into AI answers faster?** Maintain an XML sitemap — a news sitemap if you publish news — ping IndexNow on every publish for the engines that accept it, and declare a WebSub hub on your feeds so subscribers are pushed rather than left to poll.

**Does IndexNow help with AI search?** It shortens the gap between publishing and being retrievable for the engines that accept it. None of this plumbing improves content; all of it speeds discovery.

Verified 2026-07-25. References: IndexNow (https://www.indexnow.org/) · WebSub (W3C) (https://www.w3.org/TR/websub/) · Google sitemap documentation (https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview)

## 10. Get your first-party AI numbers where they exist

Mechanism: Search Console, Bing Webmaster Tools

The engines have started shipping first-party AI-surface data: Search Console added generative-AI performance reports, and Bing Webmaster Tools previews citation-share and AI-visibility insights. First-party beats scraped estimates wherever both exist — verify your site in both consoles and take the numbers the engines will actually give you. The surface keeps widening: Microsoft Clarity now splits AI citation data into branded and non-branded views, and Search Console's platform properties extend first-party measurement to social and video accounts. Clarity's Bot Analytics dashboard added an AI Scrape-to-Referral Ratio card on August 13, 2026, ranking which AI crawlers send return visitors against those that mostly extract content without driving traffic back — though Clarity itself cautions a high ratio doesn't by itself signal failing AI visibility, since many AI platforms strip referrer data and likely undercount real referrals.

That first-party data matters because your own analytics can't be trusted to sort AI traffic correctly. A nine-month GA4 tracking study of 51,200 AI Overview citation clicks found an average of 22.4% of that traffic was misattributed to the Direct channel instead of Organic Search — worst in May 2026 at 29.3% — so a site's GA4 organic-search numbers alone understate AI Overview's real reach without cross-checking against Search Console.

**Where can I see my own AI search performance data?** Search Console added generative-AI performance reports and Bing Webmaster Tools previews citation-share and AI-visibility insights. Verify your site in both consoles: first-party beats scraped estimates wherever both exist.

**Can I see which AI answers cite my site?** Bing Webmaster Tools previews citation-level insights, and Microsoft Clarity now splits AI citation data into branded and non-branded views. Search Console reports impressions and clicks from AI surfaces, but not which pages an answer cited.

**Does Google Analytics correctly attribute AI Overview traffic?** Not reliably. A nine-month GA4 tracking study found an average of 22.4% of AI Overview citation-click sessions were misattributed to the Direct channel instead of Organic Search — cross-check against Search Console's generative-AI reports rather than trusting GA4's channel breakdown alone.

**Can I tell whether an AI crawler that scrapes my content ever sends traffic back?** Microsoft Clarity's Bot Analytics dashboard added an AI Scrape-to-Referral Ratio card in August 2026 that ranks AI sources by crawl volume against referral visits — though Clarity cautions a high ratio doesn't by itself mean failing AI visibility, since many AI platforms strip referrer data and likely undercount real referrals.

Verified 2026-08-19. References: Search Console adds generative-AI reports (wire) (https://anythingengineoptimization.com/item/2026-07-20-search-console-adds-generative-ai-performance-reports/) · Bing WMT previews AI visibility insights (wire) (https://anythingengineoptimization.com/item/2026-07-20-bing-webmaster-tools-previews-ai-visibility-insights-including/) · Clarity splits AI citations branded/non-branded (wire) (https://anythingengineoptimization.com/item/2026-08-03-microsoft-clarity-splits-ai-citation-data-into-branded-and-non/) · Platform properties roll out globally (wire) (https://anythingengineoptimization.com/item/2026-08-01-search-console-platform-properties-roll-out-globally-for-social-and/) · Study finds AI Overview traffic misattributed 22% of time (wire) (https://anythingengineoptimization.com/item/2026-08-18-study-finds-ai-overview-traffic-misattributed-22-of-time/) · Microsoft Clarity adds an AI scrape-to-referral ratio (wire) (https://anythingengineoptimization.com/item/2026-08-19-microsoft-clarity-adds-an-ai-scrape-to-referral-ratio/)

## 11. Track citation share, and expect it to move when models do

Mechanism: Per-engine citation tracking

Citation share — the fraction of an engine's citations your domain earns — is the discipline's emerging visibility metric. Measure it per engine, because the same content earns different shares in different engines; and treat every model swap inside an engine as a potential discontinuity in your numbers, because the model doing the citing just changed. Not every citation names the source, either: a Writesonic analysis of roughly 16 million brand appearances found engines omit the brand name anywhere from 19% (Microsoft Copilot) to 52% (Perplexity) of the time, so a citation-share number built only on named mentions will undercount actual retrieval. And citation volume alone doesn't establish category leadership: a Kevin Indig analysis of Semrush's ChatGPT citation data found the most-cited domain matched the most-mentioned brand only 20.8% of the time across 1,094 categories.

Only 16% of brands track this at all, and the vendors selling the other 84% a dashboard often disagree on the same brand's numbers. The IAB's "Measuring Visibility in the AI Era" framework gives buyers a checklist for that gap: don't trust a citation-share number until the vendor discloses platform coverage, prompt-library construction, and query volume — the IAB sets 50 queries as the floor before a program counts as more than exploratory — and separates hallucination rate from plain factual-accuracy error.

A large-scale drift study puts numbers to "expect it to move": SISTRIX re-sampled 82,619 prompts weekly for 17 weeks and found Google replaces 56% of its cited sources every week, ChatGPT as much as 74%. The churn isn't uniform — 86% of prompts keep a stable core of a few domains while everything outside it rotates at 89% per week — and Google AI Overviews and AI Mode cite different domains 83% of the time, so track the two Google surfaces separately rather than as one number. For a publisher, a single news-article citation behaves less like an asset that accrues than a placement that expires: only 1.4% of cited news articles held their spot across all 17 weeks, against 43% of brand queries that kept the brand's own domain present throughout even as its co-citations kept turning over.

The model-swap risk above isn't hypothetical. Two days after Google rolled Gemini 3.8 Flash out to AI Mode, SEO practitioners found citations vanishing from top-of-funnel queries entirely, and Google's Search VP confirmed it as a bug rather than a design change. A citation tracker that only samples periodically reads an incident like that as a sudden collapse in visibility, indistinguishable at a glance from an actual ranking loss — check for an engine-side incident before concluding a citation drop reflects on your content.

**How should I track citation share?** Measure it per engine, because the same content earns different shares in different engines, and treat every model swap inside an engine as a potential discontinuity in your numbers.

**Why do AI visibility tools disagree about the same brand?** The IAB's framework says not to trust a citation-share number until the vendor discloses platform coverage, prompt-library construction and query volume — with 50 queries as the floor before a program counts as more than exploratory.

**Does a high citation count mean my brand is winning the category?** Not reliably. An analysis of Semrush's ChatGPT citation data found the most-cited domain matched the most-mentioned brand only 20.8% of the time across 1,094 categories, and Writesonic found engines omit the source brand 19% to 52% of the time depending on the engine.

**How much does AI citation drift week to week?** A lot, structurally. SISTRIX found Google replaces 56% of its cited sources weekly and ChatGPT as much as 74%, across 82,619 prompts re-sampled for 17 weeks — but 86% of prompts keep a stable core of a few domains while everything outside it rotates at 89% per week.

**Should I track Google AI Overviews and AI Mode as one citation number?** No. The same SISTRIX study found the two Google surfaces cite different domains 83% of the time, so a citation-share figure measured on one is not a reading of the other.

**Can a citation drop be a bug rather than a real visibility loss?** Yes. When Google rolled Gemini 3.8 Flash out to AI Mode on September 2, 2026, AI Mode responses briefly stopped showing citations on some queries; Google's Search VP Robby Stein confirmed it was a bug two days later. Check for an engine-side incident before concluding a citation drop reflects your content.

Verified 2026-09-08. References: Citation share — glossary (https://anythingengineoptimization.com/glossary/citation-share/) · Ahrefs' four-surface citation rankings (wire) (https://anythingengineoptimization.com/item/2026-07-23-ahrefs-ranks-the-50-most-cited-domains-across-four-ai-surfaces/) · Search now runs Gemini 3.5 Flash-Lite (wire) (https://anythingengineoptimization.com/item/2026-07-23-google-search-now-runs-the-latest-gemini-3-5-flash-lite/) · Writesonic: AI engines often skip naming source brands (wire) (https://anythingengineoptimization.com/item/2026-07-29-writesonic-study-finds-ai-engines-often-skip-naming-source-brands/) · Analysis: most AI search demand has no clear ChatGPT category leader (wire) (https://anythingengineoptimization.com/item/2026-07-29-analysis-most-ai-search-demand-still-has-no-clear-chatgpt-category/) · IAB sets first measurement standard for AI-visibility tools (wire) (https://anythingengineoptimization.com/item/2026-08-04-iab-sets-first-measurement-standard-for-ai-visibility-tracking-tools/) · SISTRIX: ChatGPT replaces 74% of its cited sources every week (wire) (https://anythingengineoptimization.com/item/2026-09-01-sistrix-chatgpt-replaces-74-of-its-cited-sources-every-week/) · Google confirms AI Mode citation bug in Gemini 3.8 Flash (wire) (https://anythingengineoptimization.com/item/2026-09-04-google-confirms-ai-mode-citation-bug-in-gemini-3-8-flash/)

## 12. Watch the engines' documentation, not just their announcements

Mechanism: Scheduled documentation diffing

The engines' most consequential changes ship quietly — help-page edits, crawler renames, policy language — hours or days before anyone announces them, if anyone ever does. Watch the primary documentation pages that govern your visibility and diff them on a schedule. The announcements are marketing; the documentation is policy.

**How do I keep up with changes to AI search?** The engines' most consequential changes ship quietly — help-page edits, crawler renames, policy language. Watch the primary documentation pages that govern your visibility and diff them on a schedule.

**Are an engine's announcements or its documentation more reliable?** The documentation. Google's AI Overviews help page spelled out how signed-in search history trains its generative models with no announcement at all.

Verified 2026-07-25. References: Help page spells out AI Overviews training (wire exclusive) (https://anythingengineoptimization.com/item/2026-07-24-google-help-page-spells-out-how-signed-in-history-trains-ai-overviews/) · Sources & method — how this site watches (https://anythingengineoptimization.com/sources/)

# Level 3 — Bets

## 13. Publish llms.txt — as a bet on every surface except Google

Mechanism: llms.txt, markdown twins

llms.txt is a proposed standard: a machine-readable index of what your site offers, with llms-full.txt carrying the content itself for systems that want the corpus. Google has now answered its half of the open question on the record: its AI-features documentation states that machine-readable files, AI text files, and markup aren't needed to appear in Search or its AI features. An Ahrefs analysis spanning roughly 15 million data points adds evidence for the file's uncertain value even on the surfaces it's meant for: AI crawlers never read 97% of sites' llms.txt files at all. A Common Crawl analysis of 584,107 llms.txt files in its July 2026 archive suggests one reason why: 68% were produced by templates or SEO plugins rather than hand-authored — Wix alone accounts for 41% of the corpus — and only about half fully follow the spec's structure of a title, summary line, and link sections. So publish llms.txt for what it is — a near-zero-cost bet on the standalone assistants and agents beyond Google, deliberately authored rather than left to a plugin default — never as a Google lever, and never instead of the schema and crawlability every engine actually reads.

The same bet now extends past the index file: publishers have begun serving agent-readable markdown versions of pages outright — Time went as far as selling ads inside its agent-readable markdown — and for markdown-native sites, page-level .md twins cost nearly nothing. This site serves one for every item and reference page.

Google's John Mueller has pushed back on the premise itself: on his own test sites, "the only crawlers who claim to accept markdown are SEO tools" — not the major AI bots. His advice before building a markdown version on the assumption AI crawlers want it: log the accept header on incoming requests and check whether anything is actually asking for one.

That markdown surface is not exempt from trust rules. Perplexity has since blocked Time's markdown ads outright, calling the sponsored FAQ content deceptive and applying a trust-score downgrade to publishers that run it — an agent-facing page answers to the same trust standard as the human-facing one.

The spec itself got its first revision since launch on August 10: llms.txt version 2 adds rel="alternate" type="text/markdown" and rel="describedby" link relations so agents can discover a page's markdown twin and the llms.txt file governing it — formalizing, as a standard, the kind of Link-header discovery a markdown-twin setup should already expose.

**Does Google read llms.txt?** No. Google's AI-features documentation states that machine-readable files, AI text files, and markup aren't needed to appear in Search or its AI features.

**Is llms.txt worth publishing?** As a near-zero-cost bet on the standalone assistants and agents beyond Google — never as a Google lever, and never instead of the schema and crawlability every engine actually reads.

**Do AI crawlers actually read llms.txt files?** Rarely. An Ahrefs analysis of roughly 15 million data points found AI crawlers never read 97% of sites' llms.txt files at all, even where the file exists.

**Are most llms.txt files hand-authored?** No. Common Crawl's analysis of 584,107 files in its July 2026 archive found 68% were produced by templates or SEO plugins — Wix alone accounts for 41% of the corpus — with only about half fully following the spec's structure.

**Are agent-readable markdown pages held to the same trust rules as regular pages?** Yes. Perplexity blocked Time's markdown ads as deceptive and downgraded the publisher's trust score, treating the agent-facing page the same as the visible one.

**Does llms.txt support linking to a page's markdown version?** As of version 2 — the spec's first revision since it launched in 2024 — yes: rel="alternate" type="text/markdown" and rel="describedby" link relations let agents find a page's markdown twin and the llms.txt file that covers it.

**Do AI crawlers actually request markdown versions of pages?** Not the major ones, per Google's John Mueller: on his own test sites, only SEO tools claimed to accept markdown. He recommends logging accept headers before building a markdown version on the assumption AI crawlers want it.

Verified 2026-09-02. References: Google: AI features and your website (https://developers.google.com/search/docs/appearance/ai-features) · llmstxt.org — the proposal (https://llmstxt.org/) · llms.txt — the record (https://anythingengineoptimization.com/entity/llms-txt/) · Ahrefs ships a free llms.txt generator (wire) (https://anythingengineoptimization.com/item/2026-07-24-ahrefs-launches-a-free-llms-txt-generator-tool/) · Time sells ads inside agent-readable markdown (wire) (https://anythingengineoptimization.com/item/2026-08-01-time-starts-selling-ads-inside-its-ai-agent-readable-markdown-pages/) · Perplexity blocks Time's markdown ads (wire) (https://anythingengineoptimization.com/item/2026-08-11-perplexity-blocks-time-s-markdown-ads-from-its-ai-results/) · llms.txt spec adds link relations for Markdown discovery (wire) (https://anythingengineoptimization.com/item/2026-08-17-llms-txt-spec-adds-link-relations-for-markdown-discovery/) · Ahrefs study: llms.txt files mostly unread by AI crawlers (wire) (https://anythingengineoptimization.com/item/2026-08-18-ahrefs-study-llms-txt-files-mostly-unread-by-ai-crawlers/) · Google's Mueller: AI crawlers aren't requesting markdown (wire) (https://anythingengineoptimization.com/item/2026-08-31-google-s-mueller-ai-crawlers-aren-t-requesting-markdown/) · Common Crawl: two-thirds of llms.txt files are templates (wire) (https://anythingengineoptimization.com/item/2026-08-31-common-crawl-two-thirds-of-llms-txt-files-are-templates/)

Canonical: https://anythingengineoptimization.com/best-practices/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/