Reference
Current Best Practices
What the record supports, practice by practice. Every entry carries the date it was last verified and the references behind it — because in this field, an undated best-practices page is fiction within a quarter.
Last updated September 8, 2026.
Maintained by the editor; practices are re-verified against the record and re-dated, revised, or retired. This page sells nothing — the wire offers no optimization services.
Level 1 Foundations
Be readable, be quotable, be unambiguous, be honestly dated. Nothing below this line substitutes for it — 76% of the passages Google's AI Mode reuses most come from pages already ranking first organically.
-
Decide your AI crawler policy deliberately — and per bot
robots.txt, per user-agent
AI companies run separate crawlers for model training, search indexing, and live retrieval, and the trade-offs differ: blocking a training bot withholds content from future models, while blocking an index or retrieval bot removes you from today's answers. Set robots.txt per user-agent with that distinction in mind, and re-check the strings periodically — companies rename crawlers as products rebrand, and a rule pointed at yesterday's string silently stops applying. IAB Australia's Bots and Crawler Guidance formalizes the same call into four verdicts per crawler category — allow, allow with conditions, require licensing, or block — a workable checklist for making the decision deliberately rather than defaulting to one blanket rule.
A robots.txt block is a request, not a lock. TollBit found roughly 15% of AI page fetchers it tracked in the first half of 2026 reached European publisher URLs marked disallowed, with OpenAI's ChatGPT-User among the bots most often crossing blocks aimed specifically at it — treat a block as a policy statement worth logging compliance against, not a guarantee.
For Cloudflare-proxied sites, part of that policy is now set in a dashboard rather than hand-edited into the file. Bot Preference Sync, rolling out across every plan from free to Enterprise starting August 21, 2026, keeps robots.txt in step with an AI-crawler policy configured in the dashboard. It prepends a managed block rather than replacing the file — Cloudflare says any existing Disallow directives are maintained — and the sync is opt-in and can be switched off at any time. It also changes what a no-training preference actually buys: a crawler that does both search and training keeps access to a site marked no-training only if it meets four conditions — honoring the preference, offering an AI-summary opt-out, reporting URL-level training use alongside search metrics, and publicly showing the block doesn't hurt search visibility. Continued access now turns on public disclosure, not a crawler's self-declared promise.
Checked against Google, OpenAI and Anthropic's own documentation on August 27, 2026, none of the three publicly meets all four conditions — Google clears three, OpenAI two, Anthropic none. The condition every operator fails is the third: URL-level reporting of which pages were used for training, alongside search metrics. Cloudflare has published no compliance list of its own, and an operator could satisfy a condition through a private arrangement it never discloses — but on the public record, a dual-purpose crawler reaching a Cloudflare site that disallows training is doing so without publicly clearing the bar Cloudflare set for it.
That fourth condition now has teeth on new domains. Starting September 15, 2026, Cloudflare's own default blocks Training and Agent crawlers on any page that displays ads, leaving Search allowed — but Cloudflare's post also states that "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training," because those crawlers carry search and training under one user-agent. A site that takes the default therefore loses Google's crawler on every ad-bearing page — the exact harm the fourth condition requires an operator to show doesn't happen. The default applies only to new domains onboarding to Cloudflare; an existing domain keeps its current configuration, and owners who don't want it can switch it off in Security settings any time before September 15. What "pages that display ads" resolves to in practice, and whether the change also reaches existing free-plan sites, is not addressed in Cloudflare's post.
The user-agent string this whole policy runs on is itself forgeable. GreyNoise found 824 addresses across 795 networks sending traffic with forged AI-crawler names between July 28 and August 23, 2026 — including 263,849 sessions claiming to be Google-Extended, a token Google's own documentation says has no separate user-agent string at all, meaning none of those sessions could have been genuine. The forged traffic was scanning for credential and configuration files (.env, .aws/credentials, .git/config). Verify the connecting IP against each company's published crawler ranges before trusting what a User-Agent header claims.
-
Serve real HTML links — AI crawlers won't run your JavaScript
Server-rendered HTML anchor links, not JS-injected navigation
GPTBot, ClaudeBot and Bingbot do not execute JavaScript to discover links. A 41-day test on a 2,400-page site found all three crawlers found zero of the pages linked only through JavaScript, while even Googlebot — which does render JS — reached just 2% of them (48% went to Google's separate, JS-capable GoogleOther crawler). The fix is mechanical, not architectural: after the site's JavaScript links were converted to plain HTML anchors, GPTBot recovered 250 of the previously unreachable pages within 48 hours.
An internal-link audit judged only against Googlebot's crawl behavior is not sufficient for AI visibility — an AI crawler can be stricter than Google's own, silently excluding JavaScript-linked pages from citation eligibility even while the same pages rank fine in Search.
-
Put the answer where a machine can lift it
Question-shaped headings, Q&A blocks
Answer engines quote passages, not pages. Content that answers a searchable question in a self-contained, quotable block — near the top, in plain declarative language — is retrievable and citable in a way that the same facts diffused across twelve paragraphs are not. Question-shaped headings and explicit Q&A blocks make the mapping from query to passage trivial. A Pillarbase analysis of 15.7 million Google AI Mode citations puts numbers to that shape: the passages Google reuses most average 117 words, answer in the first sentence about 80% of the time, and are self-contained about 85% of the time.
-
Ship structured data for what the page actually is
Schema.org JSON-LD
Schema.org markup — NewsArticle for news, FAQPage for question-and-answer content, DefinedTerm for definitions, Organization and Person for who stands behind it — is machine-readable self-description, and it is cheap. Mark up what the page genuinely is; structured data that misdescribes a page is a trust liability, not an optimization. That framing now has a direct test behind it: an Ahrefs analysis of 1,885 pages found adding schema markup produced no measurable increase in AI citations after 30 days — treat it as descriptive infrastructure, not a lever for citation volume.
-
Be unambiguous about your entities
Canonical hub pages, sameAs
Knowledge graphs and answer engines resolve content to entities — companies, products, people. Give each entity you own a canonical hub page, keep names consistent, and declare external identities with sameAs links to official sites and reference pages. Ambiguity about which thing you mean is a retrieval failure you inflicted on yourself.
-
Give readers a one-click way to name you a Preferred Source
Preferred Sources embed button
Google's Preferred Sources feature lets a reader mark a site as preferred across Search, AI Overviews and AI Mode, and Google says a preferred pick makes that site surface more often in all three. As of August 2026, publishers can embed a one-click "Add to Preferred Sources" button directly on their own pages rather than relying on readers to find the pick buried in search settings, and Google has now extended the same mechanism into AI Mode's new developing-topic link carousel. Adoption is real and growing: unique sources selected as Preferred rose from over 345,000 at the feature's May 2026 launch to over 600,000 by August.
-
Date everything, visibly and in markup
Visible dates, honest dateModified
Answer engines weigh freshness and increasingly surface dates alongside citations. Publish visible dates, keep dateModified truthful in markup, and never launder old content as new — an engine that catches a laundered date has a durable reason to distrust the domain. Honest dating is both a trust signal and a defense. The effect is measurable: a controlled Otterly experiment adding the current year to 11 page titles lifted their AI-search citations 61% — with the obvious corollary that the year must be true, and the content actually current.
A separate Seer Interactive analysis of 47,097 citations across ChatGPT, Gemini and Perplexity found the same pattern at scale: 75% of cited pages had been updated within the past year, and among 4,124 pages with both dates recorded, 72% looked fresh by last-modified date against just 42% by original publish date — a genuinely refreshed old page outcited a newly published one more often than not.
The same signal predicts traditional Google ranking, not just AI citation: a Growth Memo analysis of nearly 60,000 search results pages found a listicle with an old or unreadable publish date had 56% lower odds of reaching the top 3 — the single strongest factor measured, ahead of author bios, schema markup and word count.
-
Know the AI Overviews opt-out — and what it doesn't reach
Search Console → Settings → Search generative AI control
Search Console carries a Search generative AI control, under Settings, that is the only lever separating appearance in AI Overviews, AI Mode and Discover's generative features from ordinary Search — unlike noindex, nosnippet or Google-Extended, which each take something else with them. Google says the control "isn't used as a ranking or inclusion signal affecting other parts of Search" and takes one to two days to apply. It rolled out to all sites worldwide as of August 31, 2026, but Google's own AI-features guidance for site owners still does not mention it at all, naming nosnippet 18 times instead.
The control covers appearance, not training — Google's documentation sends publishers to Google-Extended for that, so it answers only one of the three uses (training, fine-tuning, grounding) the UK CMA's June 3, 2026 Publisher Conduct Requirement ordered Google to let publishers separately opt out of. There's a further caveat for news publishers specifically: Google renders Top Stories carousels inside AI Overviews on roughly 15.5% of US news searches, per an unconfirmed NewzDash analysis, so opting a news site out of AI Overviews may cost the Top Stories placement rendered within them, not just the summary.
Level 2 Measurement
Knowing whether any of it worked, and noticing when the engines change underneath you. Only 16% of brands track their AI visibility at all.
-
Wire the freshness signals: sitemaps, IndexNow, WebSub
Sitemaps, IndexNow, WebSub
Engines can only be as current as their discovery. Maintain an XML sitemap (a news sitemap if you publish news), ping IndexNow on every publish for the engines that accept it, and declare a WebSub hub on your feeds so subscribers are pushed rather than left to poll. None of this improves content; all of it shortens the gap between publishing and being retrievable.
-
Get your first-party AI numbers where they exist
Search Console, Bing Webmaster Tools
The engines have started shipping first-party AI-surface data: Search Console added generative-AI performance reports, and Bing Webmaster Tools previews citation-share and AI-visibility insights. First-party beats scraped estimates wherever both exist — verify your site in both consoles and take the numbers the engines will actually give you. The surface keeps widening: Microsoft Clarity now splits AI citation data into branded and non-branded views, and Search Console's platform properties extend first-party measurement to social and video accounts. Clarity's Bot Analytics dashboard added an AI Scrape-to-Referral Ratio card on August 13, 2026, ranking which AI crawlers send return visitors against those that mostly extract content without driving traffic back — though Clarity itself cautions a high ratio doesn't by itself signal failing AI visibility, since many AI platforms strip referrer data and likely undercount real referrals.
That first-party data matters because your own analytics can't be trusted to sort AI traffic correctly. A nine-month GA4 tracking study of 51,200 AI Overview citation clicks found an average of 22.4% of that traffic was misattributed to the Direct channel instead of Organic Search — worst in May 2026 at 29.3% — so a site's GA4 organic-search numbers alone understate AI Overview's real reach without cross-checking against Search Console.
-
Watch the engines' documentation, not just their announcements
Scheduled documentation diffing
The engines' most consequential changes ship quietly — help-page edits, crawler renames, policy language — hours or days before anyone announces them, if anyone ever does. Watch the primary documentation pages that govern your visibility and diff them on a schedule. The announcements are marketing; the documentation is policy.
Level 3 Bets
Cheap options on where this goes, labeled as bets rather than dressed as practice. The agent-protocol layer belongs here too — this page will carry it once the wire has reported it, and not before.
-
Publish llms.txt — as a bet on every surface except Google
llms.txt, markdown twins
llms.txt is a proposed standard: a machine-readable index of what your site offers, with llms-full.txt carrying the content itself for systems that want the corpus. Google has now answered its half of the open question on the record: its AI-features documentation states that machine-readable files, AI text files, and markup aren't needed to appear in Search or its AI features. An Ahrefs analysis spanning roughly 15 million data points adds evidence for the file's uncertain value even on the surfaces it's meant for: AI crawlers never read 97% of sites' llms.txt files at all. A Common Crawl analysis of 584,107 llms.txt files in its July 2026 archive suggests one reason why: 68% were produced by templates or SEO plugins rather than hand-authored — Wix alone accounts for 41% of the corpus — and only about half fully follow the spec's structure of a title, summary line, and link sections. So publish llms.txt for what it is — a near-zero-cost bet on the standalone assistants and agents beyond Google, deliberately authored rather than left to a plugin default — never as a Google lever, and never instead of the schema and crawlability every engine actually reads.
The same bet now extends past the index file: publishers have begun serving agent-readable markdown versions of pages outright — Time went as far as selling ads inside its agent-readable markdown — and for markdown-native sites, page-level .md twins cost nearly nothing. This site serves one for every item and reference page.
Google's John Mueller has pushed back on the premise itself: on his own test sites, "the only crawlers who claim to accept markdown are SEO tools" — not the major AI bots. His advice before building a markdown version on the assumption AI crawlers want it: log the accept header on incoming requests and check whether anything is actually asking for one.
That markdown surface is not exempt from trust rules. Perplexity has since blocked Time's markdown ads outright, calling the sponsored FAQ content deceptive and applying a trust-score downgrade to publishers that run it — an agent-facing page answers to the same trust standard as the human-facing one.
The spec itself got its first revision since launch on August 10: llms.txt version 2 adds rel="alternate" type="text/markdown" and rel="describedby" link relations so agents can discover a page's markdown twin and the llms.txt file governing it — formalizing, as a standard, the kind of Link-header discovery a markdown-twin setup should already expose.