Current Best Practices

Decide your AI crawler policy deliberately — and per bot

robots.txt, per user-agent

AI companies run separate crawlers for model training, search indexing, and live retrieval, and the trade-offs differ: blocking a training bot withholds content from future models, while blocking an index or retrieval bot removes you from today's answers. Set robots.txt per user-agent with that distinction in mind, and re-check the strings periodically — companies rename crawlers as products rebrand, and a rule pointed at yesterday's string silently stops applying. IAB Australia's Bots and Crawler Guidance formalizes the same call into four verdicts per crawler category — allow, allow with conditions, require licensing, or block — a workable checklist for making the decision deliberately rather than defaulting to one blanket rule.

A robots.txt block is a request, not a lock. TollBit found roughly 15% of AI page fetchers it tracked in the first half of 2026 reached European publisher URLs marked disallowed, with OpenAI's ChatGPT-User among the bots most often crossing blocks aimed specifically at it — treat a block as a policy statement worth logging compliance against, not a guarantee.

For Cloudflare-proxied sites, part of that policy is now set in a dashboard rather than hand-edited into the file. Bot Preference Sync, rolling out across every plan from free to Enterprise starting August 21, 2026, keeps robots.txt in step with an AI-crawler policy configured in the dashboard. It prepends a managed block rather than replacing the file — Cloudflare says any existing Disallow directives are maintained — and the sync is opt-in and can be switched off at any time. It also changes what a no-training preference actually buys: a crawler that does both search and training keeps access to a site marked no-training only if it meets four conditions — honoring the preference, offering an AI-summary opt-out, reporting URL-level training use alongside search metrics, and publicly showing the block doesn't hurt search visibility. Continued access now turns on public disclosure, not a crawler's self-declared promise.

Checked against Google, OpenAI and Anthropic's own documentation on August 27, 2026, none of the three publicly meets all four conditions — Google clears three, OpenAI two, Anthropic none. The condition every operator fails is the third: URL-level reporting of which pages were used for training, alongside search metrics. Cloudflare has published no compliance list of its own, and an operator could satisfy a condition through a private arrangement it never discloses — but on the public record, a dual-purpose crawler reaching a Cloudflare site that disallows training is doing so without publicly clearing the bar Cloudflare set for it.

That fourth condition now has teeth on new domains. Starting September 15, 2026, Cloudflare's own default blocks Training and Agent crawlers on any page that displays ads, leaving Search allowed — but Cloudflare's post also states that "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training," because those crawlers carry search and training under one user-agent. A site that takes the default therefore loses Google's crawler on every ad-bearing page — the exact harm the fourth condition requires an operator to show doesn't happen. The default applies only to new domains onboarding to Cloudflare; an existing domain keeps its current configuration, and owners who don't want it can switch it off in Security settings any time before September 15. What "pages that display ads" resolves to in practice, and whether the change also reaches existing free-plan sites, is not addressed in Cloudflare's post.

The user-agent string this whole policy runs on is itself forgeable. GreyNoise found 824 addresses across 795 networks sending traffic with forged AI-crawler names between July 28 and August 23, 2026 — including 263,849 sessions claiming to be Google-Extended, a token Google's own documentation says has no separate user-agent string at all, meaning none of those sessions could have been genuine. The forged traffic was scanning for credential and configuration files (.env, .aws/credentials, .git/config). Verify the connecting IP against each company's published crawler ranges before trusting what a User-Agent header claims.

That policy differentiation is now visible in practice, not just in Cloudflare's disclosure requirement. The wire's own crawler-access panel — 55 publisher domains probed daily — caught three People Inc. properties (Allrecipes, Investopedia, People) and The Atlantic making an identical change on September 13, 2026: OpenAI's GPTBot and a plain browser request both went from a 200 to a Cloudflare-managed 403, while Anthropic's ClaudeBot and Perplexity's crawler kept drawing the same Pay Per Crawl 402 as before — the same site charging two crawlers for the access it now refuses a third outright. Not every large site is taking that route: Quora hardened the same day to a flat 403 for GPTBot, ClaudeBot, PerplexityBot and a browser alike, and its newly-readable robots.txt disallows all three crawlers by name and bars AI-training use outright.

Three days later the same panel caught a second large publisher rejecting the split outright and one moving in the other direction. El País's homepage flipped from a 200 to a 403 for GPTBot, ClaudeBot and PerplexityBot alike between the September 13 and 16 captures. Stack Overflow moved just one crawler: it took Anthropic's ClaudeBot off Cloudflare's Pay Per Crawl paid track and onto a flat 403 block, while GPTBot and PerplexityBot stayed behind the same Cloudflare challenge as before — the panel's first capture of a crawler moving from paid access to blocked rather than the reverse.

A block set one week is not necessarily the block still in place the next. A third capture, September 20, moved two of those September 16 findings in different directions. El País's homepage returned to a 200 for both a browser request and ClaudeBot — reversing its own September 16 block — even though El País's robots.txt still disallows ClaudeBot by name, so the file and the enforcement behind it now disagree. The Atlantic moved PerplexityBot off Cloudflare's Pay Per Crawl track onto a flat 403, while keeping Anthropic's ClaudeBot on the paid track unchanged — the first capture of a publisher ending one crawler's paid access while leaving another's in place. Treat any snapshot of a site's crawler policy as dated the moment it's taken, and re-check enforcement rather than assuming a robots.txt file describes it.

A fourth capture, September 27, caught the divergence running the other way: a robots.txt more permissive than the enforcement in front of it. The Atlantic's robots.txt became readable again for the first time since September 13 and now names GPTBot, OAI-SearchBot and ChatGPT-User as allowed — but its live homepage still returns Cloudflare's 403 to GPTBot itself, unchanged since September 13. Neither document alone is reliable: check the file and the live response, because they can disagree in either direction. The same capture found Search Engine Land's robots.txt, readable throughout the panel's tracking, start returning the same Cloudflare challenge as its homepage — a trade publication covering this beat losing machine-readable access to its own stated policy.

A separate measurement adds scale to the compliance question itself, and a new failure mode to the Cloudflare migration above it. HasData's AI Crawler Block Index tested 10,894 domains directly and found 592 with a declared GPTBot ban in robots.txt — of those, 234 (39.5%) still served GPTBot a live page when fetched, a declaration-level echo of TollBit's per-fetch 15% crossing rate. The same study found the migration itself cost 118 sites their existing opt-out outright: when Cloudflare retired its older "Managed Robots.txt" and "Block AI Bots" features on September 15, 2026, the replacement left robots.txt blank until the customer reviewed and confirmed new rules — a step Cloudflare's own announcements did not flag to existing customers. Named among the sites that lost their opt-out: the Smithsonian, Ko-fi, Find a Grave, Vatican News, National Review, Bellingcat and Patreon.

Verified 2026-10-08 · References: AI crawlers — glossary · Google robots.txt documentation · NotebookLM crawler renamed (wire) · IAB Australia sorts AI crawlers into four access verdicts (wire) · ChatGPT ads expand to Europe as crawler bypasses blocks (wire) · Cloudflare syncs robots.txt, sets four conditions for crawlers that also train (wire) · No major AI crawler publicly meets all four of Cloudflare's conditions (wire) · Cloudflare's September 15 default blocks Googlebot on ad-bearing pages (wire) · GreyNoise finds forged ClaudeBot and GPTBot traffic hunting for credentials (wire) · People Inc. sites and The Atlantic block GPTBot, still bill Claude, Perplexity (wire) · El País blocks GPTBot, ClaudeBot and PerplexityBot (wire) · El País reopens to ClaudeBot; The Atlantic ends Perplexity's paid access (wire) · The Atlantic's robots.txt allows GPTBot, but Cloudflare still blocks it (wire) · HasData: Cloudflare's robots.txt change erased GPTBot bans at 118 sites (wire)

Serve real HTML links — AI crawlers won't run your JavaScript

Server-rendered HTML anchor links, not JS-injected navigation

GPTBot, ClaudeBot and Bingbot do not execute JavaScript to discover links. A 41-day test on a 2,400-page site found all three crawlers found zero of the pages linked only through JavaScript, while even Googlebot — which does render JS — reached just 2% of them (48% went to Google's separate, JS-capable GoogleOther crawler). The fix is mechanical, not architectural: after the site's JavaScript links were converted to plain HTML anchors, GPTBot recovered 250 of the previously unreachable pages within 48 hours.

An internal-link audit judged only against Googlebot's crawl behavior is not sufficient for AI visibility — an AI crawler can be stricter than Google's own, silently excluding JavaScript-linked pages from citation eligibility even while the same pages rank fine in Search.

Fetchability pays off in the answer, not only in discovery. A September preprint ran 37,927 simulated buyer-question journeys across four agent harnesses and 1,056 real businesses matched on fame and prior model knowledge: for businesses an agent could successfully fetch and read, answers were built from their own pages 78% of the time against 56% for the rest, they were clearly recommended 1.9 times as often, and the answers were 41% more accurate. It is one preprint, and a co-author co-founded an AI-agent search company; the direction matches the crawler test above.

Verified 2026-10-09 · References: 41-day test: AI crawlers miss JavaScript-only links (wire) · Study ties agent-ready websites to more AI recommendations (wire)

Put the answer where a machine can lift it

Question-shaped headings, Q&A blocks

Answer engines quote passages, not pages. Content that answers a searchable question in a self-contained, quotable block — near the top, in plain declarative language — is retrievable and citable in a way that the same facts diffused across twelve paragraphs are not. Question-shaped headings and explicit Q&A blocks make the mapping from query to passage trivial. A Pillarbase analysis of 15.7 million Google AI Mode citations puts numbers to that shape: the passages Google reuses most average 117 words, answer in the first sentence about 80% of the time, and are self-contained about 85% of the time.

Wording matters as much as shape. A preprint analyzing roughly 2 million citations from ChatGPT, Claude, Google AI and Gemini, joined to 10,000 pages across 19 B2B SaaS workspaces, found prompt-content alignment, the word overlap between a page and the wording of real prompts, was the strongest page-level predictor of citation (beta +0.37), while FAQ blocks, structured data and Core Web Vitals stopped predicting it once domain-level effects were controlled for. The authors run an AEO agency and the paper is not peer reviewed, but it points the same way as the passage data: answer in the words people actually ask with.

A later study supplies the mechanism behind that domain-level finding. Tracking 90 days of ChatGPT, Gemini and Perplexity answers (158,099 answers, 468,037 citations), University of Toronto researchers found that whether ChatGPT's own rewritten search queries retrieved a page explained 32% of which pages it cited within a given site, against just 2.8% for page content and structure — a gap the study calls statistically decisive. Content still mattered, but mainly through matching the wording of those rewritten queries, not through FAQ blocks, structured data or page speed, which only showed an effect when comparing across different websites rather than between pages of the same one. A separate experiment in the same study found the models' own prior knowledge skews results too: with every version of a product holding identical facts, revealing its brand name changed ChatGPT's top-three ranking rate 2.1 times as much as merely repeating the prompt, and 3.3 times as much for Gemini.

Heading structure and question-phrased titles show the shape this practice recommends in a newer study — on the figure, not the mechanism this practice tracked before. Sistrix matched 48,527 AI-cited pages against uncited pages on the same domain across 96,418 URLs and found pages with five or more H2 headings were cited 1.6 times as often as matched pages with fewer; on ChatGPT specifically, a question-phrased title carried a separate 1.9x factor. The same study found FAQPage schema and other markup also correlate with citation once domain is controlled for — in tension with the domain-authority study above, which found markup's apparent effect disappear under the same control. See the structured-data practice for that disagreement; it doesn't touch the heading-count and question-title findings here, which have no competing null result on the record.

Verified 2026-10-22 · References: RAG — glossary (why retrieval units matter) · AEO — glossary · Pillarbase: 15.7M AI Mode citations show what Google reuses (wire) · Study of 2M AI citations: domain authority beats SEO signals (wire) · Study: ChatGPT's own search explains citations better than page content (wire) · Sistrix ranks GEO tactics by how often AI engines cite them (wire)

Ship structured data for what the page actually is

Schema.org JSON-LD

Schema.org markup — NewsArticle for news, FAQPage for question-and-answer content, DefinedTerm for definitions, Organization and Person for who stands behind it — is machine-readable self-description, and it is cheap. Mark up what the page genuinely is; structured data that misdescribes a page is a trust liability, not an optimization. That framing now has a direct test behind it: an Ahrefs analysis of 1,885 pages found adding schema markup produced no measurable increase in AI citations after 30 days — treat it as descriptive infrastructure, not a lever for citation volume.

A separate study widens that finding past schema specifically. Re-testing the original 2023 GEO citation-lever effect sizes against ten modern AI engine families found none of them moved citations any longer, and a deterministic content score built from those levers correlated with actual citations at just 0.11 within-query — weak enough that the authors position such scores as content-quality filters, not citation predictors. The on-page levers that still measure are structural (crawlability, recency, passage shape), not scored content attributes.

A third study explains why schema keeps testing null: across roughly 2 million citations tied to 10,000 pages, domain-level AI authority outweighed every page-level feature — schema included — sixfold in SHAP value, and FAQ blocks, structured data and Core Web Vitals all showed a positive correlation with citations that reversed or fell to zero once that domain effect was controlled for. The strongest surviving page-level predictor was how closely a page's wording matched real user prompts, not any traditional SEO signal.

One case now makes markup a gate rather than a lever. Google's UGC Fresh Data Program, documented October 8, 2026, lets qualifying forum and social platforms submit fresh user-generated content directly to Search outside normal crawling — but only if the platform uses schema.org markup such as SocialMediaPosting or DiscussionForumPosting, alongside OAuth 2.0 authentication and active moderation. That's a different role for structured data than the citation-ranking tests above: here it's an eligibility requirement for a faster ingestion channel, not something Google claims moves citations once content is in.

A fourth study complicates the null result rather than confirming it. Sistrix matched 48,527 pages cited by ChatGPT, Google AI Mode and AI Overviews against uncited pages on the same domain — controlling for domain-level authority by the same logic as the domain-authority study above — across 96,418 URLs and 17,380 domains, and still found Article schema markup carried a 1.6x citation-odds factor and FAQPage schema a separate 1.5x, the third- and fifth-strongest of 43 features tested (current year in the title led at 2.7x; a table of contents, used by only 14% of pages, carried 1.7x). That is a disagreement between two domain-controlled studies, not a resolution of the earlier one: the record holds both rather than picking a winner. Schema's citation effect, if any, is contested, not settled either way.

Verified 2026-10-21 · References: Google structured data documentation · Schema.org · Ahrefs study: llms.txt files mostly unread by AI crawlers (wire) · GEO content scores barely predict AI citations, study finds (wire) · Study of 2M AI citations: domain authority beats SEO signals (wire) · Google launches UGC Fresh Data Program for forums and social platforms (wire) · Sistrix ranks GEO tactics by how often AI engines cite them (wire)

Be unambiguous about your entities

Canonical hub pages, sameAs

Knowledge graphs and answer engines resolve content to entities — companies, products, people. Give each entity you own a canonical hub page, keep names consistent, and declare external identities with sameAs links to official sites and reference pages. Ambiguity about which thing you mean is a retrieval failure you inflicted on yourself.

The pages other people keep about you are part of your entity footprint. On searches for a brand's name alone, Ahrefs found Google's AI Overviews built mostly from third-party pages: of 57 brand-name overviews whose sources it could see, 41 cited Wikipedia, 22 YouTube and 13 LinkedIn, and only one a competitor's site (vendor-published). Wikipedia carries similar weight in ChatGPT, where it alone made up 10 to 14% of the domains cited in answers to SaaS buying queries, per a Growth Memo analysis. Check what the reference pages you declare with sameAs say about you, because answer engines read them as you.

Verified 2026-10-09 · References: The record — this site's entity hubs · schema.org/sameAs · Ahrefs: AI Overviews now appear on 83% of branded searches (wire) · UGC platforms outcite review sites in ChatGPT answers (wire)

Give readers a one-click way to name you a Preferred Source

Preferred Sources embed button

Google's Preferred Sources feature lets a reader mark a site as preferred across Search, AI Overviews and AI Mode, and Google says a preferred pick makes that site surface more often in all three. As of August 2026, publishers can embed a one-click "Add to Preferred Sources" button directly on their own pages rather than relying on readers to find the pick buried in search settings, and Google has now extended the same mechanism into AI Mode's new developing-topic link carousel. Adoption is real and growing: unique sources selected as Preferred rose from over 345,000 at the feature's May 2026 launch to over 600,000 by August, and Google said in October 2026 that use is up more than 7x since the start of the year — though it gave no absolute count alongside that multiple.

Google names only Top Stories, AI Overviews and AI Mode as surfaces a preferred pick affects — Discover is not on that list, even though Discover now sends news publishers more than twice the Google traffic Web Search does (67.5% versus 27% in Q4 2025, reversed from 2023), per publishers tracked by NewzDash. A reader-facing pick does not substitute for Discover-specific work.

Verified 2026-10-16 · References: Google: personalize your search and Discover with preferred sources · Google lets publishers embed a Preferred Sources button (wire) · Google's AI Mode gets link carousels for developing topics (wire) · Google Discover now sends news publishers more than twice what Search does (wire) · Google adds sources section, more links to AI Overviews and AI Mode (wire)

Date everything, visibly and in markup

Visible dates, honest dateModified

Answer engines weigh freshness and increasingly surface dates alongside citations. Publish visible dates, keep dateModified truthful in markup, and never launder old content as new — an engine that catches a laundered date has a durable reason to distrust the domain. Honest dating is both a trust signal and a defense. The effect is measurable: a controlled Otterly experiment adding the current year to 11 page titles lifted their AI-search citations 61% — with the obvious corollary that the year must be true, and the content actually current.

A separate Seer Interactive analysis of 47,097 citations across ChatGPT, Gemini and Perplexity found the same pattern at scale: 75% of cited pages had been updated within the past year, and among 4,124 pages with both dates recorded, 72% looked fresh by last-modified date against just 42% by original publish date — a genuinely refreshed old page outcited a newly published one more often than not.

The same signal predicts traditional Google ranking, not just AI citation: a Growth Memo analysis of nearly 60,000 search results pages found a listicle with an old or unreadable publish date had 56% lower odds of reaching the top 3 — the single strongest factor measured, ahead of author bios, schema markup and word count.

Seasonal content shows the same failure at the category level, and on a deadline. A BrightEdge analysis of 13,915 holiday prompts against ChatGPT across six major US retailers (June-August 2026) found dated press releases outciting a retailer's own current, correct seasonal page: at Best Buy, a dated release drew 71.7% of Black Friday citations versus 10.5% for the live store hub; Home Depot split 63.1% to 13.3%. Weighted across all six retailers, about 11% of cited holiday pages carried a year that had already passed — nothing ages a stale press release out of the citation pool until the retailer replaces it, so a seasonal page needs updating before peak demand, not just published once and left.

Verified 2026-10-08 · References: This site's editorial standards · Otterly title-year experiment — vendor blog, not wire-verified · Seer: content recency's impact on AI visibility (wire) · Growth Memo: freshness and Google listicle ranking (wire) · BrightEdge: ChatGPT's Black Friday citations still favor dated press releases (wire)

Know the AI Overviews opt-out — and what it doesn't reach

Search Console → Settings → Search generative AI control

Search Console carries a Search generative AI control, under Settings, that is the only lever separating appearance in AI Overviews, AI Mode and Discover's generative features from ordinary Search — unlike noindex, nosnippet or Google-Extended, which each take something else with them. Google says the control "isn't used as a ranking or inclusion signal affecting other parts of Search" and takes one to two days to apply. It rolled out to all sites worldwide as of August 31, 2026, but Google's own AI-features-specific guidance for site owners still does not mention it at all, naming nosnippet 18 times instead — unchanged when rechecked October 21, 2026. A separate, general Search Central page, "Control what you share with Google Search," was updated that same day to list the control alongside the robots meta tag, data-nosnippet and X-Robots-Tag, so the control is now documented somewhere in Google's help center — just not in the page aimed at this question.

The control covers appearance, not training — Google's documentation sends publishers to Google-Extended for that, so it answers only one of the three uses (training, fine-tuning, grounding) the UK CMA's June 3, 2026 Publisher Conduct Requirement ordered Google to let publishers separately opt out of. There's a further caveat for news publishers specifically: Google renders Top Stories carousels inside AI Overviews on roughly 15.5% of US news searches, per an unconfirmed NewzDash analysis, so opting a news site out of AI Overviews may cost the Top Stories placement rendered within them, not just the summary.

The control is set per Search Console property, not per domain. A site managing subdomains or subfolders as separate properties could previously have a child property quietly opt back into AI features after the parent domain opted out, with no visible flag. Search Console's documentation now says the top-level domain property shows a notification when a child property overrides the parent's setting, per Search Engine Roundtable.

Verified 2026-10-21 · References: Google Search Console Help: Search generative AI control · Google: AI features and your website · What Google's AI Overviews opt-out does, and two things it does not (wire) · Search Console's AI Overviews opt-out control rolls out to all sites (wire) · Search Console flags child-property overrides of its AI opt-out control (wire) · Google adds AI Overviews opt-out to 'Control what you share' guide (wire)

Wire the freshness signals: sitemaps, IndexNow, WebSub

Sitemaps, IndexNow, WebSub

Engines can only be as current as their discovery. Maintain an XML sitemap (a news sitemap if you publish news), ping IndexNow on every publish for the engines that accept it, and declare a WebSub hub on your feeds so subscribers are pushed rather than left to poll. None of this improves content; all of it shortens the gap between publishing and being retrievable.

A sitemap "couldn't fetch" error in Search Console doesn't always mean the file is broken. Google's John Mueller and Martin Splitt said on the Oct. 1, 2026 "Search Off the Record" podcast that the error can also reflect Google's systems being too busy, or a judgment that a site's perceived quality doesn't justify more crawl budget — check the file is actually valid before treating the error as a technical fault to fix. Mueller also said RSS feeds can double as short-form sitemaps, typically listing 10-20 dated entries, which is a lighter-weight option than a full XML sitemap for a feed that updates often. Neither Google statement extends any role to llms.txt: Mueller said Google's search systems don't process llms.txt files as sitemaps at all and that site owners shouldn't treat it as a priority, consistent with llms.txt's standing as a bet on other engines, not Google.

PDF files carry their own freshness ceiling. Google's documentation caps Googlebot's PDF fetch at the first 64MB of a file — a limit Google's own search advocates Splitt and Mueller couldn't recall on their own podcast despite it appearing in Google's crawler documentation. Converted PDFs rank in AI Overviews and AI Mode the same as standard HTML pages, but Google refetches PDFs far less often than HTML; Mueller's advice for updating one is to give it a new filename rather than edit the file at the same URL, since an edit left at the same address may sit unrefetched for a long stretch.

Verified 2026-10-14 · References: IndexNow · WebSub (W3C) · Google sitemap documentation · Google: sitemap "couldn't fetch" errors can mean low crawl demand (wire) · Google's own hosts can't name its documented 64MB PDF crawl limit (wire)

Get your first-party AI numbers where they exist

Search Console, Bing Webmaster Tools

The engines have started shipping first-party AI-surface data: Search Console added generative-AI performance reports, and Bing Webmaster Tools previews citation-share and AI-visibility insights. First-party beats scraped estimates wherever both exist — verify your site in both consoles and take the numbers the engines will actually give you. The surface keeps widening: Microsoft Clarity now splits AI citation data into branded and non-branded views, and Search Console's platform properties extend first-party measurement to social and video accounts. Clarity's Bot Analytics dashboard added an AI Scrape-to-Referral Ratio card on August 13, 2026, ranking which AI crawlers send return visitors against those that mostly extract content without driving traffic back — though Clarity itself cautions a high ratio doesn't by itself signal failing AI visibility, since many AI platforms strip referrer data and likely undercount real referrals. Baidu joined with an AI Traffic Analysis section in its own webmaster tools around September 15, 2026, but its numbers aren't directly comparable to the others: Baidu's AI CTR divides clicks by citations rather than by impressions, which can understate real link performance, per Search Engine Journal; the dashboards were still empty as the rollout continued in phases, and the tool covers only citations inside Baidu's traditional search results, not its standalone AI products or rival Chinese platforms. Treat each engine's first-party AI metric as defined by that engine, not as a common unit across consoles.

That first-party data matters because your own analytics can't be trusted to sort AI traffic correctly. A nine-month GA4 tracking study of 51,200 AI Overview citation clicks found an average of 22.4% of that traffic was misattributed to the Direct channel instead of Organic Search — worst in May 2026 at 29.3% — so a site's GA4 organic-search numbers alone understate AI Overview's real reach without cross-checking against Search Console.

Gemini has started fixing a piece of that gap itself: as of October 2026 it appends UTM parameters to the links it sends users to, letting site owners separate Gemini referrals from the Direct bucket in GA4 for the first time, per Search Engine Journal, a change Google's John Mueller said he'd seen too. The tag mostly survives only on desktop web, with roughly 9% pass-through from Gemini's mobile app, and Google hasn't documented when the tagging triggers — treat it as a partial fix, not full coverage, until confirmed.

ChatGPT moved the opposite direction the same month. Around October 9, 2026 it stopped appending a UTM tag to its outbound links and started opening them with `noopener,noreferrer`, which also blocks the referrer header — so visits that used to register as ChatGPT traffic now show up as Direct, per Search Engine Roundtable. Adobe's Cedric Huesler, analyzing roughly 500 pages that got at least 5% of their traffic from ChatGPT, found ChatGPT-attributed visits fell by about 31,000 a day from October 9-13 while direct traffic on the same pages rose by about 26,000 a day. Profound ties the cause to its newer Intelligent UI answer format rather than a bug: the share of ChatGPT answers carrying a tagged citation link fell from about 98% on October 8 to 66% by October 14, tracking Intelligent UI's share of answers rising from 2% to roughly 35% over the same days. If a site's GA4 Direct channel is growing unexpectedly alongside ChatGPT citations elsewhere, this is a likely cause, not a traffic-quality problem to chase.

But don't assume every rise in Direct traffic hides AI referrals. A Growth Memo analysis of GA4 data across 12 sites from January through September 2026 found raw Direct traffic rose 140.9%, while engaged (non-bot) Direct traffic fell 1.0% in raw monthly counts (up 2.3% once normalized for days per month) — one site in the sample logged 409,000 Direct sessions of which only 198 counted as engaged, illustrating how much of the apparent growth was bot activity rather than real visits. Only 0.003% of Direct sessions carried an identifiable AI trace, and testing 632 comparisons of AI-visibility metrics against branded-search and Direct traffic across four brands turned up correlations as high as 0.72 that didn't hold up against randomized control data, suggesting the pattern was likely coincidental rather than causal. Check engaged sessions and bot filtering before reading a Direct-traffic increase as a sign of unmeasured AI traffic.

Google's own results page may be testing the same fix a day later. Marketer Tom Pool spotted a `?st_source=ai_overview` parameter on links inside AI Overviews and AI Mode, which would let site owners separate that click traffic from ordinary search clicks the same way Gemini's UTM tag does. Treat it as a sighting, not a scheme to build reporting around yet: Google hasn't confirmed the test, and Search Engine Roundtable's own attempt to reproduce it across several browsers didn't surface the parameter.

First-party doesn't mean link-level, though. Responding to a Reddit thread, Google's John Mueller said Search Console reports AI Overview position "as a block" — the same way it handles other search features — rather than isolating a per-link position, which he said "is hard to do." Links revealed only via a "Show More" expansion are undercounted, and an impression doesn't confirm a user actually saw the link. Mueller asked for input on what better tracking would look like but described no fix in progress. Per Search Engine Journal's follow-up, that block flattening typically reports every link in the box at position 1 regardless of whether it's shown prominently or buried behind "Show More" — so treat a Search Console AI Overview position number as directional rather than a rank tracker's link-level reading, and weigh actual clicks and conversions over the position figure for AI Overview traffic.

Bing's first-party view is moving from single queries toward portfolio-level strategy. Topics Portfolio, a beta added to its AI Performance report, groups a site's AI Search citations into thematic clusters, each scored by Topic Size (total citations across all sites, a demand signal), Sites Cited (competing sites) and Queries — and separates a site's own "grounding queries" from "Opportunity Queries," where competitors are cited and the site is not. Microsoft says the beta appears only for sites with "sufficient data" and names no threshold, so smaller sites may see little or nothing yet.

Verified 2026-10-21 · References: Search Console adds generative-AI reports (wire) · Bing WMT previews AI visibility insights (wire) · Clarity splits AI citations branded/non-branded (wire) · Platform properties roll out globally (wire) · Study finds AI Overview traffic misattributed 22% of time (wire) · Microsoft Clarity adds an AI scrape-to-referral ratio (wire) · Google: Search Console tracks AI Overviews position only as a block (wire) · Baidu adds AI citation and CTR tracking to webmaster tools (wire) · Search Console's block flattening masks AI Overview link rank (wire) · Gemini quietly adds UTM parameters to outgoing links (wire) · Google tests a tracking parameter on AI Overview and AI Mode links (wire) · Bing Webmaster Tools adds Topics Portfolio to AI Performance (wire) · ChatGPT removes UTM tracking from its outbound links (wire) · Profound ties missing ChatGPT UTM tags to Intelligent UI (wire) · Analysis: GA4 Direct traffic shows little hidden AI activity (wire)

Track citation share, and expect it to move when models do

Per-engine citation tracking

Citation share — the fraction of an engine's citations your domain earns — is the discipline's emerging visibility metric. Measure it per engine, because the same content earns different shares in different engines; and treat every model swap inside an engine as a potential discontinuity in your numbers, because the model doing the citing just changed. Not every citation names the source, either: a Writesonic analysis of roughly 16 million brand appearances found engines omit the brand name anywhere from 19% (Microsoft Copilot) to 52% (Perplexity) of the time, so a citation-share number built only on named mentions will undercount actual retrieval. And citation volume alone doesn't establish category leadership: a Kevin Indig analysis of Semrush's ChatGPT citation data found the most-cited domain matched the most-mentioned brand only 20.8% of the time across 1,094 categories.

Only 16% of brands track this at all, and the vendors selling the other 84% a dashboard often disagree on the same brand's numbers. The IAB's "Measuring Visibility in the AI Era" framework gives buyers a checklist for that gap: don't trust a citation-share number until the vendor discloses platform coverage, prompt-library construction, and query volume — the IAB sets 50 queries as the floor before a program counts as more than exploratory — and separates hallucination rate from plain factual-accuracy error.

The coverage gap the IAB asks vendors to disclose now has a number attached. A study replaying the same ChatGPT Web queries through OpenAI's Responses API and Google Search — the two external proxies most GEO tracking tools lean on in place of real ChatGPT Web citations — found the two combined recovered only about a quarter of the exact URLs ChatGPT Web actually cited. A model combining both signals tracked hostname- and entity-level visibility far better (AUC above 0.74 for GPT-5.3 and GPT-5.5), which argues for weighting domain- and entity-level visibility over exact-URL counts in any tool built on these proxies.

A large-scale drift study puts numbers to "expect it to move": SISTRIX re-sampled 82,619 prompts weekly for 17 weeks and found Google replaces 56% of its cited sources every week, ChatGPT as much as 74%. The churn isn't uniform — 86% of prompts keep a stable core of a few domains while everything outside it rotates at 89% per week — and Google AI Overviews and AI Mode cite different domains 83% of the time, so track the two Google surfaces separately rather than as one number. For a publisher, a single news-article citation behaves less like an asset that accrues than a placement that expires: only 1.4% of cited news articles held their spot across all 17 weeks, against 43% of brand queries that kept the brand's own domain present throughout even as its co-citations kept turning over.

The model-swap risk above isn't hypothetical. Two days after Google rolled Gemini 3.8 Flash out to AI Mode, SEO practitioners found citations vanishing from top-of-funnel queries entirely, and Google's Search VP confirmed it as a bug rather than a design change. A citation tracker that only samples periodically reads an incident like that as a sudden collapse in visibility, indistinguishable at a glance from an actual ranking loss — check for an engine-side incident before concluding a citation drop reflects on your content.

Not every model swap is a bug — some are a durable shift in how the engine retrieves at all. When ChatGPT-5.6 Luna became OpenAI's default model, Seer Interactive measured it using the site: operator to limit a query to one domain in 76% of 346 matched fan-out queries, versus 21.6% for GPT-5.5, while consulting 43% fewer sources overall and generating about 13% fewer inline citations. A far larger Ahrefs sample — 15 million fan-out queries drawn from 66.6 million logged-out responses in August-September 2026 — measured site: operator use at just 6.8%, spread across roughly 116,000 domains; Ahrefs attributes the gap to sample scope and prompt mix rather than either study being wrong, so read the two as a range, not a resolved number. The same Ahrefs data found only 14.4% of prompts triggered a web search at all, averaging 1.6 fan-out queries when they did — in the other six answers out of seven, what the model already associates with a brand likely matters more than anything a page does in that moment. If an engine increasingly searches named domains directly rather than the open web, a domain it doesn't search for by name risks exclusion from retrieval, not just under-citation — raising the bar from earning a citation to earning a targeted site: lookup in the first place.

A citation also decays on its own measurable schedule, separate from weekly churn. Profound's Citation Decay tracker found that across 883,000 pages on seven engines over a year, the median page's citation share loses half its peak within 11 days, with the full rise-to-half-peak arc running 42 days. Durability is engine-specific — a sustained Claude citation is 2.8 times as likely as a ChatGPT one to still hold half its peak two months out — and a page cited by five or more engines is 5.5 times as likely as a single-engine page to stay above half its peak after eight weeks. Treat a citation as a position that decays on a schedule, not a one-time win: refreshing and re-promoting cited content is what keeps it above half-peak.

The per-engine split isn't just how much gets cited — it's which pages. BrightEdge compared the top 50 cited rows per domain on ChatGPT and Google AI Overviews for holiday-shopping prompts across four US retailers: AI Overviews put 64.6% of its citations on commercial product pages versus 35.4% for ChatGPT, while ChatGPT sent 51.5% of its citation weight to newsroom and customer-support pages versus 18.1% for AI Overviews. A page built to rank for commercial intent can still miss an engine's citations if that engine is drawing from a different part of the site.

The split can also flip which brand leads a category outright. In a separate BrightEdge analysis of holiday-travel queries, Hilton led hotel brands on Google AI Overviews visibility (28.3%) while Marriott led instead on ChatGPT (29.4%) — the two engines didn't just weight the same leader differently, they disagreed on who the leader was. A citation-share report that tracks only one engine can name the wrong brand as the category winner.

Verified 2026-10-21 · References: Citation share — glossary · Ahrefs' four-surface citation rankings (wire) · Search now runs Gemini 3.5 Flash-Lite (wire) · Writesonic: AI engines often skip naming source brands (wire) · Analysis: most AI search demand has no clear ChatGPT category leader (wire) · IAB sets first measurement standard for AI-visibility tools (wire) · SISTRIX: ChatGPT replaces 74% of its cited sources every week (wire) · Google confirms AI Mode citation bug in Gemini 3.8 Flash (wire) · ChatGPT-5.6 Luna shifts to site: search, cites fewer sources (wire) · Profound: AI citation half-life is 11 days (wire) · Ahrefs: ChatGPT triggers a web search on just 14% of prompts (wire) · Study: OpenAI's Responses API captures only a quarter of ChatGPT's citations (wire) · BrightEdge: Google AI Overviews, ChatGPT cite different kinds of holiday pages (wire) · BrightEdge: Hilton leads AI Overviews, Marriott leads ChatGPT on holiday travel (wire)

Watch the engines' documentation, not just their announcements

Scheduled documentation diffing

The engines' most consequential changes ship quietly — help-page edits, crawler renames, policy language — hours or days before anyone announces them, if anyone ever does. Watch the primary documentation pages that govern your visibility and diff them on a schedule. The announcements are marketing; the documentation is policy.

Late September into October 2026 ran five such changes in two weeks, none announced separately from the page itself: Search Console's help page split its "Web" search-type filter into text-based and multimodal (image-in-query) sub-types; Google's VideoObject structured data documentation added a recommended creator property and spelled out which interactions count toward interactionStatistic; Gemini began tagging outgoing links with UTM parameters, confirmed only after someone noticed on Reddit; Merchant Center's Loyalty Customer Match documentation named AI Mode and Gemini for the first time while expanding to 14 countries; and a new UGC Fresh Data Program for forums and social platforms surfaced only in a Search Central changelog.

Verified 2026-10-09 · References: Help page spells out AI Overviews training (wire exclusive) · Search Console now separates text and multimodal web search (wire) · Google adds creator property to VideoObject structured data (wire) · Gemini quietly adds UTM parameters to outgoing links (wire) · Google adds AI Mode and Gemini to Loyalty Customer Match, expands to 14 countries (wire) · Google launches UGC Fresh Data Program for forums and social platforms (wire) · Sources & method — how this site watches

Publish llms.txt — as a bet on every surface except Google

llms.txt, markdown twins

llms.txt is a proposed standard: a machine-readable index of what your site offers, with llms-full.txt carrying the content itself for systems that want the corpus. Google has now answered its half of the open question on the record: its AI-features documentation states that machine-readable files, AI text files, and markup aren't needed to appear in Search or its AI features. An Ahrefs analysis spanning roughly 15 million data points adds evidence for the file's uncertain value even on the surfaces it's meant for: AI crawlers never read 97% of sites' llms.txt files at all. A Common Crawl analysis of 584,107 llms.txt files in its July 2026 archive suggests one reason why: 68% were produced by templates or SEO plugins rather than hand-authored — Wix alone accounts for 41% of the corpus — and only about half fully follow the spec's structure of a title, summary line, and link sections. So publish llms.txt for what it is — a near-zero-cost bet on the standalone assistants and agents beyond Google, deliberately authored rather than left to a plugin default — never as a Google lever, and never instead of the schema and crawlability every engine actually reads.

The same bet now extends past the index file: publishers have begun serving agent-readable markdown versions of pages outright — Time went as far as selling ads inside its agent-readable markdown — and for markdown-native sites, page-level .md twins cost nearly nothing. This site serves one for every item and reference page.

Google's John Mueller has pushed back on the premise itself: on his own test sites, "the only crawlers who claim to accept markdown are SEO tools" — not the major AI bots. His advice before building a markdown version on the assumption AI crawlers want it: log the accept header on incoming requests and check whether anything is actually asking for one.

That markdown surface is not exempt from trust rules. Perplexity has since blocked Time's markdown ads outright, calling the sponsored FAQ content deceptive and applying a trust-score downgrade to publishers that run it — an agent-facing page answers to the same trust standard as the human-facing one.

The spec itself got its first revision since launch on August 10: llms.txt version 2 adds rel="alternate" type="text/markdown" and rel="describedby" link relations so agents can discover a page's markdown twin and the llms.txt file governing it — formalizing, as a standard, the kind of Link-header discovery a markdown-twin setup should already expose.

Adoption is still rare among the sites with the most to gain. LintLab crawled the 1,000 most-visited sites on Chrome's August 2026 top-sites list on October 1 and found only 66 of them, 6.6%, serving a valid llms.txt file; nearly two-thirds returned a 404, per PPC Land's report of the vendor-published crawl. Read alongside Ahrefs' finding that 97% of published files go unread, the file remains a cheap bet rather than a convention the large sites have adopted.

Verified 2026-10-09 · References: Google: AI features and your website · llmstxt.org — the proposal · llms.txt — the record · Ahrefs ships a free llms.txt generator (wire) · Time sells ads inside agent-readable markdown (wire) · Perplexity blocks Time's markdown ads (wire) · llms.txt spec adds link relations for Markdown discovery (wire) · Ahrefs study: llms.txt files mostly unread by AI crawlers (wire) · Google's Mueller: AI crawlers aren't requesting markdown (wire) · Common Crawl: two-thirds of llms.txt files are templates (wire) · LintLab: sitemap errors common, llms.txt adoption still rare (wire)