Vercatus: AI crawlers read SaaS sites, not travel sites
What this answers
- Which website categories are most and least accessible to AI crawlers?
- Per Vercatus's test of 64 well-known sites across 8 niches (dated September 18, 2026), software and SaaS sites served GPTBot, ClaudeBot and PerplexityBot their home page most often (7 of 8), while travel and hospitality sites did so least (1 of 8).
- How many major websites block all AI crawlers?
- 27 of the 64 sites Vercatus tested refused or challenged all three named AI crawlers (GPTBot, ClaudeBot and PerplexityBot) with a 401, 403 or bot-management interstitial; 33 served all three, and 4 let some crawlers in while blocking others.
Vercatus, an AI-visibility platform, probed the home page and one deep page of 64 well-known sites across eight niches — eight sites each — with GPTBot, ClaudeBot and PerplexityBot identities on September 18, 2026. Software and SaaS sites let all three crawlers in most often (7 of 8), followed by banking and health (6 of 8 each); travel and hospitality let them in least (1 of 8), with news and media, e-commerce and consumer brands each at 3 of 8.
Across all 64 sites, 33 served their home page to all three crawlers, 27 refused or challenged all three with a 401, 403 or bot-management interstitial, and 4 let some crawlers in while blocking others. Separately, 17 of the 64 publish an llms.txt and 9 keep over half their deep page’s content behind JavaScript. Among named examples, the BBC served every crawler tested; The New York Times and Reuters challenged all of them.
Why it matters: Niche determines AI access more than any blanket industry stance — travel sites, whose business is answering exactly the questions assistants get asked about trip planning, are the category assistants can least read directly.
The record: OpenAIAnthropicPerplexityllms.txt
Glossary: AI crawlersllms.txt
Via Vercatus ↗
Posted to the wire October 25, 2026. Edited by Joe Balewski.