The Mental Model Most Marketers Are Still Using Is Wrong
As of 2026, there are actually two paths, each with a different set of rules and that serve different purposes. The first is a frozen snapshot. It’s used by LLMs when they’re pretraining and becoming more fluent. The second is a live retrieval system. It pulls fresh content to answer queries with relevant, up-to-date information.
Interestingly, Google AI Overviews and Perplexity don’t just pull from pretraining data. Instead, they index your site like a search engine (or use an existing search engineand find answers on the fly. This is where the majority of AI search visibility actually take place.
“Mindset wise, it’s realising that the field is changing drastically and what was true before is not true anymore — Abhishek Iyer, founder of ACME.BOT”
Getting the distinction wrong means that you’re optimizing for the wrong game completely. The work really begins when you know which pathway governs ChatGPT’s training and which governs Google AI Overviews.
Two Paths, Two Different Games
Your content arrives to AI systems in two very different ways. Path one: frozen snapshot. Website → datasets → model pretraining/fine-tuning. Here, content is frozen at training time in datasets, without any follow-up updates. Path two: live retrieval. Website → crawlers/indexes → retrieval systems → AI answer. For example, Google AI Overviews functions as a retrieval-augmented generation system, where the AI first searches a live index for relevant content before generating an answer. Interestingly, Perplexity’s PerplexityBot is noted as explicitly not being used for foundation-model training; it only surfaces pages in search results. Similarly, OpenAI’s OAI-SearchBot surfaces websites in ChatGPT’s search features, while ChatGPT-User visits pages when users ask questions — both are live retrieval, not training.
| Aspect | Pathway 1 | Pathway 2 |
|---|---|---|
| Update cycle | Locked at training time | Live at query time |
| Key players | GPTBot, CCBot (training crawlers) | Google AI Overviews, Perplexity, ChatGPT search |
| Optimization | Get into training data | Stay crawlable and indexed |
If Pathway 1 is a book printed once, Pathway 2 is a search engine crawling to an answer engine. For most of 2026, you’re optimizing for Pathway 2 — which means crawlability, indexing, and content quality matter more than betting on the next model checkpoint.
What AI Crawlers Actually Look For on Your Page
AI crawlers extract meaning by parsing the HTML structure of a page using actual tables, lists, and heading tags. They also read the hierarchy of headings—following an H1 tag with an H4 tag is breaking the semantic outline that the AI will follow. In addition to HTML parsing, some crawlers also render JavaScript. Google, for instance, does this, but many don’t, so be careful with putting important content in the client-side rendering. There are no LLM parsers just yet—different crawlers will read pages in different ways, so the best way to support any system traversing your site is to have a clean structure.
Heading hierarchy (H1→H2→H3): No skipping levels. Use tags and not styled divs.
Real list and table markup: If you want something to be a list, use
- ,
,
. Don’t use paragraph breaks. For tables, use
markup instead of divs that are styled like tables. This will let crawlers know what’s a list and what’s a table.
Avoid hiding content behind JavaScript: Although Google renders JavaScript, many crawlers may not. To ensure your essential content is crawled, include it in your HTML.
According to Stackmatix, websites with full Tier 1 schema markup can increase their appearance in AI Overviews by up to 40%. Content with the right schema markup is 2.5× as likely to be shown in AI answers.
No matter what crawler pulls up your page, clean structure is the for a guaranteed parse hit. Instead of guessing the right parser, you’re simply making things easier for any system to read your page.
Answering Some Common LLM Visibility Questions
Don’t worry about blocking CCBot when it comes to AI visibility. This bot may be used by some LLMs for pretraining data, but it doesn’t pull up any live, crawled content in AI answers. So, it’s not influencing direct citations to your site. Meanwhile, Googlebot and PerplexityBot do pull up real citations. You don’t need llms.txt or other “special” file types so long as you’re not blocking the content’s eligibility to be indexed. And, because LLMs are non-deterministic — with each user seeing a different set of results — tracking individual citations on AIs is fruitless. Instead, focus on topical authority.
Q: Will blocking CCBot hurt my AI search results visibility/AI features?
No. CCBot may be used by some LLMs for pretraining datasets, but it’s not used for live AI search results. PerplexityBot is responsible for surfacing website links in Perplexity results. This is separate from training a foundation model. Googlebot indexing continues to dictate eligibility for Google AI features, and blocking CCBot should not impact AI Overviews or Perplexity citations.
Q: Do I need llms.txt?
No. Google explicitly notes that other than being eligible for Search indexing, there are no additional technical requirements. All you need is standard indexing.
Q: Should I obsess over tracking AI citations?
This isn’t something you should obsess over right now. Here’s why:
“Directly tracking AI citations imo is meaningless at this point. Tech and LLMs are changing rapidly. Every user sees a different result. Also, LLMs are non deterministic.” — Abhishek Iyer, founder of ACME.BOT
Each session produces unique results. Fan-out query coverage and building topical authority are the things that move the needle for AI search visibility.
What Actually Moves the Needle for AI Search Visibility
A page that doesn’t rank in traditional search can’t get cited by AI, so SEO fundamentals are critical and non-negotiable. Above that, what we’re seeing move the needle for AI visibility is 1) topical authority (with coverage of long tail queries), 2) recency (with content recently updated cited 38% more than content updated a month ago ), 3) branded mentions correlate with an AI citation, and 4) clean content structures make it easier for LLMs to surface your answers. Importantly, our research at ACME.BOT proves these signals work in tandem — we don’t see them as a substitute for one another.
“In my opinion, long-tail / fan out query visibility is the most important metric for AI search. It’s not enough just to rank for the main query anymore, you need to address fan out queries as well.”
— Abhishek Iyer, founder of ACME.BOT
The real meat is in fan-out coverage. Ranking for the main query gets you in the game; addressing the surrounding questions builds the topical authority that retrieval systems like Perplexity actually reward when selecting sources to cite.
Recency is a powerful lever. Perplexity is reranking content that is less than 3 months old with way higher weight. Content that was updated two hours ago is 38% more likely to be cited than content updated a month ago.
Authority still moves the needle. According to GoodFirms, branded mentions still have a solid 0.664 correlation with AI citations — so even in an LLM-powered world, trust still matters.
Structure is important too — clean HTML, direct answers, self-contained sections. LLM don’t actually reward clever prose, they reward content they can parse.
The Takeaway: Two Pathways, But One Core Set of Fundamentals
Before optimizing for one of these two distinct pathways, it’s important to understand that they truly each have different rules. That said, Google says that AI Overviews are eligible for indexing in the same way that any normal webpage would be. So, crawlability, structured data, content quality, and other SEO fundamentals still stand for either pathway. Things are moving quickly, however.
As of now, AI summaries show in around 50% of Google searches, expected to ring in at over 75% by 2028.
You don’t necessarily have to have perfect crawl or exotic signals to be picked up by these technologies. It’s more about having crawlable content and SEO hygiene, and breadth and coverage in the topics and long-tail search queries you serve. If you do that, and build content that has topic depth, long-tail query coverage, and ownership, then AI citations will naturally come with time.
About the editors
Read next