How AI Decides Which Pages to Search & Cite

Retrieval augmented technology is a framework that determines which content material AI instruments retrieve and cite earlier than producing a solution.

It’s essential perceive RAG as a result of it’s one of many methods ChatGPT, AI Mode and different AI engines like google select which pages get included in its reply.

This information explains how RAG works (in plain English), what makes content material extra prone to be retrieved, and easy methods to measure your visibility in AI programs that use RAG with Ahrefs Model Radar.

Retrieval augmented technology (RAG) is a way the place an LLM queries an index—like a search engine, data base, or vector database—to search out further, contextually related data for its response—moderately than simply defaulting to what it realized throughout coaching.

Massive language fashions are educated on large datasets, however that coaching has a cutoff date.

Ask an AI mannequin what occurred final week, or what’s in your reside manufacturing database, and also you’re asking it to work from reminiscence with no reference materials in entrance of it.

Whenever you question an AI mannequin on data it doesn’t but have, that’s when it’s most probably to go rogue and begin telling you that poison is nice for you…

That is in any other case referred to as an AI “hallucination”.

RAG offers AI fashions entry to the right, up-to-date materials to keep away from this destiny.

With RAG, LLMs are both supplementing or overriding their inner data—referred to as their “parametric reminiscence”—in an try to present a extra dependable reply.

This course of can also be typically referred to as “grounding”; anchoring the response to particular sources so the mannequin isn’t simply freestyling from its coaching information.

The three phrases map to the three levels of the method:

• Retrieval: The AI mannequin runs a search question to search out (or retrieve) related content material
• Augmented: It provides that retrieved content material to its enter (augmenting its data)
• Era: It makes use of the question and the retrieved content material to put in writing (or generate) a response

Most AI instruments use each RAG and educated data in tandem.

Most AI instruments have at the very least two issues working below the hood: the bottom mannequin generates language from patterns realized throughout coaching. The retrieval layer goes on the lookout for sources to connect.

Moving into the bottom mannequin’s data means being a part of its coaching information, and that isn’t one thing you’ll be able to simply management.

However entering into the retrieval outcomes is, in some ways, an extension of search engine optimisation.

Each RAG-powered AI reply follows three steps: search, retrieve, generate.

To grasp the main points of what doubtless happens at every stage, right here’s what we learn about how ChatGPT retrieves its sources.

Step 1: The AI decides whether or not or to not run a search

Earlier than something will get retrieved, the AI will resolve whether or not it even wants to complement its data with outdoors information.

Easy fact-finding queries like “What’s a VPN?” can normally be fielded by the core mannequin based mostly on its present coaching data. No retrieval wanted.

In ChatGPT’s case, a smaller classifier mannequin (a part of the “sonicberry” system in response to David McSweeney, who put within the work to search out out simply how ChatGPT retrieves sources) will run first, assigning chance scores to find out whether or not a question wants: no search, a easy search, or a advanced multi-step search.

Different AI instruments will deal with this step in a different way, however all of them share the identical underlying logic: not each question triggers a search.

Step 2: The AI runs a search

At any time when somebody asks ChatGPT a query that requires extra context, it expands that question into a number of associated queries, then sends them to an exterior search index like Bing or Google to gather outcomes.

That growth course of is named question fan-out (do not forget that for later).

As soon as a number of pages are collected, on-page search engine optimisation components just like the title, meta description/abstract, and URL decide which web page will get learn in full, in response to analysis by AI Knowledgeable Dan Petrovic.

From there, he found that sources are shortlisted for scraping based mostly on “relevance, authority, recency, and variety of perspective”.

Step 2: Content material will get damaged into chunks—and the closest match wins

Earlier than it may be totally retrieved and served within the response, the scraped internet content material will get damaged into smaller items known as chunks.

Consider chunking like tearing a e book into particular person chapters. The system breaks the web page into items, then asks which piece greatest solutions the query.

ChatGPT converts the search question and every chunk right into a numerical illustration of that means, referred to as an embedding, then measures their cosine similarity—a rating of how semantically shut they are.

The only solution to image this: think about an enormous map the place comparable concepts sit shut collectively and unrelated concepts are far aside. On this map, “canine” and “pet” could be close to one another. “Canine” and “skateboard” could be on reverse ends.

Embeddings are like GPS coordinates on that map—every bit of textual content will get assigned coordinates based mostly on its that means.

Cosine similarity is the measure of how shut collectively two units of coordinates are.

The AI retrieves the chunks whose coordinates are closest to the fan-out question’s coordinates, and the closest match wins.

That is why particular, clear language helps retrieval—it’s simpler to map to the right vector “coordinate”.

Step 3: Retrieved content material masses into the AI’s working reminiscence—briefly

The matching chunks are loaded into the AI’s context window—its short-term working reminiscence—alongside the consumer’s unique query.

It synthesizes a solution utilizing each, then it deletes the chunks.

Dan Petrovic examined this instantly: he requested an AI mannequin to retrieve data on a well known individual, then in a follow-up message requested it to recall a particular snippet from its sources. It couldn’t.

The uncooked content material is “purged” the second a response is generated.

RAG and coaching information typically get conflated, however they do very various things.

Coaching information is what builds an AI mannequin’s parametric reminiscence—the interior data I discussed earlier.

This occurs throughout pretraining, when the mannequin learns from an enormous, basic corpus of textual content scraped from the net and different sources (e.g. CommonCrawl), and may occur once more throughout fine-tuning, when a mannequin is additional educated on a narrower dataset to shift its habits or data.

Both method, that data will get baked into the mannequin itself. No lookup wanted—it’s simply a part of what the mannequin “is aware of”.

However you don’t get a say in it. It occurs on the mannequin developer’s schedule, utilizing no matter information they select to coach on. It’s not one thing you’ll be able to request, goal, or confirm occurred in your content material.

RAG, alternatively, is a course of you might have some management over. The standard, construction, and indexing of your content material instantly impacts whether or not it will get retrieved.

At any time when a consumer’s question triggers a retrieval step, the mannequin pulls in present data from outdoors information without having to be retrained.

For many industrial AI instruments, that is the mechanism behind most modern solutions they offer you.

	RAG	Coaching information
The way it works	Retrieves exterior content material at question time—by no means turns into a part of the mannequin	Content material is absorbed into the mannequin’s parameters throughout coaching, turning into a part of what it “is aware of” internally
Replace price	Low. Updates the data base and the mannequin’s subsequent reply displays it	Excessive. Solely modifications when the mannequin is retrained—on the developer’s schedule, not yours
One thing you’ll be able to affect?	Sure. Content material high quality, indexing, and construction all have an effect on whether or not your content material will get retrieved to floor present data, cite you as a supply, or point out your particular model	In a roundabout way. You may publish content material and hope it will get crawled however, not like RAG, you’ll be able to’t optimize one web page and hope to see it cited. What you are able to do is construct a constant model narrative throughout sufficient content material over time that it shapes how future fashions describe you.

All the pieces we cowl on this article—main with definitions, together with entities, Q&A construction, freshness—instantly influences the retrieval course of.

That’s the layer you’ll be able to truly affect by means of content material.

Being cited in an AI response is a win, but it surely’s not the identical as being baked into what an AI essentially is aware of about your model.

Search Consumer Optimization skilled Dorron Shapow places it nicely:

Optimizing for retrieval isn’t flawed. In programs that rely closely on reside seek for industrial queries, it may possibly completely affect what will get surfaced. However assuming retrieval visibility is similar as foundational mannequin weighting is the place the technique breaks. One takes weeks. The opposite is the gradual work of entity coherence—how persistently and clearly your model is known throughout the broader internet—and it takes years.”

Question fan-out is the method that occurs behind the scenes once you submit a question to an AI system.

Fairly than looking your actual phrases, it breaks your query into a number of associated sub-queries, runs each individually, and pulls sources from the mixed outcomes.

Say somebody searches “What's going to occur if I swap out common flour for wholemeal flour in a lemon drizzle” in Google, the underlying AI search mannequin wouldn’t simply search that phrase, it may additionally look up:

• Greatest flour for lemon drizzle
• Baking with wholemeal flour ideas
• How does wholemeal flour have an effect on cake density?

Earlier than synthesizing a solution.

AI does the identical factor robotically on most advanced queries.

Some SEOs have been in a position to extract these inner sub-queries instantly.

As an illustration, Metehan Yeşilyurt has developed a way to immediate Google AI Mode into outputting the search queries it used for grounding.

However if you happen to don’t have time to go digging, it’s also possible to see the fan-out queries generated by ChatGPT, Grok, and Perplexity within the AI Responses report in Ahrefs Model Radar.

Throughout question fan-out, the AI splits your query into smaller sub-queries, searches all of them without delay, combines and re-ranks the outcomes, then merges the pages that do nicely throughout a number of searches into one remaining checklist.

That checklist is what the AI truly reads to put in writing your reply.

We’ve simplified the fan-out course of right here for ease of understanding, however for a deeper-dive learn our information: What’s Question Fan-Out? Understanding the Hidden Queries Driving AI Search.

For ChatGPT and different AI engines like google, Retrieval Augmented Era runs on search engine optimisation.

The truth is, many entrepreneurs and SEOs view AI search as a “wrapper” on prime of “conventional” engines like google like Google, since some AI assistants draw so closely from them.

When ChatGPT, Perplexity, or Google AI Overviews have to reply a query, they run precise internet searches ¹ ² ³

Google Gemini and AI Overviews use Google Search. Microsoft Copilot makes use of Bing. ChatGPT pulls from each Google and Bing. Claude makes use of Courageous Search.

Which means the retrieval layer of each main AI instrument is powered by a standard search engine.

• Listed content material is the beginning pool. You want your content material to point out up in Google earlier than it reveals up in AI.
• Search optimized content material will get you cited: Even when search and AI outcomes don’t at all times neatly overlap, each prioritize authoritative, well-structured, well-optimized content material.
• Model mentions in search correlate strongly with AI visibility: AI programs choose up on how typically and the place your model is referenced throughout the net—search-optimized content material and digital PR instantly feeds this ¹

Regardless of some variations, search engine optimisation and GEO are intrinsically linked.

In case your content material doesn’t present up in a search index, an AI bot goes to have a tough time discovering it, and if it may possibly’t discover it, it may possibly’t retrieve it.

Observe these seven greatest observe ideas if you wish to get your content material cited in RAG search.

Be sure content material is accessible to AI crawlers

After they exit to fetch content material, many AI crawlers are unable to learn and cite sure pages.

JavaScript content material (like tabs or accordions) or textual content in photos is usually inaccessible to AI bots.

As an alternative, AI programs retrieve static HTML content material.

Right here’s what occurs when a web page comprises JavaScript.

Suganthan Mohanadasan lately tapped into the community recordsdata of dozens of ChatGPT conversations, and studied the mannequin’s chain-of-thought course of, the place it describes the way it sources data in layman’s phrases.

For a related B2B SaaS question, ChatGPT situated official pricing for Ahrefs however struggled to search out costs for Profound and Peec, reasoning that this data was hidden inside JavaScript.

ChatGPT deferred to third-party sources like G2 since “the official web page is tough to parse and doesn’t present costs”.

The ethical of the story: if you’d like your most essential data—like your pricing— to be precisely portrayed in AI search, your content material ought to ideally be served through HTML, not JavaScript.

JavaScript isn’t the one solution to lock a crawler out—you additionally have to keep away from blocking AI crawlers (like OAI_SearchBot) in your robots.txt and firewall guidelines if you wish to be cited through retrieval ¹ ².

If you happen to use Cloudflare, you’ll be able to monitor how AI bots are crawling your web site—together with which pages they go to most frequently and which of them they miss—through Ahrefs Bot Analytics.

Lead together with your greatest data

AI pays probably the most consideration to the start of your web page, however its consideration drops steadily from there.

In accordance with Kevin Indig’s examine of 1.2 million ChatGPT citations, the primary 30% of a web page’s content material generates 44.2% of all citations.

The center third generates 31.1%, and the underside third: simply 24.7%.

Your most essential data—definitions, key claims, distinctive information—must be on the very prime of your content material.

That is the alternative of the standard “save the most effective for final” strategy. In content material optimized for AI citations, the punchline goes first.

This is named serving the Backside Line Up Entrance (BLUF).

Reply the question instantly within the first sentence beneath the subheading—don’t bury the reply two paragraphs in.

This instantly mirrors how RAG programs match content material to queries—but in addition, how customers learn, so that you’re satisfying each beings and bots alike!

This eye-tracking information reveals readers focus probably the most consideration on the very prime of a web page and scan much less and fewer as they transfer down, so in case your key takeaway is buried in paragraph three, most readers by no means truly see it—therefore, “backside line up entrance”.

Optimize for fan-out subjects

To point out up within the fan-out outcomes that AI programs draw on, it’s useful to create subject clusters—the associated questions, definitions, comparisons, and subtopics that AI may seek for whereas getting ready a solution.

If you happen to’re on the lookout for hints as to what these sub-topics is likely to be, faucet into “Individuals additionally ask” bins and “Individuals additionally seek for” queries on the backside of Google.

They mirror the most-asked questions and angles round your subject, which are typically much like the queries AI generates in a fan-out.

Optimize your web page pace

Sluggish pages are unhealthy information in any search engine, however in AI search the associated fee is even steeper.

In his breakdown of how ChatGPT works, search engine optimisation Guide David McSweeney notes that ChatGPT seems to fetch grounding pages on a tough timeout of round two seconds: in case your server is gradual, your web page will get minimize, and even when it responds in time, a excessive time-to-first-byte (TTFB) means your content material will get truncated.

Underneath 1 second TTFB: you’re in all probability advantageous. Your full web page has time to load, get chunked up, and fed to the mannequin.

Over 1 second: you’re playing. The connection may get minimize mid-download—typically so early that solely your

tag made it by means of, that means the mannequin by no means even noticed your precise content material.

Pace decides whether or not you make it into the mannequin’s context window at all.

Examine your time-to-first-byte in Website Audit.

1. Head to the Efficiency report
2. Discover the “Time to first byte distribution” chart
3. Click on “Medium: 200–300 ms” in your quick-win optimization alternatives

Then kind by natural visitors to search out your most essential content material which will have to be optimized

In case your server is simply too gradual, your web page could by no means make it into an AI reply—however in some circumstances you’ll by no means know, as a result of the customer (on this case, a bot) merely gave up and left.

Jan-Willem Bobbink appears to be like for cases of this by figuring out the HTTP standing code 499 in his server logs.

A 499 standing code means the consumer closed the connection earlier than the server completed responding.

That is one other clear sign that your web site is simply too gradual for AI retrieval.

Create deep, entity-led content material

The content material that will get cited most frequently through RAG search comprises roughly 20.6% entity density.

Which means, 20.6% of its phrases are correct nouns—named instruments, manufacturers, folks, firms, research—in comparison with 5-8% in “common” content material.

An entity is any particular named factor. For instance, “An search engine optimisation instrument” is just not an entity— however “Ahrefs” is.

The extra named entities you embrace, the extra anchor factors your content material has on the that means map—making it retrievable for a broader vary of associated queries.

However you’re not going to win citations by randomly “entity stuffing”. Your content material, and its entities, have to be related to the consumer’s question.

Right here’s another excuse entities matter.

Fan-out queries typically use a “synonym cloud” method to steer retrieval in the direction of particular angles and entities, and in the end higher match the intent of the consumer’s unique question.

For instance, ChatGPT’s frontier mannequin could rework a question like “What are the ten greatest trainers?” into synonym-rich fan-out queries like:

• greatest trainers 2026
• evaluations trainers
• prime picks
• awards

To nudge the embedding towards “better of” intent, as seen beneath through Model Radar.

So what does this imply in your content material?

Nicely, to paraphrase David McSweeney: Generic pages that point out every thing rating okay throughout the board.

However specialised pages that go deep on one angle win that angle outright.

Getting cited is due to this fact about anchoring your content material to particular entities.

Right here’s a sensible solution to enrich your content material with entities: undergo your again catalog and change generics with specifics.

Change:

• “A search engine” → “Google”
• “Analysis suggests” → “A 2024 examine from Waseda College discovered”
• “An AI assistant” → “ChatGPT” or “Perplexity”

You may confirm your work utilizing Google’s Pure Language API.

The free demo model reveals you each entity Google detected in your web page, and the class it assigned your content material to.

If you happen to pay for full entry, you’ll additionally get the salience rating—a price for a way outstanding and essential Google thinks an entity is to your web page.

Run the API in your web page, then run it on the top-ranking web page in your goal key phrase.

The hole between these two outputs offers you your entity optimization guidelines:

• Entity crossover
• Entity gaps
• Salience scores (greater when the subject is called earlier and extra prominently)
• Class crossover
• Class gaps

Alternatively, run your draft by means of Ahrefs’ AI Content material Helper.

It grades your content material in opposition to your prime rivals in your goal key phrase and highlights the subjects they cowl that you just’re lacking—helpful for catching subject gaps which may make you invisible in fan-out outcomes.

Add data acquire—say one thing the mannequin doesn’t already know

Entity protection will get you retrieved, however there’s one thing that comes earlier than that: does your content material even qualify for retrieval within the first place?

A leaked Claude system immediate revealed that AI programs like Claude have a never_search command for queries about “timeless or steady” data.

Claude solutions never_search questions from coaching information alone, and doesn’t go on the lookout for exterior URLs to cite.

Development Advisor Gaetano DiNardi thinks different LLMs are doubtless following the identical logic. In his phrases:

“the worth of publishing pages on generalized data is zero.”

That is the knowledge acquire downside.

Consider every thing a mannequin already is aware of because the overculture—the averaged-out, consensus model of a subject that’s been listed 1000’s of occasions.

In case your content material solely restates it, you’re redundant from the RAG framework—an AI mannequin has nothing to realize from citing you.

What it does cite is content material that provides one thing new: proprietary information, a named concept, a particular discovering from a examine, a conclusion the mannequin can’t synthesize from its present data base.

OpenAI researcher Karthik Narasimhan printed a paper on Generative Engine Optimization that provides additional proof of this.

Together with friends at Princeton College, he studied which strategies are most probably to spice up visibility in RAG AI programs like Perplexity.

Their findings revealed that web sites that includes distinctive data like quotes and statistics have been mostly referenced; seeing 30-40% visibility uplift in AI responses.

LLMO methodology examined	Place-adjusted phrase depend (visibility) 👇	Subjective impression (relevance, click on potential)
Quotes	27.2	24.7
Statistics	25.2	23.7
Fluency	24.7	21.9
Citing sources	24.6	21.9
Technical phrases	22.7	21.4
Simple-to-understand	22	20.5
Authoritative	21.3	22.9
Distinctive phrases	20.5	20.4
No optimization	19.3	19.3
Key phrase stuffing	17.7	20.2

Kevin Indig additionally discovered that date and quantity are the entity sorts that predict ChatGPT citations most.

And Eric Lancheres studied 150 rating pages and located the most important rating predictor was their variety of distinctive information factors.

Having your content material retrieved is a matter of surfacing recent data and distinctive information, not chorusing what different pages have already coated.

Embody a question-and-answer construction

Content material structured as query → instant reply is cited twice as typically as content material that doesn’t observe this conference (18% vs. 8.9%), in response to Kevin Indig’s information.

That is one more instance of BLUF in play.

AI fashions attempt to match consumer queries (nearly at all times a query) to a piece that solutions it.

Within the phrases of Suganthan Mohanadansan:

“Citations bind to a particular sentence, not the entire reply, so being topically related isn’t sufficient, it’s important to be the most effective assist for a exact declare.”

Formatting your content material as a Q&A might help AI fashions like ChatGPT make a direct, unambiguous match.

Mohanadasan additionally discovered that ChatGPT deduplicates outcomes by area, so 20 skinny pages in your web site don’t add as much as 20 probabilities at quotation.

ChatGPT selects the one web page that greatest matches the consumer’s preliminary question and fan-out subqueries.

Put your strongest reply on that web page, not unfold throughout all 20.

Hold content material recent

RAG search programs have a desire for present content material.

We ran a examine of 17 million citations, and located that AI assistants persistently choose to quote more energizing content material than engines like google.

URLs cited by AI assistants are 25.7% more energizing on common than URLs in commonplace natural SERPs—and ChatGPT and Perplexity truly order their citations from latest to oldest.

However don’t simply take our phrase for it. Freshness is a confirmed, documented sign in AI retrieval.

Metehan Yeşilyurt’s analysis confirmed this. He found that ChatGPT has a configuration setting known as use_freshness_scoring_profile: true , which bakes in a scientific recency bias.

So, your content material has a a lot better likelihood of being retrieved and finally cited if you happen to replace your key pages usually.

Even minor updates can reset the freshness sign. Refresh statistics and examples yearly and add a visual “final up to date” date.

Optimizing your content material for RAG is significant, however it is advisable know if it’s working.

Ahrefs Model Radar was constructed to assist manufacturers monitor their visibility in retrieval augmented AI outcomes.

Right here’s how I recommend utilizing it to enhance your visibility in RAG.

Observe your baseline visibility

Earlier than altering something, discover out the place you truly stand.

Search your model in Model Radar to see how typically you’re showing in AI solutions in your goal subjects, and which platforms are citing you.

If mentions are low or absent, see who’s being cited as an alternative.

Discover out which AI platforms are citing you (and which aren’t)

Totally different AI platforms have completely different retrieval architectures with completely different biases towards freshness, authority, and construction.

Model Radar’s platform breakdown can reveal gaps like “AI Mode cites us usually, however we lack visibility in Perplexity.”

In case your web site performs badly on just one platform, the difficulty is probably going with how that platform evaluates it—not the content material itself.

For instance, if a web page ranked nicely on Google however not on Bing, we’d see that as a Bing-specific sign (like hyperlinks, entities, or indexing) moderately than the web page being low high quality total—the identical is true of AI visibility.

Uncover which queries are triggering your citations

Seeing the precise queries that result in quotation tells you what’s working, and flags associated queries the place you’re not showing but.

Due to question fan-out, you could already be getting cited for queries you’d by no means have thought to focus on.

Model Radar’s database comprises thousands and thousands of present queries, that means you’ll be able to discover new content material alternatives you wouldn’t in any other case know existed.

Observe whether or not content material updates change your quotation fee

When you’ve made modifications to optimize your content material for retrieval—making use of BLUF, focusing on fan-out queries, incorporating statistics—monitor Model Radar to see whether or not your citations develop within the following weeks.

This allows you to construct a suggestions loop: optimize → publish → measure → iterate.

The identical type of methodology that works for monitoring natural rankings additionally applies to AI quotation monitoring.

Benchmark in opposition to rivals

Discover out which of your rivals is being persistently cited by AI for queries you care about, then analyze the construction and content material of their most-cited pages.

Simply add a Your model: Not talked about and Your model: Discovered however not cited filter to an AI Responses or Cited Pages report in Model Radar.

This can present you the subjects and third-party discussions your model tends to be ignored of.

Then it’s only a case of reverse-engineering your rivals’ strikes to shut the hole.

RAG is the bridge between search and AI. It follows predictable guidelines, selling pages it may possibly entry, fetch shortly, and topic-match instantly to present the very best reply.

Observe your AI visibility with Ahrefs Model Radar to see whether or not your content material is exhibiting up throughout ChatGPT, Perplexity, Google AI Overviews, and the opposite instruments your viewers truly makes use of.

Obtained questions? Ping me on LinkedIn.