How AI Answer Engines Choose and Cite Sources: A Mental Model

AI answer systems that cite sources generally work in stages: decide whether to search, split the question into queries, retrieve pages from a search index, filter what to read, write an answer, and attach citations. The vendors document the access and eligibility rules, but none publish how they rank candidates internally. This guide builds a six-stage model from official documentation (checked on 13 November 2026) and marks what you can control.

Key Takeaways

• Not every question triggers a search. Anthropic's web search tool documentation says Claude searches when a request depends on current, changing or outside-training information, and answers directly for stable knowledge.
• One question can become many searches. Google says its AI features may issue multiple related searches across subtopics (query fan-out), per its AI features guide. Anthropic says single-fact queries typically use 1 to 3 searches and comparative research can use 10 or more.
• Eligibility comes first. Google says a page must be indexed and eligible to be shown in Search with a snippet to appear as a supporting link, and that no extra technical requirements or special optimizations exist.
• Each vendor uses a named search crawler: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, per their documentation. Blocking the search crawler affects that product's results.
• What happens between retrieval and citation is not public. Anthropic documents that its tool can filter results with code before they reach the model and that citations carry up to 150 characters of cited text.
• Site owners now have an explicit exclusion control in Google. Per its Search Console help page, a generative AI control reached all websites worldwide as of August 31, 2026.

The six-stage model

This is my model, assembled from vendor documents. It is a simplification. Each product differs, and the vendors reserve the details.

Stage 1: the trigger

Anthropic's documentation lists what makes Claude search: recent events, current prices and statistics, information about organizations or products that may have changed, and explicit requests to look something up. Stable facts, math and creative writing get direct answers. The same logic explains why some of your pages never appear in AI answers: if the question can be answered from the model alone, no retrieval happens.

Google's phrasing differs but points the same way. AI Overviews appear only when its systems judge them additive to classic Search, so they often do not trigger. Pages about changing facts (pricing, rules, product comparisons, current guidance) are exposed to retrieval more often than pages about settled definitions.

Stage 2: query planning

Google says AI Overviews and AI Mode may issue multiple related searches across subtopics and data sources, and that this lets them show a wider and more diverse set of links. Anthropic's documentation includes a max_uses control that caps searches per turn and notes that comparative or multi-entity research can use 10 or more. The consequence is the same: a page can be retrieved for a query the user never typed. The longer treatment is in Query Fan-Out Explained.

Stage 3: retrieval from an index

The retrieval layer is mostly ordinary web search. For Google, eligibility is the standard Search requirement: indexed, and eligible to be shown with a snippet. Google's guide recommends allowing crawling in robots.txt and through any CDN or hosting layer, using internal links, keeping important content in text and keeping structured data consistent with visible text.

The other vendors publish crawler pages:

• OpenAI's crawler documentation recommends allowing OAI-SearchBot and its published IP ranges for search visibility, and says opted-out sites will not be shown in ChatGPT search answers, though they may appear as navigational links.
• Anthropic's crawler help article says Claude-SearchBot improves search result relevance and that blocking may reduce visibility.
• Perplexity's bot page says PerplexityBot surfaces and links sites in search results and is not used for foundation model training.

The robots.txt mechanics are in Robots.txt Rules for Search and AI Crawlers.

Stage 4: reading and filtering

This stage explains why a page can be retrieved and still not cited. Anthropic's documentation says that with the dynamic filtering versions of its tool, Claude writes and runs code that filters search results first so only relevant content reaches the context window. That is one vendor's implementation, and I have no equivalent statement from the others. It does show that a retrieved page is not automatically read in full.

Developers can also restrict where results come from. Anthropic's tool accepts allowed_domains or blocked_domains lists. If an app built on a search API limits itself to a set of domains, ranking well elsewhere changes nothing for that app.

Stage 5 and 6: composition and attribution

No vendor documents how it decides which retrieved passage supports which sentence. What Anthropic does document is the output: each citation includes the URL, the title and up to 150 characters of cited text, and results carry a page_age field describing when the site was last updated. Anthropic also says that citations must be shown to end users when displaying API outputs directly.

For AI Overviews and AI Mode, Google says the features surface links to help people explore content, and that the two may use different models so their links vary. That variation is why a single check of a single prompt tells you little. The decoupling of citations from classic rankings is covered in AI Overview Citations Have Decoupled From Top-10 Rankings.

A hypothetical walk-through

This example is invented to show the model, not observed behavior.

A user asks an AI assistant, "Which project management tool is best for a five-person agency?" Walking the stages:

1. Trigger: the question depends on current products and prices, so a search is likely.
2. Planning: the system may search for several sub-questions: pricing tiers, team-size limits, reviews, integrations.
3. Retrieval: pages that answer those sub-questions and are crawlable by the product's search crawler enter the candidate pool.
4. Filtering: pages with clear tables or direct statements are easier to use than pages that bury the answer.
5. Composition: the answer restates a few of those statements.
6. Attribution: two or three sources get links.

A vendor page that says "pricing on request" and blocks the search crawler fails at stage 3. One with a clear comparison table and a visible update date has a better chance at stages 4 to 6. Whether it is chosen at all is outside your control.

What this means for marketers and founders

• Fix stage 3 first. Confirm the search crawlers are allowed at the CDN and in robots.txt, and confirm key pages are indexed with snippets. This is cheap and everything else depends on it. The technical SEO checklist for AI crawlers lists the checks.
• Write for sub-questions. If a page can answer the three or four things a buyer would ask next, it matches more fan-out queries.
• Put the answer where it can be lifted. Short, specific statements with a number, a date or a named source are easier to restate than long narrative.
• Measure by prompt panel, not by single prompt. Use a fixed set of questions over time; Prompt Tracking: Build Your First 50-Prompt Panel shows the method.
• Do not treat a citation as a recommendation. A cited page is a source for a sentence, not an endorsement. See Citations Are Not Recommendations.

What we do not know

• How any vendor ranks candidates before reading them. None document it.
• Whether the stages are the same across products. Anthropic's and Google's documents describe different systems, and I have merged them for clarity.
• How often a retrieved page is dropped at the filtering stage. No vendor publishes that rate.
• Whether blocking a training crawler affects what the same company's assistant says about you. The documents I read treat search and training as separate settings and do not promise either outcome.

FAQ

How do AI search engines pick which websites to cite?

Vendors do not publish the ranking details. They do document the steps around it: the system decides to search, runs one or more queries, retrieves pages from a search index through its search crawler, filters them and then attributes sources. Access and indexing are the parts you can verify.

Do I need special optimization to appear in Google AI Overviews?

Google says no additional requirements or special optimizations are necessary. A page needs to be indexed and eligible to appear in Search with a snippet. Google also says you do not need new machine-readable files or special schema markup for these features.

Why is my page ranking well but not cited by AI?

Ranking and citation are separate steps. The system may issue different queries than the one you rank for, may filter your page out when reading results, or may be limited to certain domains. Check access first, then whether the page answers the sub-questions.

Does blocking GPTBot remove me from ChatGPT answers?

OpenAI documents GPTBot as the training crawler and OAI-SearchBot as the search crawler, and says each setting is independent. Blocking GPTBot does not remove you from ChatGPT search results according to that page; blocking OAI-SearchBot does.

Can I opt out of Google's AI features without leaving Search?

Google says a Search generative AI control in Search Console rolled out worldwide as of August 31, 2026. Excluding content stops it appearing in AI Overviews, AI Mode and Discover generative AI features, and means no impressions or traffic from them. It does not change AI training, which Google-Extended governs.

How many searches does an AI assistant run per question?

It depends on the question. Anthropic says single-fact queries typically use 1 to 3 searches and comparative or multi-entity research can use 10 or more. Google says its AI features may issue multiple related searches but gives no number.

Is llms.txt part of this pipeline?

None of the vendor documents I read for this guide describe llms.txt as a retrieval requirement, and Google says it does not require new AI text files.

Want help with this?

I'm Younus Fardeen, an organic growth, SEO and AEO strategist with 4+ years of marketing experience across edtech and startup teams. If you want help auditing how AI answer engines find and cite your pages, see my work and reach me through the contact form at younusfardeen.com.