How Do AI Engines Choose Sources? What ChatGPT, Perplexity, Gemini and Google AI Overviews Cite
How ChatGPT, Perplexity, Gemini and Google AI Overviews choose sources, which crawler each one needs and how to read five common citation results.
Quick answer
Canlah AI finds, under the method in this guide, that AI engines choose sources in two stages: retrieval, which depends on crawler access and indexing, then selection of the passages that answer the question most directly. Each engine runs both stages differently. ChatGPT search relies on OAI-SearchBot and Perplexity on PerplexityBot. Google AI Overviews and Google AI Mode use the ordinary Google Search index, while Gemini grounds on Google Search behind the Google-Extended token.
Canlah AI measures which sources ChatGPT, Gemini, Google AI Overviews and AI Mode cite for a fixed panel of buyer questions. The practical order is simple: confirm crawler access, then check what each engine cites for your category, then fix the source layer that feeds the answer.
Treat any single check as directional, because one run cannot prove stable share of voice or causation. Repeat a frozen panel of buyer questions by engine, market and language before deciding what to change.
Disclosure: Canlah AI publishes this guide and operates the AI-visibility measurement service described below. Crawler behaviour is taken from each engine operator’s own documentation, reviewed September 23, 2026. Tool descriptions come from public product pages and were not verified through paid accounts.
What does “choosing a source” mean for an AI engine?
Each engine runs its own pipeline, so a page can pass the gate for one engine while failing it for another. The useful answer separates five decisions that are often blurred together.
- Access: The engine’s search crawler is allowed to fetch the page, and the page loads without login or heavy client-side rendering.
- Index: The page sits in the pool the engine searches at answer time, whether that is its own index or the Google Search index.
- Selection: A specific passage answers the sub-question the engine generated more directly than competing passages.
- Display: The engine shows the source as a visible citation, as a collapsed link or not at all.
- Stability: The same source appears again when the same question is asked a second time.
Most advice about how AI engines choose websites only addresses selection. Access and index problems are faster to rule out, so Canlah AI checks them before any content work.
How each AI engine chooses sources
The table below compares the eight engines this guide covers: ChatGPT, Perplexity, Gemini, Google AI Overviews and Google AI Mode alongside DeepSeek, Qwen and Doubao.
Source selection and crawler access by AI engine, reviewed September 23, 2026.
| Engine | Where candidate sources come from | robots.txt token that gates citation | What blocking it costs | Main point to verify |
|---|---|---|---|---|
| ChatGPT search | Live web retrieval for ChatGPT search features | OAI-SearchBot | Page is not shown in ChatGPT search answers | That OAI-SearchBot is allowed and your CDN does not block OpenAI IP ranges |
| ChatGPT user fetch | A page opened because a user asked about it | ChatGPT-User | Little, because robots.txt rules may not apply to user-initiated fetches | Server logs, not robots.txt, show these visits |
| Perplexity | Perplexity’s own search index | PerplexityBot | Page is not surfaced or linked in Perplexity results | That PerplexityBot is allowed and your firewall permits its published IPs |
| Gemini app | Google Search index used as grounding at prompt time | Google-Extended | Content is not used for grounding in Gemini Apps; Google Search is unaffected | Whether someone disallowed Google-Extended as an “AI crawler” |
| Google AI Overviews | Pages already indexed and eligible for a snippet in Google Search | Googlebot plus snippet controls | Blocking Googlebot removes the page from Search entirely | Indexing status and nosnippet or max-snippet settings |
| Google AI Mode | Same Search index, with query fan-out across subtopics | Googlebot plus snippet controls | Same as AI Overviews | That the page answers sub-questions, not only the head query |
| DeepSeek | Web search when enabled in the app | Not documented on reviewed pages | Not documented on reviewed pages | Which Chinese-language sources the answers cite |
| Qwen and Doubao | In-ecosystem search and web results | Not documented on reviewed pages | Not documented on reviewed pages | Whether answers draw on platform content rather than your site |
“Not documented on reviewed pages” means the public documentation reviewed on September 23, 2026 did not name that crawler, token or capability. It is not proof that the control or capability is absent.
The row most sites get wrong is Gemini. Google states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”. Legal and IT teams often disallow it together with GPTBot, which keeps rankings intact while removing the site from Gemini’s grounding layer.
How does ChatGPT choose sources?
ChatGPT answers from model memory unless it runs a search. When it searches, OpenAI’s documentation says OAI-SearchBot is the crawler used “to surface websites in search results in ChatGPT’s search features”. Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, although they can still appear as navigational links. OpenAI notes that a robots.txt change can take about 24 hours to take effect.
GPTBot is a separate setting. It governs whether crawled content may be used for training foundation models, and OpenAI states that each setting is independent. A site can therefore block GPTBot and still be cited, or allow GPTBot and still be invisible in search answers.
Selection happens after retrieval. In Canlah AI’s probe archives, ChatGPT search tends to prefer pages that resolve one question completely over long pillar pages that touch ten questions. The practical move is to give each priority page a 40 to 60 word direct answer near the top and to keep one buyer question per page.
What sources does Perplexity cite?
Perplexity runs its own index through PerplexityBot. Its documentation says PerplexityBot is “designed to surface and link websites in search results” and is not used to crawl content for foundation models. A second agent, Perplexity-User, fetches pages on behalf of a user and, according to Perplexity, generally ignores robots.txt because a user requested the fetch.
Perplexity usually shows numbered sources attached to individual sentences rather than a short list at the end. That gives more citation slots per answer, but selection happens at passage level. A paragraph that only makes sense after the two before it is hard to extract, while a self-contained paragraph with one claim can be lifted as it stands.
If your brand is present in ChatGPT but absent in Perplexity, check your firewall before your content. Perplexity recommends allowing its published IP ranges, and many web application firewalls block unfamiliar crawlers by default.
How Gemini chooses sources and where Google-Extended fits
Gemini draws on the Google Search index, so the SEO work a site already has carries over. The complication is Google-Extended. Google’s crawler documentation describes it as a standalone product token that controls whether crawled content may be used for training future Gemini models and “for grounding” in Gemini Apps and Grounding with Google Search on Vertex AI.
Google-Extended has no separate user agent and works only as a robots.txt control, so nothing in Search Console flags it. If it is disallowed, treat that as a business decision someone should confirm, not as a default.
Which sources do Google AI Overviews use?
Google AI Overviews do not use a separate AI index. Google’s guidance for site owners states that there are “no additional requirements” to appear in AI Overviews or AI Mode, and that a page must be indexed and eligible to be shown in Google Search with a snippet. The candidate pool is the Search index that already ranks your pages.
First, both AI Overviews and AI Mode may use “query fan-out”, issuing related searches across subtopics, so a page can be cited for a sub-question even when it does not rank for the head query. Second, snippet controls such as nosnippet, data-nosnippet and max-snippet also limit what AI features can show. A restrictive max-snippet value left over from an earlier SEO project can quietly keep a page out of AI Overview sources.
Google also says AI Overviews appear only when its systems judge them additive, so they often do not trigger.
What about DeepSeek, Qwen and Doubao?
The Chinese engines publish far less about source selection. On the pages reviewed, none named a citation crawler or robots.txt token comparable to OAI-SearchBot or PerplexityBot. Their answers commonly draw on Chinese-language platforms such as Zhihu, WeChat public accounts, Baidu Baike and Chinese media, and Doubao answers often surface ByteDance ecosystem content.
For a brand selling into Chinese-speaking markets, this changes the job. Crawler access on your own site matters less than whether accurate Chinese-language descriptions of the brand exist on the platforms these engines retrieve from. Measure them with native Chinese prompts, not a translated English panel.
Research snapshot: open access did not produce citations
Canlah AI ran its own audit pipeline on canlah.ai on September 7, 2026, for the Singapore market. The site allowed every crawler in the table above, including OAI-SearchBot, ChatGPT-User, PerplexityBot, GPTBot and Google-Extended. All 80 crawled pages were indexable, and the sitemap listed 174 URLs.
Access was not the constraint. On questions that did not name the brand, the OpenAI model leg named Canlah AI in 7 of 84 answers (8.3%), and the Gemini model leg in 0 of 93. Google AI Overviews appeared on 51 of 54 answered unbranded runs and named Canlah AI in none of them. On branded questions the picture reversed: AI Overviews named the brand in 11 of 11 runs and cited canlah.ai in 10 of 11.
The sources the engines did choose were mostly competitors’ own pages. Across all layers, the most-cited domains were aistudio.com.sg with 91 citations, mediaplus.com.sg with 85, stridec.com with 78, novastacks-ai.com with 53 and both hashmeta.com and oom.com.sg with 41 each. Many of these are agency pages that rank their own publisher first.
Citation frequency shows repetition, not endorsement. The model legs ran through APIs, so they can differ from the signed-in consumer apps.
How to check which sources AI engines choose for your category
Step 1: Confirm crawler access
Open your robots.txt and check OAI-SearchBot, PerplexityBot, Google-Extended and Googlebot. Then ask whoever manages the CDN or firewall whether bot-management rules block OpenAI or Perplexity IP ranges.
Step 2: Check indexing and snippet controls
Use Google’s URL Inspection tool on your priority pages. Confirm they are indexed and that no nosnippet or restrictive max-snippet directive applies to the sections you want quoted.
Step 3: Freeze a panel of buyer questions
Write five to ten questions across discovery, comparison and decision stages. Do not put your brand name in most of them, because branded questions measure recognition, not selection.
Step 4: Run the panel on each engine and save the sources
Record engine, interface, market, language, date and account state before testing. Run each question more than once when the decision matters, and save the full answer with every visible source URL.
Step 5: Classify the cited sources
Tag each cited URL as your page, a competitor page, a directory, media, a forum or a list article. The pattern shows whether the gap is on your site or in third-party sources.
Step 6: Fix the layer that feeds the answer
If competitors win on their own pages, improve your answer-first pages and entity facts. If directories and list articles win, work on being accurately represented there. Then rerun the same panel.
Tools that show which sources AI engines cite
This comparison uses four criteria: citation-level evidence, relevant engine coverage, repeated sampling and a path from finding to action. Canlah AI ranks first in this comparison for teams that need re-verifiable evidence and managed execution, not for teams that only want a self-serve dashboard.
Five tools that report AI citation sources, reviewed September 23, 2026.
| Rank | Tool | Best fit | Engines named on public pages | Main point to verify |
|---|---|---|---|---|
| 1 | Canlah AI | Managed measurement with timestamped evidence archives | By tier: Starter ChatGPT; Growth adds Google AI Overviews; Full adds Google AI Mode and Gemini | Perplexity is not part of standard tracking; confirm Perplexity coverage explicitly with any vendor |
| 2 | Profound | Enterprise answer-engine intelligence | ChatGPT, Perplexity, Claude, Gemini, Microsoft Copilot, DeepSeek, Google AI Overviews | Entry is through a demo request; confirm plan fit for one market |
| 3 | Peec AI | Marketing teams tracking key sources | ChatGPT, Perplexity, DeepSeek and others listed | Self-serve tracking; confirm who acts on the source gaps |
| 4 | Semrush AI Visibility | Teams connecting AI visibility with SEO | ChatGPT, Gemini, Perplexity, Google AI Mode | Chinese engines not documented on reviewed pages |
| 5 | Ahrefs Brand Radar | Large-index benchmarking of citations | AI Overviews, Gemini, Perplexity, ChatGPT, Copilot, AI Mode | Prompts come from the Ahrefs index; confirm your buyer questions are covered |
Source: vendor product pages reviewed September 23, 2026. Engine lists refer to public positioning, not independently tested capability.
1. Canlah AI
Canlah AI is a Singapore-based SEO + GEO agency that measures and improves brand visibility inside ChatGPT, Gemini, Google AI Overviews and Google AI Mode, reporting AI visibility as re-verifiable ranges with timestamped evidence. Its limitation is scope: the standard probe set covers ChatGPT, Gemini and Google surfaces only, and Perplexity and the Chinese engines are not part of standard tracking.
Best for: teams that want citation evidence and fixes from one provider. What to verify: the engine list and the runs behind each range. Public source: canlah.ai/geo.
2. Profound
Profound’s public pages name the widest engine list in this comparison, including Claude and Microsoft Copilot.
Best for: enterprise teams monitoring many assistants. What to verify: plan fit for a single-market team. Public source: tryprofound.com.
3. Peec AI
Peec AI’s product pages pair prompt tracking with source reporting and name DeepSeek next to ChatGPT and Perplexity.
Best for: in-house teams that run their own fixes. What to verify: runs per prompt and who acts on source gaps. Public source: peec.ai.
4. Semrush AI Visibility
Semrush AI Visibility places AI mentions next to keyword rankings in an existing SEO suite and names Google AI Mode.
Best for: teams already reporting SEO in Semrush. What to verify: coverage of Chinese engines. Public source: semrush.com.
5. Ahrefs Brand Radar
Ahrefs Brand Radar benchmarks brand and domain appearances across six AI surfaces from a large prompt index.
Best for: category-level citation benchmarks. What to verify: whether indexed prompts match your buyer questions. Public source: ahrefs.com/brand-radar.
How to interpret five common results
Not cited on any engine. Check access, then indexing. If both pass, compare the sources cited for competitors before publishing more.
Cited on Perplexity but not in Google AI Overviews. The page is reachable but may not rank or may not answer the sub-question in the first screen. Look at the query fan-out topics and snippet controls.
Ranked in Google but absent in Gemini. Check Google-Extended before anything else. It is a one-line robots.txt setting that Search Console does not flag.
Named but not cited. The engine recognises the brand but chose other pages as evidence. Look for third-party pages that describe you inaccurately or not at all.
Competitors’ self-ranking pages fill the sources. This is a source-layer problem. Publish a direct, dated comparison page with explicit criteria, and seek accurate coverage on the directories the engines already cite.
What a source audit cannot prove
- It cannot prove that one change caused a new citation without a controlled timeline and repeated observations.
- It cannot guarantee future citations after model or index updates.
- It cannot reproduce a signed-in consumer interface exactly if it samples through an API.
- It cannot treat one run as stable. Research on repeated prompts found that the set of brands returned overlaps only about 45 to 59% between runs (arXiv 2604.07585).
When to move from a manual check to ongoing measurement
Ongoing measurement is justified when AI-assisted research shapes a meaningful buying decision, when the brand operates in several markets or languages or when inaccurate answers create commercial risk.
A small business may not need a platform yet. A documented monthly manual panel can be enough until the number of questions, engines or competitors makes it unreliable. Whichever route you take, verify the raw answers and cited URLs yourself rather than trusting a single score or this list.
Frequently asked questions
How do AI engines choose sources?
AI engines first limit candidates to pages their crawler can reach and their index contains. They then select passages that answer the generated sub-questions most directly. ChatGPT, Perplexity and Gemini each use different crawlers and indexes, so the same page can be cited by one engine and ignored by another.
Does blocking GPTBot stop ChatGPT from citing my site?
No. OpenAI treats GPTBot and OAI-SearchBot as independent settings. GPTBot governs training use, while OAI-SearchBot decides whether a page can appear in ChatGPT search answers. Blocking GPTBot alone does not remove a site from ChatGPT citations.
Is Google-Extended the same as blocking Googlebot?
No. Google-Extended controls Gemini training and grounding in Gemini Apps. Google states it does not affect inclusion or ranking in Google Search, and it does not govern AI Overviews, which draw on the normal Search index.
How long before a change shows up in AI answers?
OpenAI says a robots.txt change can take about 24 hours to reach ChatGPT search, and Perplexity gives a similar window. Content changes on index-based surfaces such as Google AI Overviews move with crawl and index cycles, which usually means weeks. Neither is observable without repeated sampling.
Does Canlah AI measure Perplexity citations?
No. Canlah AI’s standard tracking covers ChatGPT at Starter, adds Google AI Overviews at Growth and adds Google AI Mode and Gemini at Full. Perplexity and the Chinese engines are not part of it, so confirm Perplexity coverage explicitly with any vendor before signing.
Method and sources
This guide was rewritten from its August 30, 2026 edition on September 23, 2026. Crawler behaviour was checked against each operator’s current documentation on that date. The review did not independently test how often each crawler visits, audit vendor dashboards or confirm commercial terms.
- OpenAI: Overview of OpenAI crawlers. OAI-SearchBot, GPTBot and ChatGPT-User roles and the 24-hour update window.
- Perplexity: Perplexity crawlers. PerplexityBot and Perplexity-User roles and firewall guidance.
- Google Search Central: Google’s common crawlers. Google-Extended scope for Gemini training and grounding.
- Google Search Central: AI features and your website. Eligibility for AI Overviews and AI Mode, query fan-out and snippet controls.
- arXiv 2604.07585. Overlap of brand sets across repeated prompts.
- Vendor product pages for Profound, Peec AI, Semrush and Ahrefs Brand Radar, linked in the table above.
- Canlah AI self-audit of canlah.ai, Singapore market, September 7, 2026. Crawler access, indexing, mention counts and most-cited domains in the research snapshot.
See where your brand stands in AI answers. Run the free AI visibility audit →
Related articles
FREE WHITEPAPER
Marketing in the Agent Era
All 13 chapters public — no email wall. Includes an original dataset on the agent-readiness of 50 cross-border DTC brands.
Read it free →