How to Do Technical AEO on Your Site: Crawl Access, Rendering and Server Logs

Quick answer

Canlah AI does technical AEO, under the method in this guide, as three checks that decide whether an answer engine can fetch, read and re-fetch your pages at all: crawl access, render dependency and server log evidence. Eleven user agents matter, and seven of them are retrieval fetchers that pull a page into an answer when a question is asked. A page that a retrieval crawler cannot fetch, or that only exists after JavaScript runs, will not be cited however well it is written.

Canlah AI runs these checks as the first stage of every engagement and publishes the manual version here. Its own robots.txt declares 21 user-agent groups, 20 of them named, and its self-audit of canlah.ai on September 7, 2026 tested eight AI user agents, found all eight allowed and recorded 80 of 80 crawled pages as indexable.

Treat the result as directional, because one crawl clears a gate and proves nothing about citations. If the audit finds a blocked or unrenderable template, fix it and re-run the same crawl before changing anything else.

Disclosure: Canlah AI publishes this guide and offers the crawler access audit and AI visibility work described below. Crawler and tool details come from each vendor’s public documentation, reviewed on September 23, 2026, not from paid accounts. The rendering and fetch-volume figures are Vercel’s measurement published December 17, 2024, not a Canlah AI replication.

What does technical AEO actually cover?

Technical AEO covers three core checks, crawl access, render dependency and log evidence, plus three supporting checks: response health, machine packaging and engine split. Only two of the six, response health and machine packaging, overlap with a classic technical SEO audit. The rest exist because answer engines fetch more often than Googlebot and discard anything missing from the first response.

• Crawl access: whether robots.txt allows each named AI user agent and whether a firewall blocks it anyway.
• Render dependency: whether the quotable text is in the server’s HTML or appears only after client-side JavaScript runs.
• Log evidence: which verified agents reached which templates on which dates.
• Response health: each crawler’s share of 2xx, 3xx, 404, 403 and 429 responses, per agent.
• Machine packaging: whether the sitemap, canonical tags, structured data and any plain-text summary file agree.
• Engine split: whether Western and Chinese crawlers get the same access decision, a policy question rather than a technical one.

Canlah AI treats the three core checks and response health as pass or fail gates, because a failed gate makes every downstream content decision unmeasurable.

Can AI bots crawl my website? Check these user agents first

Eleven user agents decide whether AI bots can crawl your website, and seven of them are retrieval fetchers that affect whether a page can be cited in a live answer. GPTBot and ClaudeBot are training-only crawlers. Google-Extended is a control token rather than a fetcher and governs Gemini training and Gemini Apps grounding, while Bytespider feeds ByteDance products including Doubao without a published retrieval or training split.

AI and search crawler user agents reviewed September 23, 2026.

User agent	Operator	robots.txt behaviour	Published IP list	What allowing it decides
OAI-SearchBot	OpenAI	Respects disallow rules; OpenAI states opted-out sites are not shown in ChatGPT search answers	openai.com/searchbot.json	Retrieval for ChatGPT search answers
ChatGPT-User	OpenAI	OpenAI states robots.txt may not apply, as the fetch is user-initiated	openai.com/chatgpt-user.json	Fetches a page a ChatGPT user action needs
GPTBot	OpenAI	Respects disallow rules	openai.com/gptbot.json	Training corpus only, not live ChatGPT citations
Claude-SearchBot	Anthropic	Respects disallow rules	claude.com/crawling/bots.json	Indexing for Claude search responses
Claude-User	Anthropic	Respects disallow rules per Anthropic’s page	claude.com/crawling/bots.json	User-initiated fetches in a Claude conversation
ClaudeBot	Anthropic	Respects disallow rules	claude.com/crawling/bots.json	Training corpus only, not live Claude citations
PerplexityBot	Perplexity	Respects robots.txt; Perplexity advises allowing it	perplexity.com/perplexitybot.json	Surfacing and linking pages in Perplexity results
Perplexity-User	Perplexity	Perplexity states this fetcher generally ignores robots.txt	perplexity.com/perplexity-user.json	Fetches a page a Perplexity user asked about
Googlebot	Google	Respects disallow rules	Google common-crawlers.json	Google Search, and therefore AI Overviews and AI Mode
Google-Extended	Google	Respects disallow rules; Google states it does not affect Google Search inclusion	Not a separate fetcher; crawling uses existing Google user agents	Gemini training and Gemini Apps grounding, not AI Overviews
Bytespider	ByteDance	Documented as respecting robots.txt	Not documented on the reviewed page	ByteDance products including Doubao; purpose split not documented on the reviewed page

Source: each operator’s public crawler documentation, reviewed September 23, 2026. “Not documented on the reviewed page” means the operator’s public materials did not state that item when reviewed on that date. It is not proof that the item does not exist.

One rule breaks more sites than any block: robots.txt matching is most-specific-group-wins, and a crawler obeys exactly one group. A site that writes Disallow: /checkout/ under User-agent: *, then adds a named User-agent: OAI-SearchBot group without repeating that line, has given OpenAI’s retrieval crawler broader access than it intended. The canlah.ai robots.txt repeats the same three disallow lines inside all 20 named groups and the wildcard group for that reason.

Do AI crawlers render JavaScript?

None of the seven major AI crawlers in Vercel’s network-wide measurement renders JavaScript, so client-side content is invisible to them even when the page looks correct in a browser. Vercel names OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot as non-rendering, with Googlebot and Applebot the exceptions that do execute JavaScript.

In the month before Vercel published its measurement on December 17, 2024, Googlebot made 4.5 billion fetches across Search and Gemini, GPTBot 569 million, Claude 370 million and PerplexityBot 24.4 million. Both ChatGPT and Claude downloaded JavaScript files without running them, at 11.50% and 23.84% of requests, so the bandwidth was spent and the content still went unread. The figures date from 2024, so test your own pages rather than rely on them.

Test it by fetching your own page the way a crawler does, with curl -s https://example.com/page/, then search that raw HTML for the sentence you want quoted. If the sentence is missing, the answer engine does not have it either. Server-side rendering, static generation or a pre-rendered fallback for the answer block each fix this; lazy-loading the copy or injecting it from a client-side API does not.

Canlah AI hit this in its own September 7, 2026 audit of canlah.ai, which used a basic fetch with no JavaScript rendering, reached 80 pages against a sitemap declaring 174 URLs and produced orphan-page and click-depth figures the report marked as not self-consistent and excluded from scoring. A non-rendering AI crawler hits the same limit on any site that builds navigation in the browser.

How to run an AI crawler access audit

An AI crawler access audit takes six steps run in order: read robots.txt, test the answer paths, fetch each template raw, separate firewall blocks, pull 90 days of logs and re-run. Each step leaves a record you can re-run after a change.

Step 1: Read your robots.txt as a crawler does

Fetch https://yourdomain.com/robots.txt and list every User-agent group. For each of the eleven agents above, work out which single group applies under most-specific-group-wins, then write down that agent’s effective allow and disallow lines.

Step 2: Test the paths that carry your answers

Take the ten URLs you most want quoted, usually pricing, comparison and service pages. Check each against the Step 1 rules, then check for noindex, nosnippet and a restrictive max-snippet value in the HTML and the X-Robots-Tag header.

Step 3: Fetch each template without a browser

Request each page template with a plain HTTP client and an AI crawler user agent string, then grep the response for the first sentence of each answer block. Record the raw HTML byte size alongside the rendered size. The gap between them is the render dependency, measured.

Step 4: Separate a firewall block from a robots block

A 403 or a challenge page means the edge rejected the crawler regardless of robots.txt. Check WAF, bot-management and rate-limit rules for the same eleven agents, because a default bot rule blocks them silently.

Step 5: Pull 90 days of server logs

Filter access logs by user agent token, then verify each hit against the operator’s published IP list or reverse DNS, because user agent strings can be spoofed. Count hits, status codes and unique URLs per agent per week. A 90-day window is the shortest that survives a crawler version change, and it is what Canlah AI asks for at kickoff.

Step 6: Write the constraint down and re-run

Name one constraint, fix it, then re-run Steps 1 to 5 with the same URL list and log window. Keep both records, because an access audit is only useful as a before-and-after pair.

Server log analysis for AI SEO: what to count

Server log analysis for AI SEO counts six fields per user agent per week, and it is the only method that shows what a crawler actually took rather than what your robots.txt permits.

What to extract from access logs, per user agent, per week.

Log field	What to record	Decision it supports
User agent token	Exact string, grouped as retrieval or training	Whether a citation-eligible crawler reaches the site
Status code	Share of 2xx, 3xx, 404, 403 and 429 per agent	Whether a firewall or rate limit is the block rather than robots.txt
Client IP	Match against the operator’s published JSON list or reverse DNS	Whether the hit is the real crawler or a spoofed user agent
URL path	Which templates are fetched and which are never fetched	Where crawl attention is skewed away from commercial pages
Timestamp	First and last hit per template, per agent	Whether a new page was fetched before an answer changed
Content type	Share of HTML, JavaScript, image and PDF responses	What the crawler could read without rendering

Source: Canlah AI log review checklist; the benchmark shares below come from Vercel’s measurement published December 17, 2024. The field list is a working checklist, not an industry standard.

Canlah AI reads these six fields per agent, never site-wide, because a blended number hides the one crawler that matters. In Vercel’s measurement, ChatGPT spent 34.82% of its fetches on 404 pages and Claude 34.16%, so a high 404 share in your own logs is not by itself evidence of a site fault. ChatGPT’s requests were 57.70% HTML while Claude’s were 35.17% images, so a site-wide content-type split hides both patterns.

Tools that show AI crawler hits

Five tools show AI crawler hits, from a free robots.txt check to a managed log pipeline, and Canlah AI ranks first in this comparison only for teams that want the audit run and repeated for them rather than a dashboard to operate.

Crawler-access tooling reviewed September 23, 2026.

Tool	What it shows for AI crawlers	Free or public entry	Main limitation
Canlah AI	Per-agent robots.txt status across eight named AI user agents, plus indexability, sitemap and render findings in one report	Free AI visibility report within 48 hours	Reports the audit crawl, not your own server logs
Cloudflare AI Crawl Control	Per-crawler request volumes, robots.txt compliance tracking and allow or block rules	Included on all plans	Covers only traffic proxied through Cloudflare
Screaming Frog Log File Analyser	Uploaded logs with search engine and AI bot verification, including spoofed IPs	Free to 1,000 log events, one project	£99 per year above that; works on uploaded log files
Known Agents	Daily-updated directory of agents across 17 types, with a robots.txt checker and agent analytics	Free checker and analytics tier	Confirm what its analytics tier records before relying on it
Google Search Console	Googlebot crawl stats by response code, file type and purpose	Free with a verified property	Googlebot data only; AI crawler reporting not documented on the reviewed page

Source: each vendor’s public product or documentation page, reviewed September 23, 2026. Capabilities refer to public documentation, not independently tested features.

Cloudflare AI Crawl Control is the fastest read for a site already proxied through Cloudflare. Screaming Frog Log File Analyser gives the most defensible read from raw logs because it verifies bot IPs. Known Agents is the reference for an unfamiliar token, and Google Search Console stays necessary for Googlebot, the crawler behind AI Overviews and AI Mode.

How to read five common crawler access results

Five crawler access results recur in audits, and each points to a different fix: discovery, edge rules, rendering, robots.txt scope or answer fit.

Allowed in robots.txt, absent from logs: The rule permits the crawler and the crawler never came, so check discovery rather than access. Confirm the URLs sit in a submitted sitemap, that internal links to them exist in the raw HTML and that at least one external page links to the template. An orphan page can be crawlable and still never found.

Allowed in robots.txt, 403 in logs: The edge is refusing the crawler, which is a WAF or bot-management rule rather than an AEO problem. Verify the client IP first, because a blocked spoof is the correct outcome.

Fetched, but the answer text is missing: The page depends on client-side rendering. Move the answer block into the server response and re-fetch. Do not add content until the existing content is readable.

Training crawler allowed, retrieval crawler blocked: This is the most expensive configuration, usually copied from a snippet written to block AI training. Blocking GPTBot while allowing OAI-SearchBot protects training data; the reverse removes citation eligibility and protects nothing.

Everything passes, still no citations: Technical AEO is a gate, not a cause. The constraint has moved to answer fit, entity clarity or the sources the engine prefers for that query, and no crawler audit resolves those. This is where Canlah AI stops reporting access and starts probing answers.

What a crawler access audit cannot prove

A crawler access audit cannot prove five things, and Canlah AI reports every run, including its own, as a cleared gate rather than a result.

• It cannot prove that an engine will retrieve a permitted page.
• It cannot prove that a fix caused a citation, because engines change retrieval independently of your site.
• It cannot prove future access, because crawler tokens, versions and IP ranges change without notice.
• It cannot predict user-triggered fetchers, which OpenAI and Perplexity both say may not follow robots.txt.
• It cannot decide whether training crawlers should read the site, which is a legal and commercial call.

When to move from a one-off audit to continuous monitoring

Continuous monitoring is justified under four conditions: weekly releases, several teams editing robots.txt or firewall rules, one template carrying most commercial pages or sales into both Western and Chinese markets. The last matters because the access decision is made twice, for Western and for Chinese crawlers, and the evidence is not interchangeable.

Canlah AI is a Singapore-based SEO + GEO agency that measures and improves brand visibility inside ChatGPT, Gemini, Google AI Overviews and Google AI Mode, reporting AI visibility as re-verifiable ranges with timestamped evidence. Its crawler access work suits teams that want the audit run, repeated and archived rather than a dashboard to staff. It is less useful for a small site with one template, where a quarterly manual run of the six steps is enough.

Do not act on this guide or on any tool’s summary alone. Ask whoever runs the audit, Canlah AI included, to show the raw fetch and the raw log line behind each finding, because access you cannot trace to a response code cannot be fixed.

Frequently asked questions

Can AI bots crawl my website if robots.txt allows Googlebot?

No. Googlebot governs Google Search, including AI Overviews and AI Mode. OpenAI, Anthropic and Perplexity each read their own named user agents, so a robots.txt naming only Googlebot leaves those crawlers under whatever the wildcard group says. Check each agent separately and confirm which group applies under most-specific-group-wins.

How do I do technical AEO on my own site?

Run three checks in order: crawl access for each named user agent in robots.txt and at the firewall, a raw-HTML fetch of each template to test render dependency and 90 days of verified server logs. Fix the one constraint those checks expose, then re-run the same URL list and log window. The six audit steps above are the manual version.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google’s crawler documentation states that Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal, and AI Overviews are served from Google Search. Blocking it does opt the site out of Gemini training and of grounding in Gemini Apps and Vertex AI, so treat it as a Gemini decision rather than an AI Overviews one. Googlebot is the token that matters for AI Overviews.

Is technical AEO different from technical SEO?

Partly. Indexability, canonical tags, sitemaps and response codes are shared, and fixing them helps both. Technical AEO adds the per-agent access decision, the render dependency test and per-agent log analysis, none of which a standard technical SEO crawl reports.

Method and sources

Crawler behaviour in this guide comes from each operator’s public documentation, checked on September 23, 2026, and from Vercel’s network measurement published December 17, 2024. The Canlah AI figures come from a self-audit of canlah.ai run on September 7, 2026 and from the live canlah.ai robots.txt, both first-party and not independently audited. Tool pricing and free-tier terms change, so confirm them on the vendor’s page.

• OpenAI, Overview of OpenAI Crawlers. Tokens, purposes and IP files. OpenAI crawler docs
• Anthropic, Does Anthropic crawl data from the web. The three Claude agents and the IP file. Anthropic support article 8896518
• Perplexity, Perplexity Crawlers. Both tokens, robots.txt behaviour and IP files. Perplexity crawler docs
• Google Search Central, Google common crawlers. Googlebot and Google-Extended scope. Google common crawlers
• Vercel, The rise of the AI crawler, December 17, 2024. Rendering, fetch volumes, 404 and content-type shares. Vercel measurement
• Cloudflare, AI Crawl Control documentation. Plan availability and control scope. Cloudflare AI Crawl Control
• Screaming Frog, Log File Analyser. Free limit, price and bot verification. Screaming Frog Log File Analyser
• Known Agents, agent directory. Categories, robots.txt checker and analytics tier. Known Agents
• Canlah AI robots.txt, read September 23, 2026. The 20 named user-agent groups plus the wildcard group and the repeated disallow lines. canlah.ai/robots.txt
• Canlah AI self-audit of canlah.ai, September 7, 2026, internal export. The eight-agent check, 80 crawled pages and the non-rendering limitation.

See whether AI crawlers can reach your pages. Run the free AI visibility audit →

Related articles

• What Is llms.txt? A Plain Guide, and What the Evidence Shows
• How to Get Cited by AI Search Engines in Answers: ChatGPT, Perplexity, Google AI Overviews
• Why Is My Website Not Showing in AI Overviews? How to Fix Indexing, Answer Fit and Competitor Gaps