AI Search Optimization Best Practices: What the Evidence Supports

Most AI search optimization checklists circulating right now are built on numbers nobody can trace to a source - so this guide grades each practice by the evidence behind it, and says plainly where there is none.

TL;DR

• AI crawlers do not execute JavaScript. Vercel and MERJ measured 569 million GPTBot fetches and 370 million Claude fetches and found that no major AI crawler renders JavaScript (17 December 2024). Client-side-rendered content is invisible to ChatGPT and Claude. Server-side rendering is the single highest-impact fix in this guide.
• Google's AI surfaces are the exception. The same study found Gemini runs on Googlebot's infrastructure with full JavaScript rendering - so a page can be readable to Google's AI Overviews and AI Mode while being blank to ChatGPT.
• Schema markup did not move AI citations. Ahrefs tracked 1,885 pages whose owners added JSON-LD, matched them against control pages on other domains and published the result on 11 May 2026: +2.4% in AI Mode and +2.2% in ChatGPT, neither distinguishable from zero, and −4.6% in AI Overviews - statistically significant but small. The authors' conclusion: "The most consistent finding is that not much really changed-schema had no clear positive or negative effect."
• llms.txt is read by coding agents, not by search. Ahrefs screened a panel of 137,210 domains for May 2026, found 38,360 serving a valid llms.txt, and found that 97% of those files received zero requests. No AI bot ever requested the file on domains that did not have one.
• The widely quoted passage-length multipliers have no traceable source. Figures such as "40–75 word passages are cited 3.1× more often" appear across dozens of GEO blogs with no named study, sample size or date behind them. Treat them as untested conventions, not findings.

What is AI search optimization?

AI search optimization is the practice of making a page retrievable and quotable by AI answer engines - ChatGPT, Google AI Overviews and AI Mode, Perplexity, Gemini, Claude, Copilot and Grok - rather than only rankable in a list of blue links. If the term itself is new to you, start with our plain-English guide to what AI visibility is and come back here for the implementation. The work splits into three questions: can the engine fetch the page, can it extract a self-contained answer from the page, and does it trust the page enough to attribute the answer to it.

That framing matters because the three questions have very different amounts of evidence behind them. Fetchability is measured, at scale, by people with server logs. Extraction is partly measured and partly folklore. Trust is almost entirely unmeasured. A checklist that presents all three with the same confidence is misleading you about where to spend the next sprint.

The rest of this guide is ordered by how much evidence exists, strongest first.

Can AI engines actually fetch your pages?

AI engines cannot cite what they cannot fetch, and the most common reason a page cannot be fetched is that its content is assembled in the browser rather than sent in the HTML response. This is the one area of AI search optimization with large-sample, log-level evidence behind it, and it is where the biggest wins still sit in October 2026.

Why JavaScript rendering is the first thing to check

Vercel and MERJ instrumented nextjs.org and the Vercel network and published the result on 17 December 2024, covering 4.5 billion Googlebot fetches, 569 million GPTBot fetches, 370 million Claude fetches, 314 million AppleBot fetches and 24.4 million PerplexityBot fetches. Their finding was unambiguous: "none of the major AI crawlers currently render JavaScript." ChatGPT's and Claude's crawlers do fetch JavaScript files, but they do not execute them, which means "they can't read client-side rendered content."

The practical consequence is that a single-page app without server-side rendering is an empty page to ChatGPT and Claude, no matter how good the content is. Content present in the initial HTML response - including JSON embedded in that response - can still be picked up.

Why Google's AI surfaces behave differently

The same study found that "Google's Gemini leverages Googlebot's infrastructure, enabling full JavaScript rendering," and that AppleBot renders through a browser-based crawler too. This is the mechanism behind a result that confuses a lot of teams: your page can appear in AI Overviews and AI Mode and be completely absent from ChatGPT answers, because one surface rendered your JavaScript and the other did not.

It is also why a single blended "AI visibility" percentage across all engines hides the problem you most need to see. The engines are not reading the same version of your site.

What the crawl mix tells you to fix

The Vercel and MERJ breakdown is worth reading as a to-do list rather than a curiosity:

These shares were measured on nextjs.org and the Vercel network, not across the web, so they are not a benchmark for your site. They are a prompt to check: if a third of the AI crawler fetches on a well-maintained estate hit 404s, it is worth pulling the same number from your own logs. The usual causes - stale internal links, dead sitemap entries, uncleaned redirect chains - cost nothing to fix and show up in your logs within a week.

What to do

• Serve the main content of every page you want cited in the initial HTML response: SSR, static generation or prerendering. Check it with curl and JavaScript disabled, not with a browser.
• Audit your sitemap and internal links against your server logs for AI bot 404s specifically, not just overall 404s.
• Do not block GPTBot, ClaudeBot, PerplexityBot or Google-Extended in robots.txt unless you have decided you do not want the citations. Confirm what you are actually serving them.

Does schema markup help you get cited?

Schema markup has not been shown to increase AI citations, and the one controlled test published so far measured a small decline in AI Overviews. This contradicts most AI SEO advice in circulation, so it is worth being precise about what was tested and what was not.

What the controlled test found

This was an observational study, not an intervention: Ahrefs screened around six million URLs to find 1,885 pages whose owners had added JSON-LD themselves between August 2025 and March 2026, matched each one to three control URLs on different domains with similar prior citation levels (~4,000 controls in total), and measured citations for 30 days before and 30 days after each page's treatment date. Ahrefs added nothing to anyone's site. The analysis was a matched difference-in-differences test, backed by a t-test, an event study and a re-run on a different window. Published 11 May 2026, written by Louise Linehan with Xibeijia Guan as contributor and Ryan Law as reviewer.

The results: +2.4% in Google AI Mode and +2.2% in ChatGPT, neither distinguishable from zero, and −4.6% in Google AI Overviews - statistically significant, but small in absolute terms, around twelve fewer daily citations per page. The authors' own conclusion: "The most consistent finding is that not much really changed-schema had no clear positive or negative effect."

What Google itself says

Google's own documentation on AI features in Search is explicit, and has been since its last update on 10 December 2025: "There's also no special schema.org structured data that you need to add," and "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." It adds a separate instruction that is easy to miss and more useful than the schema debate: structured data should match the page's visible text.

Why this is not a reason to rip out your schema

The honest reading is narrower than "schema does not work."

The Ahrefs test looked at pages that were already heavily cited - every page in the dataset had 100+ AI Overview citations in February 2025, before any schema was added - so it tells you about pages a model already knows, not about pages trying to get known. It also pooled schema types (Article, FAQ, Product, HowTo and Organization together), so it cannot tell you whether any single type works where the others do not. Schema still earns rich results in classic search, still drives merchant and product eligibility, and still carries real value for a catalogue - the field-level work that matters for a product feed is a separate job, covered in how to structure product data so AI search can recommend you. What the evidence does not support is treating schema as an AI citation lever or as a KPI. It is cheap hygiene, and it belongs in the hygiene column of your plan rather than the growth column.

The claim that a model which does not already know an entity might be helped by schema remains untested rather than disproven. Nobody has run that experiment.

Do you need an llms.txt file?

No evidence currently supports llms.txt as an AI search optimization tactic, and the largest log study available found almost nobody reading it. It is, however, genuinely useful for a different job, and conflating the two is why the debate stays confused.

What the server logs show

Ahrefs screened 137,210 domains that received traffic in May 2026, checked which served a valid Markdown llms.txt at the root, and then looked at who requested it. Published 15 June 2026 by Louise Linehan, with Xibeijia Guan as contributor:

That last row is the one that settles the "but they might start using it" argument for now. No AI bot went looking for a file that was not there, so publishing one is not positioning you for a crawler that is already checking.

Who is actually reading it

Within the AI bot share, GPTBot was the top named fetcher at 4.51% of requests. Ahrefs reports Claude-Code, Anthropic's coding agent, as the next named AI bot, though its own breakdown also shows an agentic-infrastructure crawler, statespace-indexer, at 3.52%; no standalone percentage is published for Claude-Code. The direction is what matters and it is not in doubt: a coding agent out-fetched every AI retrieval bot, and AI retrieval bots as a category accounted for 1.1% of requests. llms.txt is being used by agents pulling documentation, not by answer engines deciding what to cite.

If you publish developer documentation or an API that agents consume, llms.txt is doing a job for you. If you are an ecommerce brand hoping it will get your category pages into ChatGPT answers, the logs say it will not.

What changed in the v2 spec

The spec was revised in August 2026, and the changes point in the same direction: link relations so a page can declare where its Markdown version lives, subpath files with the most specific one applying, and a consumption model built around agents reading the file and following its links. These are refinements for agent retrieval, not a route into search citations. We walk through the full spec and who should implement it in our llms.txt guide.

What to do

• Publishing developer or product documentation that agents consume: write one, and follow v2.
• Anything else: leave it on the backlog. It is ten minutes of work and there is no measured return, so it is neither urgent nor harmful.
• Either way, do not report it as an AI visibility action in a client deck. It is not one.

How do AI engines choose which passage to cite?

This is where the evidence thins out dramatically, and where most AI search optimization advice is at its most confident. The structural reasoning is sound; the specific numbers attached to it are not sourced.

The numbers you should stop quoting

Figures such as "passages of 40–75 words are cited 3.1× more often than longer ones," "well-sized passages are 4.2× more likely to be cited with attribution," and "pages with appropriate schema are 3× more likely to be cited" appear across dozens of GEO and AEO blogs. We traced them on 13 October 2026 and found no named researcher, institution, publication date or link behind any of them.

The 3.1× figure is the clearest case. It is attributed to "a 2025 analysis of 10,000 AI citations" - a year and a sample size, with no author, organisation, publication or link, on the page that most other blogs copy it from. Another widely-copied source introduces its central claim with nothing more specific than "research on passage-based retrieval."

This is not a reason to dismiss the underlying idea. It is a reason not to build a content standard on a multiplier you cannot check, and not to put one in a client report.

What is defensible to say instead

There is one properly sourced dataset on where citations come from inside a page, and it is more useful than any of the multipliers. Kevin Indig isolated 18,012 verified citations from ChatGPT responses, reported by Search Engine Land on 18 February 2026:

At paragraph level the same analysis found 53% of citations came from the middle of paragraphs, 24.5% from first sentences and 22.5% from last sentences.

That second table is worth sitting with, because it complicates the standard advice. "Put the answer in the first sentence" is good practice for a reader scanning and for a heading-boundary chunk, but it is not where most cited text actually sat. The safer reading is that citable material needs to be dense throughout the section, not concentrated in one opening line, and that the first 30% of the page is where it pays most to put your strongest, most quotable claims.

Beyond that dataset, two things can be said without inventing precision.

The first is mechanical. Retrieval systems operate on passages, not whole documents, and heading boundaries are the most obvious place to split a document into passages. A section that answers one question completely, between its own heading and the next, is a unit that can be lifted intact. A section where the answer is spread across three paragraphs with the qualifying clause in the last one can be lifted in a way that changes its meaning. That is an argument for self-contained sections regardless of any multiplier.

The second is observable. Open any AI answer and look at what was quoted: it is almost always a short, declarative, self-contained statement that names its subject. A sentence that opens with "this also means" cannot be lifted, because the referent is in the previous paragraph. A sentence that opens with the exact entity from the question can be.

The practices that follow from the mechanism

• Answer first, in the first sentence of each section, leading with the exact entity from the question rather than a reference to it.
• Make each section stand alone. No "this", "these results", "as shown above" carrying meaning across a heading boundary.
• Use question-shaped H2s and H3s, and use plenty of them. Each one is a chunk boundary you control.
• Put your strongest claims in the first 30% of the page, which is where 44.2% of cited text sat in the Indig dataset.
• Attach the scope to the claim, not to the paragraph. "Of the 38,360 domains serving a valid llms.txt, 97% received zero requests in May 2026 (Ahrefs, panel of 137,210 domains)" survives extraction; "97% received no requests" does not.
• Name your sources in the visible text with dates. This one has a second-order benefit: it makes your page the easier thing to cite, because the citing system can see what it is standing on.

What about Google's position on all of this?

Google's documented position is that optimizing for its generative features is ordinary SEO, and its guidance explicitly tells site owners not to build AI-specific artefacts. Its AI features documentation, last updated 10 December 2025, states: "You don't need to create new machine readable files, AI text files, or markup to appear in these features," and "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary."

Two caveats keep that from being the whole story.

First, it is a statement about Google. AI Overviews and AI Mode draw on the same index as classic ranking, which is exactly why Gemini renders JavaScript and ChatGPT does not. The advice does not transfer to engines that crawl independently.

Second, "no special requirements" is not the same as "nothing changes." The same documentation asks that structured data match visible text and that content be crawlable - and the Vercel and MERJ data shows a large share of the web failing the crawlable part for non-Google engines.

How do you know whether any of it worked?

You measure citations, not rankings, and you measure them per engine rather than as one blended score. Rank tracking answers a different question from the one AI search optimization is asking, and a page can hold position four and lose the answer above it.

Why rank tracking is the wrong instrument

Ranking and citation are two separate events, and only one of them shows up in a rank tracker. A page can rank fourth on a commercial query and take almost none of the clicks, because the AI Overview above it cites a different source. The rank tracker reports position four and calls it a good day.

What Search Console can and cannot tell you

Google announced Search Generative AI performance reports in Search Console in June 2026. Checking them in our own property on 13 October 2026, the limit that matters is that they carry no query dimension - so they cannot answer "did we lose this specific query to an AI Overview". They tell you the aggregate is moving; they will not tell you which query moved it.

Nothing in Search Console reports the composition of a SERP either. It reports your site's appearances. An absence in a report means your page was not included, not that no AI feature was present. The only thing that settles whether an AI Overview appears on a given query is checking the live SERP - on the market you care about, because the result differs by country.

The CTR anomaly test

The fastest diagnostic you can run without any new tooling is to compare each query's click-through rate against your own site's median CTR at the same average position.

Export twelve months of query-level data from Search Console, bucket it by average position, and take the median CTR within each bucket. That median is your baseline - it already accounts for your brand, your category and your SERP features, which no published benchmark can. Then list every query sitting well below its own bucket's median.

Those are the queries where something is taking the click before the first organic result. Often it is an AI Overview; sometimes it is a shopping block, a video carousel or a competitor's sitelinks. The test does not tell you which - it tells you where to look at the live SERP.

We are deliberately not printing a table of expected CTR by position here. The figures in circulation vary widely by category and most of them trace back to no published dataset, and an article arguing that you should delete untraceable multipliers has no business printing one.

Where a tracking tool earns its place

Everything above gets you a diagnosis on Google. It gets you nothing on ChatGPT, Perplexity, Claude or Copilot, because those engines have no equivalent of Search Console - there is no report that tells you whether you were named in an answer.

This is the gap AI visibility platforms exist to close, including ours. If you are choosing between them, we compare the field by what each one actually measures in the best LLM optimization tools for AI visibility and the 11 best AEO tools. Ranketta runs the same prompts repeatedly through browser sessions on each engine rather than through vendor APIs, cross-validates the results across runs (why we use browser sessions rather than vendor APIs), and reports per engine and per SKU, so an absence in ChatGPT does not get averaged away by a presence in AI Overviews. That per-engine split is not a nicety: given that the engines are not even reading the same version of your site, a blended figure is averaging across incompatible conditions.

It is also how we found the fan-out pattern in the next section, which is not something any rank tracker would surface.

What AI engines do before they recommend a tool

Before an AI engine recommends a product, it frequently runs targeted site-restricted lookups against individual vendor domains. We can see this in our own data, and it changes what a vendor's site needs to do.

Of the eight most frequent fan-out queries recorded in our own Ranketta workspace for ChatGPT between September and October 2026, five were site:-restricted lookups aimed at vendor domains: site:otterly.ai AI search visibility tracking, site:otterly.ai AI search tracking citations, site:peec.ai AI search visibility features, site:ahrefs.com AI brand mentions tracking citations and site:profound.ai AI search visibility citations. The remaining three were multi-vendor comparison queries naming several tools at once.

The fifth one is worth pausing on, because it is wrong. Profound's site is tryprofound.com; profound.ai is not their domain. The model constructed a plausible-looking vendor domain and ran a site-restricted query against it - which returns nothing about the vendor it was trying to check. If your brand name does not obviously resolve to your domain, that is a failure mode worth knowing about, and it is an argument for making sure your brand name and your domain are tied together everywhere a model might read them.

This is a small sample from one workspace and one engine, and we are not presenting it as a general law. But the shape of it is instructive. The model was not browsing for a category explainer. It was checking specific claims on specific vendor domains before deciding who to name - which means the page that answers "does this vendor actually do X" has to exist on your own domain, in extractable form, with the feature named in the text. A feature mentioned only in a pricing table image, or only in a sales deck, does not survive that check.

For anyone selling software, that is a more actionable finding than any passage-length multiplier: the fan-out queries are visible, and they tell you which claims the model is trying to verify about you. Seeing them is a tooling question, which we cover in the best AI citation tracking tools.

What to do next

• Disable JavaScript and load your five most important pages. If the content disappears, nothing else in this guide matters yet. Fix rendering first.
• Pull AI bot 404s from your server logs. Vercel measured about a third of ChatGPT and Claude fetches hitting 404s on its own estate - find your own rate rather than assuming theirs, then fix what it surfaces.
• Move schema and llms.txt into the hygiene column of your plan. Keep the schema you have, make it match visible text, and stop reporting either as an AI visibility action.
• Rewrite the opening sentence of your top ten sections so each one answers its heading completely and names its subject explicitly.
• Run the CTR anomaly test on your Search Console export: bucket queries by average position, take your own median CTR per bucket, and list every query well below its bucket's median.
• Add per-engine citation tracking for the engines Search Console cannot see, and keep the engines reported separately. If you sell products rather than software, the SKU-level version of this work is in our ChatGPT product recommendation playbook.
• Delete the unsourced multipliers from your internal content standard and your client reporting.

How we verified this

Every claim above is traced to a named source with a date, checked on 13 October 2026. Where a widely-repeated figure has no traceable source, we say so rather than repeating it. One figure is our own: the fan-out sample in the vendor section comes from our Ranketta workspace for ChatGPT, September to October 2026, and is labelled as the small single-workspace sample it is. This category changes monthly, so re-check anything specific before acting on it.

Ready to see what AI engines say about you?

Run a free AI visibility audit and see, per engine, which prompts name you and which name someone else.

Free audit · Talk to us · Start tracking

Sources

• Vercel and MERJ, The rise of the AI crawler, 17 December 2024
• Ahrefs (Louise Linehan; Xibeijia Guan, contributor), We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, 15 June 2026
• Ahrefs (Louise Linehan; Xibeijia Guan, contributor; reviewed by Ryan Law), We Tracked 1,885 Pages Adding Schema. AI Citations Barely Moved., 11 May 2026
• Google Search Central, AI features and your website, last updated 10 December 2025
• Google Search Central Blog, Introducing Search Generative AI performance reports in Search Console, June 2026
• Ranketta Search Console property, generative AI reports checked 13 October 2026 (first-party)
• llms.txt specification and v2 change notes, llmstxt.org/changes.html, v2 August 2026
• Kevin Indig via Search Engine Land (Danny Goodwin), 44% of ChatGPT citations come from the first third of content, 18 February 2026 — 18,012 verified citations
• Ranketta workspace fan-out queries, ChatGPT, September–October 2026 (first-party)

FAQs