llms.txt has been recommended as an AI visibility tactic for two years, and the first large-scale server-log study found that almost nobody is reading it – but the bots that do read it tell you exactly who the file is for, according to Ranketta’s latest research.
TL;DR
llms.txt is a Markdown file at your site root that lists your most important pages with short descriptions, so an AI agent can find the useful parts of your site without crawling it all. Jeremy Howard of Answer.AI proposed it in September 2024; the spec lives at llmstxt.org and was revised to v2 in August 2026.
97% of valid llms.txt files received zero requests. Ahrefs screened a panel of 137,210 domains for May 2026 and published the result on 15 June 2026. Around 28% - 38,360 domains - served a valid file, and 97% of those files were never fetched.
No AI bot goes looking for the file. In the same dataset, the AI bot share of requests for missing llms.txt files was zero. Publishing one is not positioning you for a crawler that is already checking.
The bots that do fetch it are coding agents, not answer engines. GPTBot was the top named AI fetcher at 4.51% of all llms.txt requests, and Ahrefs reports that Anthropic's coding agent Claude-Code out-fetched every AI retrieval bot. AI retrieval bots - the ones that decide what gets cited - accounted for 1.1%, or 233 of roughly 22,000 requests.
Google says it is not needed. Its AI features documentation, last updated 10 December 2025, states: "You don't need to create new machine readable files, AI text files, or markup to appear in these features."
Publish one if agents consume your documentation. Skip it if you are hoping it will get your category or product pages cited in ChatGPT or AI Overviews. There is no measured return for that use.
What is llms.txt?
llms.txt is a plain Markdown file served at the root of a domain - example.com/llms.txt - that gives a large language model a curated map of the site: an H1 with the project name, a short summary, and then sections of links with one-line descriptions of what each page contains. Jeremy Howard, co-founder of Answer.AI and fast.ai, published the proposal in September 2024, and the specification is maintained at llmstxt.org.
The problem it was designed to solve is real. A model working with a limited context window cannot ingest an entire documentation site, and HTML pages are full of navigation, cookie banners and markup that waste tokens. A short Markdown index that points to clean Markdown versions of the important pages is genuinely more useful to an agent than a sitemap.
Where the idea went sideways is in how it was marketed. It became a standard item on AEO and GEO checklists as a way to get cited in AI search, which is a different job from the one it was designed for - and the one job the evidence says it does not do.
How it differs from robots.txt and sitemap.xml
The three files are often grouped together and they do unrelated things.
robots.txt is a permission file. It tells crawlers which paths they may and may not request, and it is honoured by the major crawlers including GPTBot, ClaudeBot and PerplexityBot.
sitemap.xml is a discovery file. It lists URLs and metadata so a crawler can find everything, and it is actively consumed by search engines.
llms.txt is a curation file. It is an editorial judgement about which pages matter and what each one is for, written for a reader that will act on the descriptions. Nothing in the specification grants or denies access, and no engine is obliged to request it.
What a file actually looks like
The format is deliberately minimal: an H1 with the name, an optional blockquote summary, optional prose, then H2 sections containing Markdown link lists where each link carries a short description.
# Acme Docs
> API and integration documentation for Acme's commerce platform.
## Core
- [Quickstart](https://acme.com/docs/quickstart.md): install, authenticate, first request
- [Authentication](https://acme.com/docs/auth.md): API keys, OAuth scopes, token rotation
## Reference
- [Products API](https://acme.com/docs/api/products.md): create, update and query products
## Optional
- [Changelog](https://acme.com/docs/changelog.md): release history
The descriptions are the part people skip and the part that does the work. A link list without them is a sitemap in Markdown clothing.
Does llms.txt work?
On the only large-scale log evidence available, no: 97% of valid llms.txt files received zero requests in a month, and no AI bot requested the file on domains that did not have one. The study is worth reading in detail, because the small amount of traffic that does exist is the most informative part of it.
What the 137,000-domain study measured
Louise Linehan, with Xibeijia Guan as contributor, took every domain in Ahrefs' Web Analytics panel that received traffic in May 2026 - 137,210 domains - checked each root for an llms.txt returning HTTP 200, confirmed the response was genuine Markdown rather than an HTML error page, and then examined who requested those paths. They published on 15 June 2026.
| Finding | Figure |
|---|---|
| Domains serving a valid llms.txt | 28% (~38,000) |
| Valid files with zero requests in May 2026 | 97% |
| Files fetched at all | ~3% (~1,100 domains, ~22,000 requests) |
| Requests from bots rather than humans | 96% |
| AI bots probing for the file where it did not exist | Zero |
The authors flag the 28% as an upper bound, because Ahrefs' customers skew technical and SEO-aware - so adoption across the open web is likely lower. They also note that "fetched" is not the same as "read", which makes every figure in the table a ceiling on actual use rather than a measure of it.
Who is actually requesting it
The breakdown of those ~22,000 requests is the finding that changes what you should do:
| Requester category | Share of requests |
|---|---|
| SEO audit tools | 21.7% |
| Other and unidentified | 14.9% |
| General web crawlers (incl. Googlebot) | 13.1% |
| Tech profiling tools | 11.6% |
| AI agents and agentic infrastructure | 10.5% |
| GEO/AEO tools | 5.8% |
| AI training crawlers | 5.3% |
| llms.txt discoverability bots | 3.6% |
| AI assistants | 2.5% |
| AI retrieval bots | 1.1% |
The single largest requester of llms.txt files is the SEO industry checking whether llms.txt files exist. Within the AI bot share, GPTBot led at 4.51% of total requests. Ahrefs names Claude-Code, Anthropic's coding agent, as the next named AI bot and reports that it out-fetched every AI retrieval bot - though its own chart also shows an agentic-infrastructure crawler, statespace-indexer, at 3.52%, and no standalone percentage is published for Claude-Code.
The ranking is fuzzy; the direction is not. Claude-Code is a coding agent reading documentation to complete a task, and it out-fetched the engines that decide what to cite in a consumer answer. Those - the AI retrieval bots - account for 1.1% of all llms.txt requests, 233 of roughly 22,000.
Why "they might start using it" is not currently an argument
The most common defence of llms.txt is that platforms may adopt it later, so you should publish now. The study tested that directly by looking at requests for files that did not exist: the AI bot share of those 404s was zero, and missing-file requests were about 98% human. Nothing is probing for it.
That could change. If it does, writing the file is a short job you can do at the time - it does not require a year of accumulated history the way a backlink profile does.
What do the AI platforms themselves say?
Google states that no AI-specific file is needed for its generative features. No other major platform has committed publicly to consuming llms.txt as a ranking or citation input, although several publish their own files for their developer documentation.
Google's documented position
Google's AI features documentation, last updated 10 December 2025, is direct: "You don't need to create new machine readable files, AI text files, or markup to appear in these features," and "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." It does not name llms.txt, but "new machine readable files, AI text files" is a description of exactly this category.
The reasoning behind it is structural rather than political. AI Overviews and AI Mode draw on the same index as classic Google ranking, so there is no separate pipeline for an AI-specific file to feed.
Google has also been publicly consistent on this through its search relations team over more than a year, with comparisons drawn to the long-dead keywords meta tag. Those remarks are paraphrased in secondary coverage - Search Engine Journal and Search Engine Land - rather than available as a single citable statement, so treat the documentation above as the authoritative version.
What the other platforms do, as opposed to say
Several platforms publish llms.txt files for their own developer documentation. We checked Anthropic's on 13 October 2026: docs.anthropic.com/llms.txt resolves to platform.claude.com/llms.txt, returns a valid Markdown index with an H1, grouped sections and most links pointing at .md versions of the page, and ships a companion llms-full.txt. It is not fully spec-compliant - no blockquote summary, links nested under H3s, no ## Optional section, and the language and service entries carry no .md - which makes it a useful real-world example rather than a model answer. That is a meaningful signal about the format's intended use - and it is often misread as a commitment to consume other people's files. It is not the same thing: publishing one so agents can read your docs says nothing about whether your crawler reads anyone else's.
As of 13 October 2026 we have found no public statement from OpenAI, Anthropic, Perplexity or Microsoft confirming that their crawlers use llms.txt as a retrieval or citation input. We will update this section if that changes.
The Lighthouse wrinkle
Chrome's Lighthouse added an Agentic Browsing audit in May 2026 which looks for an llms.txt. It treats the file as optional - a missing file is marked Not Applicable rather than failed. This gets cited as evidence that Google wants the file, and it is worth separating: Lighthouse is a Chrome developer tool auditing a site's readiness for browsing agents, not a Google Search ranking signal. In the Ahrefs dataset, Lighthouse accounted for 22 requests - roughly one in a thousand.
What changed in llms.txt v2?
The v2 revision, published in August 2026, moved the specification further towards agent retrieval and away from bulk text expansion. Six things changed, and they matter if you are writing or maintaining a file now.
Discoverability through link relations. A page can now declare where its Markdown version lives and which llms.txt describes it, either as HTML link elements or an HTTP Link header, with rel="describedby" pointing to the covering file. This is the most practically useful addition: it means an agent that has landed on one of your pages can find the map without guessing at the root.
Two valid Markdown URL forms. v1 appended .md to the full page URL. v2 accepts both that form and one that replaces the page's existing extension, so page.html.md and page.md are both legitimate.
Subpath files. An llms.txt now applies to the pages beneath its path, and the most specific file wins. This matters for anyone who controls a path rather than a domain - a docs subdirectory, a documentation platform, a team inside a large site - because they can now participate without root access.
Consumption model. The spec no longer covers the llms_txt2ctx expansion tool. The expectation is now that an agent reads or searches the file and follows its links to LLM-friendly content, rather than bundling everything into one context blob.
Optional sections lost their mechanics. ## Optional is still allowed for secondary links but no longer carries mechanical semantics for parsers. It is a convention for human authors now, not an instruction.
Updated background and examples describing how agents actually use websites today.
Read together, these changes describe a format for agents navigating documentation. None of them is a step towards search engines consuming the file.
Should you publish an llms.txt file?
Publish one if AI agents read your documentation to do work. Skip it if your goal is citations in AI search answers. The dividing line is whether an agent needs to navigate your content to complete a task, or whether an answer engine needs to quote you.
Publish one if this describes you
You ship developer documentation, an API or an SDK. This is the format's home ground, and the Claude-Code fetch data shows coding agents using it in practice.
You run a documentation platform or a large docs subdirectory. v2's subpath rule was added for exactly this case.
You sell a product that agents configure or integrate. If a developer's agent is reading your docs to wire you up, a clean map shortens the path and reduces the chance it works from a stale page.
You already publish Markdown versions of your pages. The marginal cost is close to zero and the file makes them discoverable. Anthropic's own documentation index is a usable working example, if an imperfectly spec-compliant one.
Skip it if this describes you
You are an ecommerce brand or publisher chasing AI citations. There is no measured return. The engines that choose citations - AI retrieval bots - accounted for 1.1% of all llms.txt requests in the Ahrefs dataset, 233 of roughly 22,000.
You are reporting it to a client as an AI visibility deliverable. It is not one, and the evidence to the contrary is a single search away.
You are considering it instead of fixing rendering. AI crawlers other than Google's do not execute JavaScript, which means a client-side-rendered page is invisible to ChatGPT and Claude regardless of what your llms.txt says. That is the work that moves citations, and we cover it in our guide to AI search optimization best practices.
If you do publish one, do it properly
Serve it at the root as text/plain or text/markdown, returning HTTP 200. Ahrefs' screening step rejected responses that were HTML error pages rather than genuine Markdown, so a file that fails this check does not count as existing. Ahrefs did not publish how many failed, and explicitly did not test files against the spec.
Write real descriptions. One clause per link saying what the page contains and who it is for. This is the only part of the file an agent cannot infer from a URL.
Link to Markdown versions where you have them, per v2.
Add the rel="describedby" link relation on the pages the file covers, so agents can find it without assuming the root.
Keep it short and keep it current. A file listing pages that moved six months ago is worse than no file, because it actively points an agent at 404s.
Do not duplicate your sitemap. If it lists every URL you have, it has stopped being a curation file and you have built a worse sitemap.
How do you tell whether it is doing anything?
Check your own server logs for requests to /llms.txt by user agent, and check whether your content is being cited in AI answers separately. These are two different measurements and conflating them is how the tactic got its reputation in the first place.
The log check
Filter your access logs for the /llms.txt path and group by user agent. If you see GPTBot, Claude-Code or other agent traffic, the file is being read and your documentation use case is live. If you see only SEO audit tools and uptime monitors, you have confirmed the Ahrefs finding on your own domain - which is useful information, and takes about ten minutes.
The citation check
Whether you are being cited in AI answers is unrelated to the log check, and no analytics property reports it directly. Search Console covers Google's AI features partially: the Search Generative AI performance reports, announced in June 2026, give site-wide impressions and clicks from AI features but carry no query dimension - checked in our own property on 13 October 2026 - so they cannot tell you which query you lost. For ChatGPT, Perplexity, Claude and Copilot there is no equivalent report at all.
This is the gap AI visibility platforms exist to close, ours included. Ranketta runs the same prompts repeatedly through browser sessions on each engine rather than through vendor APIs, cross-validates across runs, and reports per engine and per SKU - so you can see whether a change moved citations on the engine you changed it for, instead of watching a blended average absorb it. Why we use browser sessions rather than vendor APIs explains the trade-off behind that choice.
That separation is the whole point here. Had more teams measured citations per engine over the last two years, llms.txt would not have spent two years on best-practice checklists without anyone noticing that the files were not being requested.
What to do next
Check your logs for /llms.txt requests by user agent. Ten minutes, and it tells you whether this debate applies to your site at all.
Decide which side of the line you are on: agents navigating your documentation, or engines citing your content.
If you publish documentation, write the file and follow v2 - real descriptions, Markdown links, rel="describedby" on the covered pages.
If you do not, move it to the backlog and do not report it as an AI visibility action.
Spend the time on rendering instead. Disable JavaScript, load your five most important pages, and fix anything that disappears.
Measure citations per engine, so the next tactic that spends two years on checklists gets caught faster.
How we verified this
Every figure above is traced to a named source with a date, checked on 13 October 2026. The Ahrefs study covers one month of one vendor's panel and its authors describe each figure as a ceiling on consumption rather than a measure of it; we have kept that caveat attached. Where we could not find a primary statement - notably public positions from OpenAI, Anthropic, Perplexity and Microsoft on consuming llms.txt - we say that we could not find one rather than inferring it. This category moves monthly, so re-check anything specific before acting on it.
Find out what is actually moving your citations
llms.txt is easy to check and easy to rule out. The harder question - whether anything you publish is being cited, and on which engine - needs per-engine measurement.
Run a free AI visibility audit and see which prompts name you, which name a competitor, and where the gap sits per engine.
Free audit · Talk to us · Start tracking
Sources
- llms.txt specification, llmstxt.org, proposal September 2024; v2 change notes, August 2026
- Ahrefs (Louise Linehan; Xibeijia Guan, contributor), We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, 15 June 2026
- Google Search Central, AI features and your website, last updated 10 December 2025
- Google Search Central Blog, Introducing Search Generative AI performance reports in Search Console, June 2026
- Vercel and MERJ, The rise of the AI crawler, 17 December 2024
- Chrome Lighthouse, llms.txt audit, Agentic Browsing, 2026
- Anthropic developer documentation llms.txt, platform.claude.com/llms.txt, checked 13 October 2026
