llms.txt
A proposed plain-text file, placed at a site's root, that gives AI systems a curated summary of its key content.
What is llms.txt?
llms.txt is a proposed plain-text file, placed at a website's root, that gives AI systems a short, curated summary of a site's most important pages, written in a format that's easy for a language model to read quickly. Instead of an AI having to wade through a page's navigation menus, ads, and JavaScript to find the useful text, llms.txt points it straight to clean, relevant content. It's meant to work similarly to how robots.txt tells crawlers what they may access, except llms.txt is about making the allowed content easier to understand rather than restricting access.
Why it matters
Jeremy Howard, co-founder of Answer.AI and fast.ai, published the original llms.txt proposal on September 3, 2024, at llmstxt.org, motivated by the observation that LLM context windows are too small to hold an entire website and that converting cluttered HTML into clean text for a model to read is laborious and error-prone. The proposed format is a Markdown file with an H1 title, an optional blockquote summary, and H2-delimited sections listing curated links to key pages, with an optional companion file, llms-full.txt, containing a site's full content concatenated into one document. Unlike RFC 9309's robots.txt, llms.txt is not backed by any standards body or major AI company as an officially supported input, and adoption has been mixed: some documentation platforms like Mintlify and GitBook auto-generate it, but there is no confirmed evidence that OpenAI, Anthropic, or Google's crawlers or models actually consume or prioritize it as of 2026.