llms.txt is a Markdown file you publish at the root of your domain - yourdomain.com/llms.txt - that lists the pages you'd most want an AI model to read, each with a one-line description. Jeremy Howard of Answer.AI proposed the llms.txt standard in September 2024, and the current spec lives at llmstxt.org. That's the whole idea: a curated reading list, not a rulebook.
Here's the part most explainers skip. Nothing gets granted and nothing gets blocked, and no major model provider - not OpenAI, not Anthropic, not Google - has publicly committed to fetching llms.txt before answering a question. A few developer tools do read it. Almost nothing else does. If you came here wondering whether GPTBot or ClaudeBot follow your redirects, that's a separate question - see what AI crawlers actually do to a link. This post is about the llms.txt file itself: what's really in it, who reads it, what it costs to maintain, and whether shipping one is worth an afternoon in 2026.
What Is llms.txt, Exactly
The spec is short enough to read in two minutes. A valid file needs exactly one required element: an H1 with your site or project name. Everything past that is optional but conventional. A blockquote directly under the H1 gives a one-sentence summary. Free-form paragraphs can add context. Then come H2 headings, each holding a bulleted list of links, and each link can carry a colon plus a short note on what it contains. A section titled "Optional" at the H2 level is a convention for secondary material a model can skip when it's working with limited context.
The format was built for one specific problem: a documentation site is mostly navigation chrome and sidebar links, and a model reading the raw HTML burns tokens on all of it. An llms.txt file hands it a clean index instead. That's a narrower job than most coverage implies: the format was designed for docs, not the open web. Weigh it against actual answer engine optimization work and that distinction is most of the answer.
llms.txt Is Not robots.txt
The comparison everyone reaches for is robots.txt, and the comparison misleads more than it helps.
robots.txt is scoped by RFC 9309 to a single host, and it works because named crawlers - Googlebot, Bingbot, the AI crawlers among them - check for it before fetching anything and honor a Disallow line as a matter of decades-old convention. It never granted access either - an agent that ignores it was always going to ignore it - but the crawlers worth worrying about do read it.
That backing doesn't exist for llms.txt. Nothing tells a model to check for /llms.txt before it answers a question about your site. A handful of developer tools do check. A general-purpose assistant answering a question in a chat window has no obligation to look for your file, and most don't bother. Calling it "the AI crawler file" or "the robots.txt for AI" - both labels show up in nearly every explainer - implies an enforcement mechanism the format simply doesn't have. It never did.
Who Is Actually Reading the File
Adoption is real. It's just narrower than the coverage suggests.
Mintlify added automatic llms.txt generation for every site it hosts in late 2024, and adoption jumped overnight, because Anthropic, Cursor, Stripe, Cloudflare, and Vercel all publish docs through platforms like Mintlify, Fern, or GitBook that now generate the file by default. Coding tools - Cursor, Windsurf, Claude Code, GitHub Copilot, Cline, Aider - do look for /llms.txt when pointed at a documentation site, because a clean index beats scraping rendered navigation for that one job.
What hasn't happened is the part most posts imply. No general-purpose consumer assistant has confirmed it checks the file before answering a question, and the people closest to search are blunt about it. Google's John Mueller put it plainly: none of the AI services have said they use it, and server logs show most don't even request it. He compared it to the keywords meta tag - a field publishers once filled in earnestly until search engines quietly stopped reading it, because it was trivial to game.
So: a real audience of coding tools and docs platforms, no confirmed audience among ChatGPT, Gemini, or Perplexity answering a shopping or research question. If AI traffic to your site matters enough to act on, don't infer it from a file nobody's confirmed reading - measure it. GA4's AI Assistant channel is where that traffic actually shows up, on the days it shows up at all, and Elido's analytics show which visits came from an assistant on the links you publish yourself.
What the File Looks Like, Line by Line
Here's a minimal, valid example, close to the pattern in the spec:
# Acme Docs
> Acme is an API for parsing invoices. This file lists the pages worth reading first.
Full documentation lives at docs.example.com. Sections below are ordered by how often a new integration needs them.
- [Quickstart](https://docs.example.com/quickstart): Authentication and your first API call
- [Rate limits](https://docs.example.com/limits): Current limits and how to request an increase
- [Endpoints](https://docs.example.com/api): Full request and response schemas
Every line does one job. The H1 is required; skipping the blockquote leaves a model guessing what the site does before it opens a single link. A real file groups links under H2s by purpose, not by site nav, which is usually the opposite of how a main menu is organized (the trimmed example above uses one group to keep it short). The colon-and-description after a link is the actual value - the one-line pitch that decides whether a model with limited room fetches that URL or skips it. The spec also allows an H2 titled "Optional" for exactly that same secondary material.
llms-full.txt is a different animal
An index is what llms.txt is. llms-full.txt inlines the content itself: the same pages converted to clean Markdown and concatenated into one file, sometimes running to tens of thousands of words, so a model can ingest it in a single fetch instead of following links one at a time. Platforms that generate llms.txt automatically - Mintlify, GitBook, Fern - usually generate llms-full.txt alongside it, since the pipeline is the same either way. A docs site under a few hundred pages, that's reasonable to keep fresh. A product catalog running to thousands of pages is too large to be useful and too expensive to keep current.
The honest cost of maintaining one
Publishing the file once is an afternoon. Keeping it honest is the part nobody budgets for.
Every link needs to still resolve, still point at the current version of that page, and still match what's actually there. Rename a doc or ship a new API version, and the file drifts quietly out of sync, because nothing breaks when it does. A stale llms.txt doesn't 404 - it just feeds a model outdated information with total confidence.
For a docs site with one team, that upkeep is a checklist item on the same review that ships doc changes. For a marketing site publishing several posts a week, deciding what belongs in a curated index becomes an editorial job of its own. Ask the question you'd ask before any other content investment: what does success look like, and how would you know? What's actually worth measuring in link analytics applies the same discipline to a different channel.
What Actually Gets You Cited Today
Set the file aside and look at what the evidence actually supports.
robots.txt directives work because named crawlers check for them before every fetch - OpenAI, Anthropic, and Google all publish which user agents honor which rules, and the mechanics of that are covered here rather than repeated in this post. Clean, fast, JavaScript-free HTML gets read completely instead of partially; a page that needs a client-side render to show its content may be abandoned before your text ever loads, which is also why redirect hygiene affects how a page gets treated whether the reader is a person or a model. Structured data - Article and FAQPage schema that matches the visible text - gives a retrieval system a shortcut to what the page claims, and Google's own guidance treats this as ordinary technical SEO, not an AI-specific trick. Being mentioned elsewhere - in reviews, forums, other people's roundups - correlates with getting cited more strongly than almost anything on your own domain, because the model weighs a reputation built across the web, not just one page.
None of that is new advice dressed up for an AI moment. It's the same list you'd hand someone for classic search, aimed at a retrieval system instead of a ranking algorithm. llms.txt doesn't appear anywhere in it, which is the whole reason this post opened with a caution instead of a tutorial.
The Verdict: Should You Publish One in 2026
So, should you ship one? It depends which side of this you're standing on.
If your docs already sit on a platform that generates llms.txt for free, there's little argument against it. The audience that reads it - coding assistants, IDE agents, the handful of documentation-aware tools built specifically to look for the file - is exactly the audience a developer product wants, and the cost is close to zero because someone else's pipeline already produces it.
If you run a marketing site, a blog, or anything a shopper reaches through a general-purpose assistant, I'd hold off on hand-rolling one. The upkeep is real, the readership among the assistants that actually send you traffic is unconfirmed, and an hour spent curating a file nobody's proven reads is an hour not spent on things with an actual paper trail: crawlable pages, structured data that matches your content, a reputation built from being cited elsewhere. Ship the file if it's free. Don't build a content strategy around it.
That's roughly where the ai cluster on this blog keeps landing, too: two minutes of attention, zero minutes of anxiety.
Related on the Blog
Frequently asked questions
What is llms.txt?
llms.txt is a plain Markdown file published at the root of a website - yourdomain.com/llms.txt - that lists the pages the site owner considers most useful for an AI model to read, each with a short description. Jeremy Howard of Answer.AI proposed the llms.txt standard in September 2024, and the current specification lives at llmstxt.org. It curates; it does not restrict or grant access to anything.
Is llms.txt the same as robots.txt?
No. robots.txt is an access-control convention that named crawlers check before fetching a page, defined by RFC 9309 and honored by decades of practice. llms.txt is an advisory reading list with no enforcement mechanism behind it - nothing requires a model to check for it, and most don't.
Do AI models like ChatGPT or Claude actually read llms.txt?
Mostly not yet. Coding tools and IDE agents such as Cursor, Windsurf, and GitHub Copilot do look for the file when pointed at a documentation site, and platforms like Mintlify auto-generate it for the docs they host. No general-purpose consumer assistant has confirmed it checks the file before answering a question.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short index of links with one-line descriptions, meant to orient a model to a site's most useful pages. llms-full.txt is a much larger file that inlines the actual page content as clean Markdown, concatenated into one document a model can ingest in a single fetch. Platforms that generate one typically generate both, since the underlying pipeline is the same.
Does llms.txt help with SEO or getting cited by AI search?
There's no evidence it does, and Google's John Mueller has compared it directly to the keywords meta tag: a field almost nobody outside the publisher ever checked. What has evidence behind it is answer-first content, structured data that matches the page, clean crawlable HTML, and being mentioned on other sites, none of which require an llms.txt file at all.
Should I add an llms.txt file to my website in 2026?
If your documentation platform already generates one for free, there's no real reason not to keep it. If you'd have to hand-curate and maintain it yourself for a marketing site or blog, the honest answer is that confirmed readership doesn't currently justify the ongoing effort, so put that time into crawlability and content quality first.
Try Elido
Paste a URL, get a working short link
No signup. Link lives for 30 days. Sign up to keep it forever.
Free, no signup required · 2 per day