llms.txt
Also known as llms-full.txt, LLM site map
4 min read · updated 2026-09-09
Definition
llms.txt is a markdown file at a site's root that describes what the site is and links its most useful pages, written for a language model rather than a browser. It is a proposed convention rather than a standard anyone is obliged to honour, and it is not an access-control file — that is robots.txt.
What goes in it
The convention is short: an H1 with the site or product name, a blockquote summarising it in a sentence or two, then curated link lists under H2s, each link followed by one line of what it is.
It is a map, not a mirror. The companion `llms-full.txt` is the mirror — the full text of the documentation in one file, for a model that wants the content rather than the index.
The most useful sections are the ones a model can act on: what the product is, who it is for, and which page answers which question. A positioning paragraph written for retrieval — the sentence you would want quoted when somebody asks what you do — earns its place.
Generate it, do not write it
A hand-maintained `llms.txt` sitting in a static folder is a file that describes last quarter's product. It drifts the same way a hand-written sitemap drifts, and with the same symptom: nothing fails, and the file quietly stops being true.
Generating it from the same data the pages render means it cannot disagree with the site. Ours is a route handler built from the crawler catalogue; every line in it is a line the directory pages also render.
Whether anything reads it
Honestly: adoption is uneven and unannounced. No operator publishes a commitment to read llms.txt, and no verification exists that any given assistant fetched it because it was llms.txt rather than because it crawls everything.
What is observable is the fetch. A crawler requesting `/llms.txt` is the clearest single signal that a machine is reading your site deliberately rather than sweeping it — which is why our collector treats a hit on that path as evidence rather than as an asset request.
How it differs from a sitemap
A sitemap is a machine-readable list of URLs with metadata about when they changed. It exists so a crawler can find everything and prioritise recrawling; it says nothing about what any page contains or which of them matter.
llms.txt is the opposite trade. It is prose, it is curated rather than exhaustive, and its value is in the one-line descriptions — a model reading it should be able to decide which page answers a question without fetching all of them. A sitemap listing four hundred URLs and an llms.txt listing twenty are both doing their job.
They also fail differently. A stale sitemap wastes crawl budget on pages that moved. A stale llms.txt actively misinforms — it describes a product that has changed, in the file a model is most likely to treat as authoritative about what the product is. That asymmetry is the argument for generating it: a sitemap that drifts is inefficient, an llms.txt that drifts is wrong.
What goes wrong
- Treating it as access control. llms.txt says what your site is; robots.txt says who may fetch it. Putting a disallow in the wrong file achieves nothing.
- Shipping it as a static file and forgetting it. Measured across two projects here: the static ones had drifted, the generated one had not.
- Duplicating the whole site into it. It is a map. `llms-full.txt` is where the full text goes, if you want one at all.
Worth doing
- Link the HTML pages, not just describe them — a model that wants more should be one fetch away.
- Watch for fetches of the file itself. It is the cleanest AI-readership signal available at the server.
Questions
- Is llms.txt a standard?
- No. It is a proposal that has gathered enough adoption to be worth publishing, and no operator has committed to honouring it. Treat it as cheap and possibly useful rather than as something that owes you anything.
- Does llms.txt affect whether AI crawlers can access my site?
- No. Access is robots.txt, plus whatever your server enforces. llms.txt is descriptive only.
- Should I write llms-full.txt as well?
- Only if you have documentation worth reading in full and a reason to want it quoted verbatim. It is a large file to keep current, and a stale one is worse than none. Generating it from the same source as the pages is what makes it affordable; hand-maintaining 50 KB of markdown is not.
Related
See which AI assistants are reading your site.
AI crawlers take your HTML and leave without running a line of JavaScript, so a browser tag records nothing. TrueStat reads them server-side, checks each address against what the operator publishes, and shows you which pages were fetched — including the ones that returned a 404.