llms.txt explained: what it is and whether it matters
llms.txt is a proposed file at the root of a site offering a clean, machine-readable map of what the site knows. It is cheap to publish and currently honoured by very few systems, so treat it as a small bet rather than a strategy.
The idea is simple and reasonable. Websites are built for browsers: navigation, banners, scripts, a great deal of markup wrapped around a small amount of meaning. A language model reading that page has to discard most of it. llms.txt proposes a plain Markdown file at the root of a domain that says, in order of importance, what this site contains and where the substantive documents are.
What goes in it
A heading with the name of the site, a short summary of what the organisation does, and then grouped lists of links with one line of description each. Some sites additionally publish a flattened Markdown version of each page alongside it. That is the whole proposal.
Who honours it?
As of this revision: not many, and none of the large assistant providers have committed to reading it as part of their retrieval pipeline. It has been adopted enthusiastically by developer-documentation platforms, where it genuinely helps coding assistants, and thinly elsewhere.
Anyone telling you it is essential for AI visibility is ahead of the observable situation. Anyone telling you it is worthless is ignoring how cheap it is to publish.
Cheap to publish, unproven to matter. Publish it, do not build on it.
What to do about it
| Do | Do not |
|---|---|
| Publish one if your site has documentation or a knowledge base | Rebuild your content strategy around it |
| Keep it short and genuinely ordered by importance | Dump every URL you have into it |
| Regenerate it when the underlying pages change | Let it drift out of date, which is worse than not having one |
| Keep your sitemap and robots file correct first | Treat it as a substitute for either |
The thing that actually matters more
Every retrieval system in production today reads HTML. Making that HTML clean — server-rendered, with the answer in the text rather than assembled by script, without a consent wall over the content, with honest structured data — does more for machine reading than any new file format will. A site that gets this right needs llms.txt far less; a site that gets it wrong is not rescued by one.
Questions this settles
No. It is a community proposal with a specification and a reference implementation, not something ratified by a standards body or committed to by search and assistant providers.
No. That is what robots.txt directives for individual training crawlers are for. llms.txt is an offer of structure, not a permission setting.
If you have a knowledge base or documentation, yes, because it costs an afternoon. If you have a five-page brochure site, your effort is better spent making those five pages answer real questions.