llms.txt is having a moment. It appears on every AI-visibility checklist, several tools will now generate one for you, and a steady supply of articles will tell you that without one the models cannot understand your site. We check for it too, in our own tool. So it is worth being straight about what it is, where it came from, and how much it is actually worth to you.
What it is
An llms.txt file is a plain-text file at the root of your domain, at yoursite.com/llms.txt, written in Markdown. It gives a short description of what the site is and a curated list of links to the pages that matter, so a language model can find the good material without wading through navigation, cookie banners and markup.
It sits in the same place as robots.txt and looks superficially similar, which is the source of most of the confusion about it.
Where it came from, and what it was for
Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed it in September 2024. The problem he was solving is worth stating precisely, because it is not the problem it now gets sold against.
Context windows are finite. Converting a full HTML page into clean text is lossy and annoying. If you are a developer using a coding assistant and you want it to read a library's documentation, handing it a curated Markdown index is enormously better than pointing it at a documentation site and hoping.
That is the original use case: inference-time access for coding tools and agents. It was not proposed as a search visibility mechanism, and the author did not present it as one.
The part the checklists leave out
No major AI engine has committed to reading llms.txt as part of its production ingestion. It is not documented in any search engine's or answer engine's crawling pipeline. Publishing one does not put you into an index, and the absence of one does not remove you from anything.
That is the honest state of it. It is a sensible proposal with real utility in a specific context, and it is being marketed well beyond that context because it is easy to check and easy to sell.
What the adoption numbers actually look like
We ran the check across 152 B2B software company sites. 78% do not publish an llms.txt.
Two ways to read that. The first is the one you will see in a sales deck: three quarters of your competitors are unprepared, move fast and get ahead. The second is the one we would offer: a file that 78% of a sophisticated, well-resourced group has not bothered with is unlikely to be the load-bearing thing in AI visibility.
The same study found something more useful. Among those sites, 32% have no structured data that describes the business, and 9% send fewer than 250 words of readable text before JavaScript runs. Those are the conditions that actually stop a model from working out what you do.
When it is genuinely worth having
There are cases where we would put one up without hesitating:
- You sell to developers. If your buyers evaluate you inside a coding assistant, this is exactly the audience the proposal was written for.
- You have substantial documentation. A curated index of a large docs site is useful to any agent that lands on it, standard or not.
- Your site is large and its navigation is not obvious. A short map costs you almost nothing.
In all three cases the reasoning is the same: it is cheap, it is harmless, and there is a plausible reader. That is a perfectly good reason to do something. It is not the same as a standard you are failing to meet.
How to write one, if you are going to
Keep it short. An H1 with your company name, a blockquote with one sentence on what you do, then a small number of linked sections with a line of context each. Point at the pages that explain your product, your pricing, and your documentation. Leave out the careers page.
The common failure is generating one automatically and shipping a link dump of every URL on the site. That recreates the problem the file was invented to solve.
One more thing worth checking: we found sites serving a 200 response at /llms.txt with an empty body, and others serving their JavaScript application shell there because the route did not exist. Both look like success to a naive checker. Open yours in a browser and read it.
What we would do first instead
If you have an afternoon and you want it to count:
- Make your structured data describe your business. Organization, Product or SoftwareApplication, FAQPage. Not just the WebPage and BreadcrumbList your plugin emits by default.
- Make sure your content is in the server HTML. If
curlreturns a near-empty page, machines are reading a shell. - Answer the question in the first two sentences of each page, under the heading that asks it. Models quote passages, not pages.
- Then publish an llms.txt, because it takes ten minutes and there is no reason not to.
That order matters more than the list does.
Check where you stand
Our free tool runs four of these checks on your site in about two seconds and shows the evidence for each, including what your schema actually says rather than whether it merely exists. Run it on your site, or read the full findings from the 152-site study.
