Does llms.txt Work? 137,000 Domains Say Not Yet
Two weeks ago I added an llms.txt file to this site. It is a single markdown file at the domain root, proposed in 2024 by Jeremy Howard, the co-founder of Answer.AI and fast.ai. The pitch is simple: instead of making a language model dig through your HTML, you hand it a clean index of who you are and where the important pages live. Mine lists my essays, a short bio, and four profile links. Writing it was mostly deciding what to leave out.
On June 15, Ahrefs published the first at-scale measurement of what happens to files like mine. They took the 137,210 domains running their web analytics, checked which ones serve an llms.txt, then read a month of server logs to see who fetched it. Adoption looks healthy: 28 percent of the domains publish the file, about 38,000 sites, a figure Ahrefs itself calls an upper bound because its customer base skews technical. Readership does not. 97 percent of those files received zero requests in May 2026. Not zero citations, not zero ranking lift. Zero fetches. Nothing, human or machine, asked for the file at all.
Who reads the other 3 percent
The surviving traffic is small and strange: roughly 1,100 domains sharing about 22,000 requests, 96 percent of them from bots. Walk down the ranking and the story tells itself. The largest requester category is SEO audit tools, at 21.7 percent. Then anonymous and unidentified bots at 14.9, general web crawlers like Googlebot at 13.1, tech profilers like BuiltWith at 11.6. The category the file was invented for, AI retrieval bots fetching pages to answer live questions in AI search, sits at the bottom of the table: 1.1 percent, 233 requests in a month, across a population of 137,000 domains. Slackbot alone fetched llms.txt files more often than PerplexityBot did. A chat app's link-preview bot outread the AI search engine the standard was supposed to feed.
View data table
| Requester category | Requests | Share of total |
|---|---|---|
| SEO audit tools | 4,776 | 21.7% |
| Other and unidentified bots | 3,278 | 14.9% |
| General web crawlers | 2,871 | 13.1% |
| Tech profiling tools | 2,546 | 11.6% |
| AI agents & agentic infrastructure | 2,302 | 10.5% |
| GEO/AEO readiness tools | 1,278 | 5.8% |
| AI training crawlers | 1,179 | 5.3% |
| llms.txt discoverability bots | 793 | 3.6% |
| Service and social bots | 645 | 2.9% |
| Research bots | 585 | 2.7% |
| AI assistants | 559 | 2.5% |
| AI retrieval bots | 233 | 1.1% |
| Humans | 930 | 4.0% |
Add the GEO scoring tools, the llms.txt checkers, and the research crawlers together and another 12 percent of the traffic is the industry studying itself. So the most reliable audience for a file addressed to language models is currently the people selling readiness reports about it.
There is one genuine reader in the data. Combine the four AI bot categories and they form the largest single bucket, 19.5 percent, and the top two individual AI bots are GPTBot and Claude-Code. The second name is the interesting one. Claude-Code is a coding agent, and it outfetched every AI search and assistant bot in the study. Google's John Mueller has called llms.txt a stopgap for coding tools, and the logs agree with him. The file found an audience. It is just not the one it was written for.
The part I cannot measure
Here is my confession as a site owner: I do not know whether my own llms.txt has ever been read. This site serves static files from a CDN, and my hosting gives me no request log for them. Ahrefs could run this study for one reason only: they sit on the server logs of 137,210 domains. Everyone else who shipped the file this year is in my position, holding a text file and a hope, with no instrument pointed at either.
That detail matters more than the headline number if you buy or sell anything touching AI visibility. Adoption of llms.txt is easy to measure from the outside. You request the file from every domain and count the 200s, which is why every vendor deck quotes adoption. Readership can only be measured from inside the logs, which is why almost nobody quotes it, and why the one company positioned to check produced a figure as unflattering as 97 percent unread. I collect web data for a living, and this is the pattern I keep running into: the number everyone cites is the one that is cheap to observe, not the one that answers the question.
I am keeping my file. It costs nothing to serve, and the bet is explicit: Ahrefs notes that no major AI platform has ever committed to reading llms.txt, so adoption is running on speculation that one eventually will. The study even checked for scouts and found none. AI bots sent zero requests for llms.txt on domains that do not publish one. The reading side of this standard has not shipped yet, and 38,000 sites, mine included, have stocked shelves for a customer who has not walked in. So the next time a deck tells you llms.txt is table stakes, ask for the requests, not the adoption curve. Adoption is 28 percent and climbing. Readership is 97 percent silence.