One of the recurring questions we get in buyer conversations this quarter is a short one: does the site need an llms.txt file. Someone in marketing has read a piece calling it the new robots.txt for AI. Someone in legal is unsure what shipping it means. Someone in engineering has been asked to add it to the backlog. Everyone is looking at each other for a decision that is not expensive either way, and that is usually a sign the answer is being made more interesting than it is.
I have been watching this one since the proposal landed in September 2024. The honest version, after sitting with the field data and the platform side both, is that the cost of shipping a competent llms.txt is close to zero and the benefit at an enterprise commerce site today is also close to zero. That is not the same answer as doing nothing. It is the answer: do it, do it cheaply, and do not let anyone build a six-month programme around it.
Two pieces of evidence moved me there. A June 2026 Ahrefs study with the honest arithmetic nobody else had bothered to publish, and a line John Mueller said on a Google podcast the same week. Together they make it very hard to treat llms.txt as the opportunity vendors were selling at the start of this year.
A File Almost Nobody Is Reading
The specification itself is short and sensible. Jeremy Howard of Answer.AI proposed /llms.txt on 3 September 2024 at answer.ai as a plain markdown file at the root of a site, listing what the site is and where its canonical content lives, written for large language models rather than crawlers. The spec moved to llmstxt.org, reached a version 2, and the AI labs themselves publish one: OpenAI, Anthropic, Stripe, Cloudflare, Vercel. The intellectual case is reasonable. If a model is going to summarise what you are, a markdown page you control is a better source than scraping your navigation.
Then Ahrefs went and counted. In We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, published 15 June 2026 by Louise Linehan, Content Marketer at Ahrefs, the finding is on the tin: of 137,210 domains with traffic in May 2026, roughly 28% published a valid llms.txt file, and of those 38,000 files, 97% received zero requests in the month. The remaining 3%, about 1,100 sites, received all of the measured llms.txt traffic.
Linehan is careful about the 28%. Ahrefs’ analytics customers skew technical and SEO-aware, so the real web-wide adoption is lower. The 97% figure, however, is a count of requests that did not happen, and it holds. Her verdict in the piece is short: “the cons outweigh the pros right now”, and on AI search specifically, “an llms.txt file is largely decoration.”
When the File Is Read, SEO Tools Are Doing the Reading
The more interesting detail inside the study is the breakdown of who made the 3% of requests that happened. Ahrefs classified every request to /llms.txt by user agent. SEO audit tools were the single largest category at 21.7% of requests. Unidentified bots were 14.9%, general web crawlers 13.1%, tech profiling tools 11.6%. AI retrieval bots, the ones you might expect to actually consume the file to answer a user’s question, were 1.1%. AI assistants were 2.5%. AI agents and agentic infrastructure were 10.5%. Combined, all AI-related categories reached 19.5% of requests. The rest was tools, crawlers and the research industry.
That matters for the buyer decision. The file is not being ignored because it is unfindable. It is being read, mostly by software that was already reaching your site through sitemaps and schema. The theory of the file was that models would consult it when forming an answer. The field data says they rarely do.
Google’s Own Position Is Blunter
Days after the Ahrefs study, John Mueller, Senior Search Analyst at Google, discussed llms.txt on Search Off the Record, the Google Search team’s own podcast. His position, reported 17 June 2026, is that language models cannot use the file to decide which site to surface, because every site self-describes as the best in its category. His line: “It’s basically you’re telling these systems, like, I have the best website ever.” The one narrow use he granted is navigational: once an AI agent is already on your site and trying to complete a task, the file might help it find the right page.
That is a precise claim. Discovery: no. Within-site navigation by an agent already on your estate: possibly. For a buyer, the distinction is the one that matters. If the goal is to be surfaced in answers, this is not the lever. If the goal is to be navigated competently by an agent already in your checkout, it might be.
The 3% That Does Get Read
Linehan does not call llms.txt dead. The 3% of sites that do get traffic to the file are a mixed group, and the pattern that comes out of her breakdown is that the files getting read live on sites with developer documentation and a technical audience. The AI labs’ own developer docs are the clearest cases. Stripe’s documentation, Cloudflare’s, Anthropic’s. These are sites whose readers are themselves building with LLMs and whose content is written to be read by one.
That is a narrow success case. A mid-market commerce site with a product catalogue and a blog is not that site. A SaaS vendor with deep developer docs is closer, and in that case the file is worth maintaining with the same discipline as an open API reference. The question for everyone in between is whether the content you would list in llms.txt is content a model would benefit from, and for most buyers the honest answer is that your sitemap already carries it and your Product schema already describes it.
What I Would Do, and What I Would Not
I would ship llms.txt, once, at close to zero cost, and move on. The reason is not that I expect it to drive traffic. It is that the file is a low-cost signal to the small number of agents that do use it, it is a convention your technical procurement counterpart is going to look for, and on a WordPress install the cost of shipping it is a plugin install and a review.
I would not build a programme around it. No six-month roadmap, no dedicated owner, no KPI tied to AI referral from this file. The Ahrefs data does not support that investment, and Mueller’s position suggests the search side of Google will not either.
The Build: What This Costs on WordPress
The practical shape inside a WooCommerce or enterprise WordPress install is small. Several plugins on the wordpress.org directory produce the file and keep it current from your published content. Website LLMs.txt reads your pages and posts, respects the noindex rules your existing SEO plugin already enforces (Yoast SEO, Rank Math, SEOPress, AIOSEO), and outputs both llms.txt and the longer llms-full.txt. The LLMs.txt and LLMs-Full.txt Generator does the same job and names Yoast and Rank Math as supported companions. Mira LLM Optimizer covers the same ground.
Setup is an afternoon at most. The content model that matters is on your side rather than the plugin’s: what do you want the file to say about you, which pages are the canonical answer to the questions a model might summarise, and what should be excluded because it is time-sensitive or private. For a commerce catalogue the answer is usually not every product page. A model is better served by a short description of what the store sells, a link to the category trees, and the policy pages that answer fulfilment, returns and shipping questions.
Three operational habits make a difference once the file exists. First, keep it in version control. The file is content, it will drift, and you want the same review you give any published page. Second, treat it as part of the robots policy, not separately. If you are blocking training crawlers in robots.txt, your llms.txt should not be describing content you are not making available to them. The two files need to agree. Third, log who reads it. The 3% is small, and the breakdown of what sits in your 3% is the one piece of local evidence worth having. Ahrefs’ categories can be checked against your own logs in a quarter.
What I would not do on WordPress: pay for the Pro tier of an SEO plugin specifically to generate llms.txt. The free standalone plugins are good enough for the job being done, and the file is thin enough that owning its output directly is worth more than the convenience.
The Honest Answer for the Buyer
If this is the question that reached a management meeting, the time to resolve it is the same meeting. The cost of shipping llms.txt is small, the measured return today is small, and the opportunity cost of making it bigger than that is a planning slot that would do more work somewhere else. Ship it, respect the robots policy, review it twice a year, and give the real attention to the content and schema work that the Ahrefs breakdown shows AI retrieval bots actually consume.
The organisations worth speaking to in a year are the ones that made twenty decisions this small, rather than the five that turned into programmes. If that is the conversation you want to have about your own estate, that is the one we are set up to have.
Last modified: November 10, 2026
