For two years, llms.txt has been sold as the "robots.txt for AI": a small file that would help your site show up in ChatGPT, Perplexity and Google's AI Overviews. In 2026 we finally have the logs to test that promise. The short answer: most llms.txt files are never read, Google Search ignores them, and the file's only proven use is far narrower than the hype suggested.
Here is what the evidence says, and what you should actually do. (New to the file itself? Start with our guide to what llms.txt is and how to create one.)
Key Takeaways
- 28% of 137,210 domains published an llms.txt file, but 97% of those files got zero requests in May 2026.
- Most requests came from SEO audit and tech-profiling tools, not AI systems.
- Google says you don't need AI text files or special markup to appear in AI Overviews or AI Mode.
- The file's real use is developer documentation read by AI coding agents.
- robots.txt, crawlable pages and citation-ready content still do the real work.
What llms.txt is, and where it came from
Jeremy Howard of Answer.AI proposed llms.txt on 3 September 2024 at llmstxt.org. The idea: put a Markdown file at your site's root (example.com/llms.txt) that gives language models a clean, short map of your most important pages, so they don't have to parse messy HTML.
The format is simple. It has an H1 with the site name, a short summary in a blockquote, optional notes, then H2 sections listing links with one-line descriptions. Some sites also publish llms-full.txt, which contains the full text of key pages.
Crucially, llms.txt is a community proposal. No standards body has adopted it, and no major search engine has committed to reading it.
What the logs show
Ahrefs ran the most useful test so far: a log study of 137,210 domains, published in June 2026 using May 2026 data.
| Finding | Result |
|---|---|
| Domains studied | 137,210 |
| Share that published an llms.txt file | 28% |
| Share of those files with zero requests in May 2026 | 97% |
| Share of the few requests that came from named AI tools | 19.5% |
| Largest requester category | SEO audit tools (21.7%) |
| Other requesters | Tech-profiling tools (11.6%), GEO tools (5.8%) |
Source: Ahrefs log study, as reported by Pepper (29 Sep 2026) and Agent Skill Packs (Sep 2026).
Read that again: more than a quarter of sites created the file, and almost all of them were never asked for it, even once, in a whole month. When the file was requested, the most frequent visitors were SEO tools checking whether it exists, not AI systems using it.
One more detail matters. According to the same analysis, AI bots do not go looking for llms.txt files that do not exist. In other words, they are not checking for it the way crawlers check robots.txt.
What Google says
Google has been consistent on this for over a year.
- In mid-2025, Google's John Mueller said no AI service had said it uses llms.txt, and pointed out that server logs show they don't even check for it.
- Google's Gary Illyes said in July 2025 that Google does not support llms.txt and is not planning to (as reported by Pepper).
- Google's AI features documentation now states it plainly: "You don't need to create new machine readable files, AI text files, or markup to appear in these features."
The same guide says robots.txt directives for Googlebot remain the control for crawling, and nosnippet, data-nosnippet, max-snippet and noindex control what is shown. llms.txt is not on that list. We unpack that in detail in Does llms.txt help you appear in Google AI Overviews?
What about ChatGPT, Perplexity and Claude?
OpenAI, Anthropic, Perplexity and Microsoft all document their crawlers and robots.txt rules. As of September 2026, none of them has publicly confirmed using llms.txt to choose what appears in its AI search answers. If someone tells you llms.txt gets you into ChatGPT answers, ask them for the evidence. There is none on the public record.
Where llms.txt actually works
There is one place the file earns its keep: developer documentation.
Coding agents and docs platforms, Anthropic, Stripe, Cursor and Mintlify among them, publish or support llms.txt so that a developer can point an AI coding assistant at clean, current documentation. In that setup, a human deliberately sends the agent to the file. It is a convenience for people building with your API, not a signal to search engines.
So the real use case is: "help a coding agent read our docs on request." It is not: "help our marketing pages rank in AI Overviews."
Myth vs. evidence
| Claim you will hear | What the evidence says |
|---|---|
| llms.txt is the robots.txt for AI | It does not control crawling; robots.txt still does |
| It helps you appear in Google AI Overviews | Google says no special AI files are needed |
| AI crawlers read it | 97% of files got zero requests in May 2026 |
| Everyone serious has one, so you need one | 28% have one; almost none are used |
| It's useless for everyone | It helps coding agents read developer docs |
What to do instead
- Leave existing files alone, but stop investing. A llms.txt file does no harm. If you already have one, keep it if it's accurate; just don't spend more time on it.
- Build one only if you publish developer docs. API references, SDK guides and product documentation are the one place it can help real users.
- Get robots.txt right. Decide which AI and search crawlers you allow. That file is still the real control.
- Make pages easy to quote. Server-rendered HTML, clear headings, short direct answers, tables, named authors and current dates are what AI answers actually draw on.
- Measure citations, not files. Track whether ChatGPT, Perplexity and AI Overviews cite your pages for your key questions. Our 12-point checklist for earning AI search citations shows how.
The bottom line
llms.txt was a reasonable idea. Two years on, the logs show it has not become how AI search finds content, and Google has said clearly that it doesn't need it. Treat it as a niche tool for documentation, not as an SEO or GEO lever. Spend the time on content, crawlability and citations instead.
Want to know whether AI Overviews specifically use it? Read Does llms.txt help you appear in Google AI Overviews?
Need a technical SEO audit that focuses on what actually moves AI visibility? See our SEO services or talk to us.
