In this article
- What llms.txt is
- What the evidence says
- What actually decides whether AI can read you
- Structure is not the same as being worth citing
- What I would do, in order
We publish an llms.txt file on our own site. It is a short markdown file that describes Geology in two sentences and lists the pages we would most like a language model to read. I am not going to pretend it is doing much. When people ask me whether they need one, my answer is "sure, if you have ten minutes, but please fix three other things first."
"llms.txt" is searched about 5,400 times a month in the US, and plugins now generate the file in one click. Here is what it is, what the evidence says, and what I would check before you spend any more time on it.
What llms.txt Is
llms.txt is a proposed convention, documented at llmstxt.org: a plain markdown file at /llms.txt with a short description of the site and a curated list of links, sometimes with a fuller /llms-full.txt holding the content itself. It is not a standard from any search engine or AI company, and it does not grant or block access to anything. That job still belongs to robots.txt.
What the evidence says
In June 2025 Google's John Mueller wrote on Bluesky: "FWIW no AI system currently uses llms.txt." He added that consumer chatbots fetch pages for training and grounding, "but none of them fetch the llms.txt file," which he said is obvious from server logs (Search Engine Roundtable).
You can check this on your own site in a few minutes. Search your access logs for requests to /llms.txt and note which user agents made them. If the only visitors are SEO tools and curious developers, you have your answer.
None of that makes the file harmful. It is cheap to publish, and some developer tools and coding assistants do read documentation in this format, so for a software company with public docs it can be a real convenience. For most brands trying to be recommended in ChatGPT, Perplexity or Google's AI answers, it is simply not where the gains are.
What actually decides whether AI can read you
File or settingWhat it controlsWhy it matters more than llms.txtrobots.txtWhich crawlers may fetch which pathsA blocked AI crawler cannot read the page at allServer-rendered HTMLWhat a fetcher sees without running JavaScriptMany AI fetchers do not run JavaScriptXML sitemapWhich URLs exist and when they changedSearch engines that feed AI answers use it for discoveryPage status and canonicalsWhich version of a page countsBroken or conflicting signals keep pages out of indexes
We keep a list of the AI crawlers and what each one does in robots.txt for every AI crawler. The mistake we run into is rarely a missing llms.txt. It is a robots.txt rule (how to set one for AI crawlers), or a security or CDN setting, that blocks GPTBot, OAI-SearchBot or PerplexityBot without anyone having decided to.
Structure is not the same as being worth citing
A lot of the llms.txt enthusiasm comes from a broader belief that tidy, machine-friendly structure is what gets a page cited. We tested a closely related claim. In our study of 3,352 AI citations across B2B SaaS buying queries, schema markup showed almost no relationship with how often a page was cited (r = 0.103), and pages with FAQ blocks were cited slightly less often than pages without them. What did line up with citations was depth of coverage and outbound links to good sources (GEO for B2B SaaS).
Structure helps machines parse a page. It does not make the page worth quoting.
What I would do, in order
- Check robots.txt for AI crawler rules nobody chose. Our free robots.txt tester shows them crawler by crawler.
- Fetch your key pages without JavaScript and confirm that prices, product names and your main claims are in the HTML.
- Keep the sitemap current (our free sitemap checker flags broken entries) and submitted in Google Search Console and Bing Webmaster Tools. Bing's index has long been one of the sources behind ChatGPT's web search, and it is the one most teams forget.
- Make the pages worth citing: cover the buyer's question completely, put the facts early, and link your sources.
- Then, if you like, add llms.txt. Ours took a few lines. Just do not report it as an AI visibility win until your logs show something reading it. If llms.txt becomes a real standard, the sites that did steps one to four will benefit from it most anyway.
#ai #aicrawler #seo #agents #software #coding #development #engineering #inclusive #community
Do You Need an llms.txt File? What It Does, What It Does Not, and What to Fix First
