What is llms.txt?
llms.txt is a Markdown file that gives AI agents a short, structured overview of a website. Jeremy Howard proposed it in 2024 as a way to "provide LLM-friendly content": brief background on the site, guidance, and links to detailed Markdown pages.
The problem it addresses is that web pages are built for people. The proposal notes that an HTML page wraps its information "in navigation, ads, and JavaScript," which makes it hard for an AI system to extract clean text, and that most websites are too large to fit in a model's context window. llms.txt points an agent straight to the information it needs.
It's a proposal, not a formal web standard. Its own description is "a proposal to standardise on using an /llms.txt file to provide information to help agents use a website."
How does an llms.txt file work?
An agent reads the llms.txt file first, then follows its links to the pages it needs. The proposal says the file should stay "small enough to fit in context," with the detail behind the links, "fetched only when needed."
The proposal pairs the file with two other ideas:
- Markdown versions of pages at the same URL with
.mdadded, such aspage.html.mdorpage.md. - Link relations that help agents find them:
rel="alternate" type="text/markdown"pointing to a page's Markdown version, andrel="describedby"pointing to the llms.txt file that covers it.
What goes in an llms.txt file?
The format is ordered Markdown sections:
- An H1 with the name of the site or project. This is the only required section.
- A blockquote with a short summary.
- Optional paragraphs or lists with more detail, without headings.
- H2 sections of links, each a list of
[name](url)links with optional notes after a colon. - An "Optional" section, by convention, for secondary links an agent can skip.
A file for a moving company could look like this:
# Example Moving Co.
> Local and long-distance moving company serving the Example City area.
## Services
- [Local moving](https://www.example.com/local-moving.md): Residential moves within the metro area
- [Long-distance moving](https://www.example.com/long-distance.md): Interstate moves and how quotes work
## Company
- [Service area](https://www.example.com/service-area.md): Cities and counties served
- [Contact](https://www.example.com/contact.md): Phone, hours, and quote request
## Optional
- [Moving checklist](https://www.example.com/moving-checklist.md)
The proposal's writing tips: use concise, clear language, give each link a brief description, avoid unexplained jargon, and test the file by asking an agent questions about the site with only the llms.txt as a starting point.
Where does llms.txt go?
At the site root, at /llms.txt, or at any subpath, such as /docs/llms.txt. A file covers the URLs under its path. When more than one file applies, the proposal says agents should use the most specific one.
How is llms.txt different from robots.txt and a sitemap?
They do different jobs:
| File | Purpose |
|---|---|
| robots.txt | Tells automated tools what access to the site is acceptable |
| sitemap.xml | Lists the indexable pages on the site for search engines |
| llms.txt | Gives AI agents a curated overview and links to LLM-friendly content |
The proposal says a sitemap isn't a substitute because it usually doesn't list LLM-readable versions of pages, doesn't include helpful external links, and covers more content than fits in a model's context window.
Does llms.txt help with Google or AI search rankings?
Not with Google. Google's guidance for AI Overviews and AI Mode says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." To be a supporting link there, a page has to be indexed and eligible to show in Google Search with a snippet.
The proposal's author describes where the file is used most: "llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials." He also reports that thousands of sites publish one and that Chrome's Lighthouse checks for one as part of its agentic browsing audits.
The proposal also lists "a business outlining its structure and policies" as a use. For a local business, llms.txt is an addition that some agents may read. It doesn't replace crawl access, indexable pages, or accurate business information, which are covered in what is answer engine optimization.