What is llms.txt, and why should your website have one?

In short
llms.txt is a Markdown text file placed at the root of a website (/llms.txt) that gives large language models a clean, concise summary of the site and links to its most important pages. Proposed in 2024, it is a community convention rather than an official web standard, and not every AI provider reads it automatically; nevertheless, it is simple to maintain and worthwhile for content-rich sites.
What is llms.txt?
llms.txt is a plain-text Markdown file served at /llms.txt. The proposal was introduced in 2024 by Jeremy Howard to help large language models (LLMs) quickly grasp a website's core structure and facts.
Full HTML pages contain navigation menus, styling markup, and scripts, whereas language models work best with clean, high-signal text inside a limited context window. llms.txt bridges this gap by stating what the site is and where key resources live.
How is the file structured?
The file uses standard Markdown in a predictable order:
- H1 heading with the name of the website or project (the only required element).
- Blockquote providing a short summary of what the site or organization does.
- Body paragraphs with essential facts or context if needed.
- H2 sections containing lists of links, each paired with a brief description.
- An “Optional” section for secondary links that can be skipped when context space is tight.
Many sites also publish an /llms-full.txt companion file containing the complete text of all pages in a single Markdown document.
How does it differ from robots.txt and sitemap.xml?
| File | Target audience | Purpose | Format |
|---|---|---|---|
| robots.txt | Web crawlers | Specifies which paths may or may not be crawled | Plain-text directives |
| sitemap.xml | Search engine crawlers | Lists all indexable URLs on the site | XML |
| llms.txt | Language models and AI agents | Summarizes the site and curates key links | Markdown |
llms.txt does not enforce access control: it neither blocks nor permits crawlers (that remains the job of robots.txt). The three files complement one another.
Does your website need one?
llms.txt is optional, and major AI providers have not made it a mandatory indexing requirement. Publishing the file alone will not automatically place a company into AI answers.
Still, several practical arguments favor including it:
- it takes very little effort to generate and maintain;
- many AI coding assistants, agents, and retrieval tools already look for it;
- writing it forces a clear, concise articulation of what your company does;
- as adoption grows, your site is already prepared.
Best practices when creating llms.txt
- Generate it automatically: build it from the same data source as your pages so it never drifts out of date.
- Match visible content: include only facts that are also publicly readable on the website.
- Keep it concise: aim for orientation rather than duplicating every word (use llms-full.txt for full text).
- Use clean URLs: provide full, working links with a one-sentence description for each.
- Set realistic expectations: llms.txt supplements, but never replaces, great content and technical SEO.
Frequently Asked Questions
Is llms.txt mandatory?
No. llms.txt is a voluntary community convention rather than a formal web standard. Websites can appear in search engines and AI assistants without it, though it helps language models orient themselves faster.
Does llms.txt improve traditional Google rankings?
There is no evidence that llms.txt directly affects traditional search engine rankings. It is designed for language models and AI agents rather than classic page-ranking algorithms.
Where should the llms.txt file be placed?
In the root directory of your domain so it is publicly accessible at /llms.txt as plain UTF-8 Markdown text without authentication.