Contact

What is llms.txt, and why should your website have one?

Harvesters work and rest under a tree in a golden wheat field, a broad landscape opening behind them. (Pieter Bruegel the Elder, The Harvesters, 1565)
Pieter Bruegel the Elder: The Harvesters, 1565 · The Metropolitan Museum of Art · Public domain
Topic
llms.txt
Updated
Reading time
~3 min read

In short

llms.txt is a Markdown text file placed at the root of a website (/llms.txt) that gives large language models a clean, concise summary of the site and links to its most important pages. Proposed in 2024, it is a community convention rather than an official web standard, and not every AI provider reads it automatically; nevertheless, it is simple to maintain and worthwhile for content-rich sites.

What is llms.txt?

llms.txt is a plain-text Markdown file served at /llms.txt. The proposal was introduced in 2024 by Jeremy Howard to help large language models (LLMs) quickly grasp a website's core structure and facts.

Full HTML pages contain navigation menus, styling markup, and scripts, whereas language models work best with clean, high-signal text inside a limited context window. llms.txt bridges this gap by stating what the site is and where key resources live.

How is the file structured?

The file uses standard Markdown in a predictable order:

  1. H1 heading with the name of the website or project (the only required element).
  2. Blockquote providing a short summary of what the site or organization does.
  3. Body paragraphs with essential facts or context if needed.
  4. H2 sections containing lists of links, each paired with a brief description.
  5. An “Optional” section for secondary links that can be skipped when context space is tight.

Many sites also publish an /llms-full.txt companion file containing the complete text of all pages in a single Markdown document.

How does it differ from robots.txt and sitemap.xml?

Comparison of llms.txt, robots.txt, and sitemap.xml
FileTarget audiencePurposeFormat
robots.txtWeb crawlersSpecifies which paths may or may not be crawledPlain-text directives
sitemap.xmlSearch engine crawlersLists all indexable URLs on the siteXML
llms.txtLanguage models and AI agentsSummarizes the site and curates key linksMarkdown

llms.txt does not enforce access control: it neither blocks nor permits crawlers (that remains the job of robots.txt). The three files complement one another.

Does your website need one?

llms.txt is optional, and major AI providers have not made it a mandatory indexing requirement. Publishing the file alone will not automatically place a company into AI answers.

Still, several practical arguments favor including it:

  • it takes very little effort to generate and maintain;
  • many AI coding assistants, agents, and retrieval tools already look for it;
  • writing it forces a clear, concise articulation of what your company does;
  • as adoption grows, your site is already prepared.

Best practices when creating llms.txt

  • Generate it automatically: build it from the same data source as your pages so it never drifts out of date.
  • Match visible content: include only facts that are also publicly readable on the website.
  • Keep it concise: aim for orientation rather than duplicating every word (use llms-full.txt for full text).
  • Use clean URLs: provide full, working links with a one-sentence description for each.
  • Set realistic expectations: llms.txt supplements, but never replaces, great content and technical SEO.

Frequently Asked Questions

Is llms.txt mandatory?

No. llms.txt is a voluntary community convention rather than a formal web standard. Websites can appear in search engines and AI assistants without it, though it helps language models orient themselves faster.

Does llms.txt improve traditional Google rankings?

There is no evidence that llms.txt directly affects traditional search engine rankings. It is designed for language models and AI agents rather than classic page-ranking algorithms.

Where should the llms.txt file be placed?

In the root directory of your domain so it is publicly accessible at /llms.txt as plain UTF-8 Markdown text without authentication.

Have an idea or an engineering challenge to solve?

Let's talk