llms.txt: what it is, how to write one, and whether it helps

What llms.txt is, and what it is not

llms.txt is a plain Markdown file served at the root of a website, at /llms.txt, that tells a language model what the site contains and where its most useful pages are. Jeremy Howard of Answer.AI published the proposal at llmstxt.org on 3 September 2024. It is a community convention with a GitHub repository, not a web standard, and nothing obliges any model or crawler to look for it.

The idea rests on one observation from the proposal: "At the moment the most widely and easily understood format for language models is Markdown. Simply showing where key Markdown files can be found is a great first step." A model with a limited context window cannot read a whole site, and an HTML page carries navigation, scripts and boilerplate that waste that context. A short, curated Markdown index lets the model pick the right page and fetch a clean version of it.

It helps to be precise about what the file is not, because the name invites comparisons.

• It is not robots.txt. robots.txt tells crawlers what they may fetch; llms.txt tells a model what is worth fetching. The two are complementary, and robots.txt must not block llms.txt for the second to be of any use. See controlling AI crawlers with robots.txt.
• It is not a sitemap. A sitemap lists every URL for a crawler that intends to index everything. llms.txt is selective and carries a human-written description per link.
• It is not a consent or policy file. It says nothing about training permissions, and no provider treats it as one.

The format, line by line

The proposal is short enough to hold in your head. In order, a file contains:

1. An H1 with the name of the project or site. The proposal calls this the only required section.
2. A blockquote with a short summary, "containing key information necessary for understanding the rest of the file".
3. Zero or more Markdown blocks of any kind except headings: paragraphs, lists, notes on how to use the site.
4. Zero or more H2 sections, each containing a list of links. Every list item is a required Markdown hyperlink [name](url), then optionally a colon and notes about that URL.
5. An ## Optional section, by convention last, for secondary links an agent can drop when it needs a shorter context.

Two details matter more than they look. First, the description after the colon is where the value is: a model deciding which page to fetch reads that sentence, not the page, so write it as a direct answer rather than a teaser. Second, the proposal recommends that each linked page also be available as clean Markdown at the same URL with .md appended, so a model that follows a link gets text rather than rendered HTML. That is the subject of serving Markdown to LLMs.

A working example

This is an abbreviated version of the file this site generates, with the domain replaced. Every description is the page's own short answer or glossary definition, taken from the same CMS field that renders on the page.

# Citable

> Citable is an open field guide to Answer Engine Optimization (AEO), Generative Engine Optimization (GEO) and LLM-readable content.

Each page is available as Markdown by appending `.md` to its URL. The full text of every guide is in https://example.com/llms-full.txt.

## Guides: Fundamentals

- [What is Answer Engine Optimization (AEO)?](https://example.com/guides/what-is-answer-engine-optimization.md): AEO is the practice of structuring content so that AI answer engines can retrieve, understand and cite it.

## Glossary

- [llms.txt](https://example.com/glossary/llms-txt.md): A proposed convention for a Markdown file at the root of a website that gives language models a curated overview of the site.

## Optional

- [Sitemap](https://example.com/sitemap.xml): every URL on the site
- [About](https://example.com/about): how the site is built

llms-full.txt

llms-full.txt is a companion convention rather than part of the core proposal. It is a single Markdown document containing the full text of every page the index points to, so that a tool can load an entire documentation set in one request. Mintlify's documentation states that it "automatically hosts an llms-full.txt file at the root of your project", alongside the index, for every site on its platform, which is a large part of why the pattern is now common.

The two files divide labour cleanly: llms.txt is small and read first; llms-full.txt is large and read only when a tool has decided it wants everything. If you generate one, generating the other is a loop over the same content.

Generating llms.txt from a CMS

Do not write the file by hand. A hand-written index goes stale the first time someone publishes a page without updating it, and a stale index sends a model to the wrong place. Treat llms.txt as a rendering of your content model, like a sitemap or an RSS feed.

In a headless CMS the ingredients already exist: a guide has a title, slug, category and short answer; a glossary term has a term and a definition. The index is those fields, formatted. The one thing worth modelling deliberately is the description: give every content type a short, self-contained summary field and the file writes itself, while the same field feeds your meta description and structured data. See modelling content for answer engines in a headless CMS.

In Next.js a route handler is enough. This is a simplified version of the handler behind this site's own file:

// app/llms.txt/route.ts
import { getAllGuides, getSiteConfig } from "@/lib/cms";

export const revalidate = 600; // regenerate at most every ten minutes

export async function GET() {
  const [site, guides] = await Promise.all([getSiteConfig(), getAllGuides()]);

  const lines = [
    `# ${site.name}`,
    "",
    `> ${site.description}`,
    "",
    "## Guides",
    "",
    ...guides.map(
      (g) => `- [${g.title}](${site.url}/guides/${g.slug}.md): ${g.shortAnswer}`,
    ),
  ];

  return new Response(lines.join("\n") + "\n", {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "Cache-Control": "public, s-maxage=600, stale-while-revalidate=86400",
    },
  });
}

The same shape works in any framework: read the published stories, build a string, return it with a text content type. If your CMS fires a webhook on publish, use it to purge the cached response so the file updates on change rather than on a timer.

Who actually reads it

This is the part most write-ups skip. As of early 2026, no major AI provider has said that its consumer products read llms.txt.

Google is the clearest. Its documentation on AI features says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." In April 2025 Google's John Mueller wrote on Reddit, as reported by Search Engine Journal, that "none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)", and compared the file to the keywords meta tag: a claim a site makes about itself that a reader would still have to verify by reading the site.

OpenAI's crawler documentation describes GPTBot, OAI-SearchBot and ChatGPT-User, among others, and how to control them in robots.txt; it says nothing about those crawlers reading llms.txt on your site. Anthropic's page on ClaudeBot, Claude-User and Claude-SearchBot is the same: robots.txt directives, no mention of llms.txt. Perplexity's documentation for PerplexityBot and Perplexity-User follows the same pattern. If these products fetched the file, their crawler pages would be the place to say so.

There is an irony here. The developer documentation sites of OpenAI, Anthropic and Perplexity each publish an llms.txt of their own, as does Cloudflare, and llmstxt.org names the OpenAI, Anthropic and Gemini developer docs as examples. The teams building the models find the file useful for their docs; the products they ship have not committed to reading it from yours.

Why it is still worth ten minutes

Given all that, the case for the file is modest and practical rather than strategic.

• It is nearly free. The generator above is a few dozen lines that read fields you already have, with no ongoing cost once it is wired to publish.
• Agents fetch it on demand. The realistic reader is not an indexing crawler but a coding assistant, research agent or retrieval pipeline that has been pointed at your domain and wants a map of it. Documentation tools adopted the file for that use. Your server logs will tell you whether it is happening: look for requests to /llms.txt and to .md URLs.
• It forces good hygiene. A useful index needs a one-sentence description for every page, absolute canonical URLs and a Markdown rendition of each page. All three help every retrieval system, whether or not it reads the index, which is the logic behind Answer Engine Optimization generally.
• It is your own best RAG source. If you build internal search or a support assistant over your content, llms-full.txt is already the corpus and llms.txt the routing table.

Publish it, generate it, and do not build a strategy on it. When a provider announces that it reads the file, you will already have one.