---
title: "llms.txt: what it is, how to write one, and whether it helps"
subtitle: "The exact format of the llmstxt.org proposal, a working example, how to generate the file from a CMS, and an honest account of who reads it."
canonical: https://citable.wiki/guides/llms-txt-what-it-is-and-how-to-write-one
category: technical
author: "Endrit Krasniqi"
date_published: 2026-03-02
date_modified: 2026-03-02
license: CC BY 4.0
---

# llms.txt: what it is, how to write one, and whether it helps

*The exact format of the llmstxt.org proposal, a working example, how to generate the file from a CMS, and an honest account of who reads it.*

## Short answer

llms.txt is a proposed convention, published by Jeremy Howard of Answer.AI in September 2024, for a Markdown file at /llms.txt that gives language models a curated index of a site: an H1 title, a blockquote summary and H2 sections of links with one-line descriptions. No major AI provider has confirmed reading it, but it costs almost nothing to generate from a CMS and is widely published by documentation sites.

## Key takeaways

- llms.txt is a Markdown index at the site root: one H1, a blockquote summary, and H2 sections of links with one-line descriptions.
- Only the H1 is required; everything else in the format is optional, which is why the file is so easy to generate.
- As of early 2026 Google, OpenAI, Anthropic and Perplexity document their crawlers without any mention of reading llms.txt.
- Google's guidance for AI features states that no new machine-readable files or AI text files are needed to appear in them.
- Generate the file from CMS fields rather than writing it by hand, so it can never drift from the published pages.
- The realistic payoff is on-demand use by agents and coding assistants, not ranking or citation gains in consumer answer engines.

## What llms.txt is, and what it is not

**llms.txt** is a plain Markdown file served at the root of a website, at `/llms.txt`, that tells a language model what the site contains and where its most useful pages are. Jeremy Howard of Answer.AI published the proposal at [llmstxt.org](https://llmstxt.org/) on 3 September 2024. It is a community convention with a GitHub repository, not a web standard, and nothing obliges any model or crawler to look for it.

The idea rests on one observation from the proposal: "At the moment the most widely and easily understood format for language models is Markdown. Simply showing where key Markdown files can be found is a great first step." A model with a limited context window cannot read a whole site, and an HTML page carries navigation, scripts and boilerplate that waste that context. A short, curated Markdown index lets the model pick the right page and fetch a clean version of it.

It helps to be precise about what the file is *not*, because the name invites comparisons.

- It is not `robots.txt`. `robots.txt` tells crawlers what they may fetch; `llms.txt` tells a model what is worth fetching. The two are complementary, and `robots.txt` must not block `llms.txt` for the second to be of any use. See [controlling AI crawlers with robots.txt](/guides/controlling-ai-crawlers-with-robots-txt).
- It is not a sitemap. A sitemap lists every URL for a crawler that intends to index everything. `llms.txt` is selective and carries a human-written description per link.
- It is not a consent or policy file. It says nothing about training permissions, and no provider treats it as one.

> **Definition: llms.txt**
>
> A Markdown file at `/llms.txt` with an H1 site name, an optional blockquote summary, optional free-text paragraphs, and H2 sections of links written as `- [Title](url): description`. See the [glossary entry](/glossary/llms-txt).

## The format, line by line

The proposal is short enough to hold in your head. In order, a file contains:

1. **An H1** with the name of the project or site. The proposal calls this the only required section.
2. **A blockquote** with a short summary, "containing key information necessary for understanding the rest of the file".
3. **Zero or more Markdown blocks** of any kind except headings: paragraphs, lists, notes on how to use the site.
4. **Zero or more H2 sections**, each containing a list of links. Every list item is a required Markdown hyperlink `[name](url)`, then optionally a colon and notes about that URL.
5. **An **`## Optional`** section**, by convention last, for secondary links an agent can drop when it needs a shorter context.

Two details matter more than they look. First, the description after the colon is where the value is: a model deciding which page to fetch reads that sentence, not the page, so write it as a direct answer rather than a teaser. Second, the proposal recommends that each linked page also be available as clean Markdown at the same URL with `.md` appended, so a model that follows a link gets text rather than rendered HTML. That is the subject of [serving Markdown to LLMs](/guides/serving-markdown-to-llms).

### A working example

This is an abbreviated version of the file this site generates, with the domain replaced. Every description is the page's own short answer or glossary definition, taken from the same CMS field that renders on the page.

```markdown
# Citable

> Citable is an open field guide to Answer Engine Optimization (AEO), Generative Engine Optimization (GEO) and LLM-readable content.

Each page is available as Markdown by appending `.md` to its URL. The full text of every guide is in https://example.com/llms-full.txt.

## Guides: Fundamentals

- [What is Answer Engine Optimization (AEO)?](https://example.com/guides/what-is-answer-engine-optimization.md): AEO is the practice of structuring content so that AI answer engines can retrieve, understand and cite it.

## Glossary

- [llms.txt](https://example.com/glossary/llms-txt.md): A proposed convention for a Markdown file at the root of a website that gives language models a curated overview of the site.

## Optional

- [Sitemap](https://example.com/sitemap.xml): every URL on the site
- [About](https://example.com/about): how the site is built
```

### llms-full.txt

`llms-full.txt` is a companion convention rather than part of the core proposal. It is a single Markdown document containing the full text of every page the index points to, so that a tool can load an entire documentation set in one request. Mintlify's documentation states that it "automatically hosts an `llms-full.txt` file at the root of your project", alongside the index, for every site on its platform, which is a large part of why the pattern is now common.

The two files divide labour cleanly: `llms.txt` is small and read first; `llms-full.txt` is large and read only when a tool has decided it wants everything. If you generate one, generating the other is a loop over the same content.

## Generating llms.txt from a CMS

Do not write the file by hand. A hand-written index goes stale the first time someone publishes a page without updating it, and a stale index sends a model to the wrong place. Treat `llms.txt` as a rendering of your content model, like a sitemap or an RSS feed.

In a headless CMS the ingredients already exist: a guide has a title, slug, category and short answer; a glossary term has a term and a definition. The index is those fields, formatted. The one thing worth modelling deliberately is the description: give every content type a short, self-contained summary field and the file writes itself, while the same field feeds your meta description and structured data. See [modelling content for answer engines in a headless CMS](/guides/modeling-content-for-answer-engines-in-a-headless-cms).

In Next.js a route handler is enough. This is a simplified version of the handler behind this site's own file:

```ts
// app/llms.txt/route.ts
import { getAllGuides, getSiteConfig } from "@/lib/cms";

export const revalidate = 600; // regenerate at most every ten minutes

export async function GET() {
  const [site, guides] = await Promise.all([getSiteConfig(), getAllGuides()]);

  const lines = [
    `# ${site.name}`,
    "",
    `> ${site.description}`,
    "",
    "## Guides",
    "",
    ...guides.map(
      (g) => `- [${g.title}](${site.url}/guides/${g.slug}.md): ${g.shortAnswer}`,
    ),
  ];

  return new Response(lines.join("\n") + "\n", {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "Cache-Control": "public, s-maxage=600, stale-while-revalidate=86400",
    },
  });
}
```

The same shape works in any framework: read the published stories, build a string, return it with a text content type. If your CMS fires a webhook on publish, use it to purge the cached response so the file updates on change rather than on a timer.

**Before you ship llms.txt**

- [ ] The file is at the site root, returns HTTP 200, and is not blocked by robots.txt or a WAF rule
- [ ] The first line is a single H1 with the site name, followed by a blockquote summary
- [ ] Every link is absolute, points at a public page, and has a one-sentence description after the colon
- [ ] Descriptions are generated from a CMS field, not typed into the file
- [ ] Linked pages have a Markdown version at the same URL with .md appended
- [ ] Secondary links live under an Optional heading
- [ ] llms-full.txt exists if you publish more than a handful of pages, and is generated by the same code
- [ ] The response carries a UTF-8 charset and a cache policy that is purged on publish

## Who actually reads it

This is the part most write-ups skip. As of early 2026, no major AI provider has said that its consumer products read `llms.txt`.

Google is the clearest. Its documentation on AI features says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." In April 2025 Google's John Mueller wrote on Reddit, as reported by Search Engine Journal, that "none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)", and compared the file to the keywords meta tag: a claim a site makes about itself that a reader would still have to verify by reading the site.

OpenAI's crawler documentation describes GPTBot, OAI-SearchBot and ChatGPT-User, among others, and how to control them in `robots.txt`; it says nothing about those crawlers reading `llms.txt` on your site. Anthropic's page on ClaudeBot, Claude-User and Claude-SearchBot is the same: `robots.txt` directives, no mention of `llms.txt`. Perplexity's documentation for PerplexityBot and Perplexity-User follows the same pattern. If these products fetched the file, their crawler pages would be the place to say so.

There is an irony here. The developer documentation sites of OpenAI, Anthropic and Perplexity each publish an `llms.txt` of their own, as does Cloudflare, and llmstxt.org names the OpenAI, Anthropic and Gemini developer docs as examples. The teams building the models find the file useful for their docs; the products they ship have not committed to reading it from yours.

> **Warning: What not to expect**
>
> Publishing `llms.txt` will not change how Google's AI Overviews, ChatGPT search or Perplexity rank or cite your pages. Anyone selling it as a citation or ranking lever is selling something the vendors have not confirmed. Treat any before-and-after statistic with suspicion unless the author can show the file being fetched in server logs.

## Why it is still worth ten minutes

Given all that, the case for the file is modest and practical rather than strategic.

- **It is nearly free.** The generator above is a few dozen lines that read fields you already have, with no ongoing cost once it is wired to publish.
- **Agents fetch it on demand.** The realistic reader is not an indexing crawler but a coding assistant, research agent or retrieval pipeline that has been pointed at your domain and wants a map of it. Documentation tools adopted the file for that use. Your server logs will tell you whether it is happening: look for requests to `/llms.txt` and to `.md` URLs.
- **It forces good hygiene.** A useful index needs a one-sentence description for every page, absolute canonical URLs and a Markdown rendition of each page. All three help every retrieval system, whether or not it reads the index, which is the logic behind [Answer Engine Optimization](/guides/what-is-answer-engine-optimization) generally.
- **It is your own best RAG source.** If you build internal search or a support assistant over your content, `llms-full.txt` is already the corpus and `llms.txt` the routing table.

Publish it, generate it, and do not build a strategy on it. When a provider announces that it reads the file, you will already have one.

> **Tip: Where to go next**
>
> The file only pays off if the pages it links to are clean: read [serving Markdown to LLMs](/guides/serving-markdown-to-llms) for the `.md` URL and content-negotiation side. Then make sure the crawlers you want can actually fetch it in [controlling AI crawlers with robots.txt](/guides/controlling-ai-crawlers-with-robots-txt).

## Frequently asked questions

### Is llms.txt an official web standard?

No. It is a community proposal published at llmstxt.org by Jeremy Howard of Answer.AI in September 2024, with a public GitHub repository for discussion. No standards body has adopted it and no AI provider is obliged to honour it. It sits in the same category as humans.txt or security.txt before RFC 9116: a convention that works only to the extent that readers choose to look for it.

### Does Google read llms.txt?

Google has not said that it does. Its documentation on AI features states that you do not need to create new machine-readable files or AI text files to appear in AI Overviews or AI Mode. In April 2025 Google's John Mueller wrote on Reddit that no AI service had said it uses the file and that server logs show they do not check for it, comparing it to the keywords meta tag.

### What is the difference between llms.txt and llms-full.txt?

llms.txt is an index: a short file of links and one-line descriptions that a model reads first to decide what to fetch. llms-full.txt is a companion convention, popularised by documentation tooling such as Mintlify, that concatenates the full Markdown text of every page into one document so a tool can load an entire site into context in a single request.

### Should llms.txt be served as text/plain or text/markdown?

The proposal describes the contents of the file, not its HTTP headers, so either works in practice. Serve it with an explicit UTF-8 charset, make sure robots.txt does not block it, and do not put it behind JavaScript or a login. The file is only useful if a plain HTTP GET returns the Markdown directly.

### How long should llms.txt be?

The proposal sets no limit, but the purpose of the file is to fit in a model's context window alongside the user's task. Keep the root index to the pages you would hand a new colleague on day one, put secondary links under an Optional heading, and move the full text of pages into llms-full.txt or per-page Markdown URLs.

## Sources

1. [The /llms.txt file](https://llmstxt.org/) — Jeremy Howard, Answer.AI (2024)
2. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Google Search Central (2025)
3. [Google Says LLMs.Txt Comparable To Keywords Meta Tag](https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/) — Search Engine Journal (2025)
4. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — OpenAI (2025)
5. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — Anthropic (2025)
6. [llms.txt](https://www.mintlify.com/docs/ai/llmstxt) — Mintlify (2025)

## Related guides

- [Serving Markdown to LLMs: content negotiation and .md endpoints](https://citable.wiki/guides/serving-markdown-to-llms)
- [Controlling AI crawlers with robots.txt: GPTBot, ClaudeBot, PerplexityBot and friends](https://citable.wiki/guides/controlling-ai-crawlers-with-robots-txt)
- [Modelling content for answer engines in a headless CMS!](https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)

---

Source: https://citable.wiki/guides/llms-txt-what-it-is-and-how-to-write-one
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt