---
title: "Serving Markdown to LLMs: content negotiation and .md endpoints"
subtitle: "How to give language models a plain-text version of every page, without duplicating content or confusing caches and search engines."
canonical: https://citable.wiki/guides/serving-markdown-to-llms
category: technical
author: "Endrit Krasniqi"
date_published: 2026-03-30
date_modified: 2026-03-30
license: CC BY 4.0
---

# Serving Markdown to LLMs: content negotiation and .md endpoints

*How to give language models a plain-text version of every page, without duplicating content or confusing caches and search engines.*

## Short answer

Serving Markdown to LLMs means publishing a plain-text representation of each page next to the HTML one, so that AI crawlers and agents can read it without rendering JavaScript or spending tokens on markup. There are two delivery patterns: a .md suffix on the canonical URL, and content negotiation on the Accept: text/markdown request header, with Vary: Accept and a Link rel=canonical header pointing back to the HTML page.

## Key takeaways

- A rendered DOM costs a model far more tokens than the same content as Markdown, and many AI fetchers do not execute JavaScript at all.
- The .md suffix pattern is the simplest to ship and to debug; llmstxt.org proposes it and Mintlify-style docs platforms link to it from llms.txt.
- Content negotiation serves Markdown from the canonical URL when a client sends Accept: text/markdown; Vercel and Cloudflare both ship this pattern.
- Every Markdown response needs Content-Type: text/markdown, Vary: Accept and a Link header with rel=canonical so caches and search engines stay consistent.
- Generate Markdown from the same CMS fields as the HTML, with YAML front matter for canonical URL, author, dates and licence.
- Copy-as-Markdown and Open-in-ChatGPT buttons reuse the same endpoint and make the feature visible to people, not only to crawlers.

## Why plain text beats the rendered DOM

A language model does not see your page the way a browser does. Whatever fetches it — a training crawler, a search indexer, or an agent acting for a user — receives bytes, and every byte it keeps costs a token. A modern HTML page is mostly bytes the model has no use for: framework bootstrap scripts, hydration payloads, inline SVG icons, class attributes, cookie-consent markup and navigation repeated on every URL. The article you wrote is a small fraction of the response.

Vercel measured this on the blog post in which it announced Markdown delivery in February 2026: the HTML version of that page was around 500KB and the Markdown version 3KB. Cloudflare's [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) documentation describes an `x-markdown-tokens` header on converted responses, alongside an `x-original-tokens` header, precisely so operators can see the difference.

**3KB** — size of the Markdown version of a Vercel blog post whose HTML response was around 500KB, as measured by Vercel (Source: [Vercel, Making agent-friendly pages with content negotiation](https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation))

Token cost is only the first problem. The second is **fetch budgets**: a crawler has a limited number of bytes and requests per host, and an agent answering a live question has seconds, not minutes. The third is **JavaScript rendering**. Google documents a three-phase pipeline for JavaScript sites — crawl, render, index — in which a page "may stay on this queue for a few seconds, but it can take longer than that" before headless Chromium executes it. OpenAI's crawler documentation lists `GPTBot`, `OAI-SearchBot`, `ChatGPT-User` and `OAI-AdsBot` and says nothing about executing JavaScript at all; you should assume a client-rendered page is an empty shell to them.

Markdown solves all three. It is small, it needs no rendering, and its structure — headings, lists, code fences, links — is the structure a model was trained on. It is also a registered media type: [RFC 7763](https://www.rfc-editor.org/rfc/rfc7763) defines `text/markdown` with a required `charset` parameter and an optional `variant`, which is what makes the second delivery pattern below possible.

> **Note: Markdown is not a ranking signal**
>
> No engine has said that offering Markdown makes a page more likely to be cited. What it does is remove the failure modes — truncation, unexecuted JavaScript, noisy extraction — that stop a good passage from being read at all. Treat it as infrastructure, like a sitemap.

## Pattern one: a .md suffix on the canonical URL

The simplest pattern is a second URL. If the HTML lives at `/guides/serving-markdown-to-llms`, the Markdown lives at `/guides/serving-markdown-to-llms.md`. The llms.txt proposal at [llmstxt.org](https://llmstxt.org/) recommends exactly this: pages "provide a clean markdown version of those pages at the same URL as the original page, either with `.md` appended (`page.html.md`) or with the extension replaced by `.md` (`page.md`)". Mintlify's hosted documentation follows the convention and [notes](https://mintlify.com/docs/ai/llmstxt) that page links in its generated `llms.txt` "include a `.md` extension so AI tools can fetch the Markdown version of each page directly".

The advantages are practical. You can open the URL in a browser to check it. Caches never confuse it with the HTML. Any client that has read your `/llms.txt` knows how to reach it. The response should look like this:

```http
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Content-Disposition: inline
Link: <https://example.com/guides/serving-markdown-to-llms>; rel="canonical"
Cache-Control: public, max-age=0, s-maxage=600, stale-while-revalidate=86400
```

Two details matter. `Content-Disposition: inline` stops browsers downloading the file, so people can read it too. The `Link` header with `rel="canonical"` tells search engines that the HTML page is the one to index; Google [documents this HTTP-header form](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) of canonicalisation for non-HTML documents such as PDFs, and Markdown is no different. Leave `.md` URLs out of your XML sitemap.

## Pattern two: content negotiation on Accept

The second pattern serves Markdown from the canonical URL itself when the client asks for it. In server-driven content negotiation the client's `Accept` header lists the media types it can process, each with an optional quality value, and the server picks the best representation. A client that prefers Markdown sends:

```http
GET /guides/serving-markdown-to-llms HTTP/1.1
Host: example.com
Accept: text/markdown, text/html;q=0.8
```

This is the pattern Vercel described in February 2026 for its own documentation, blog and changelog, and it is what Cloudflare's Markdown for Agents implements at the edge for sites that enable it: when a request carries `Accept: text/markdown`, Cloudflare converts the origin's HTML "on the fly" and returns `content-type: text/markdown` with a token count header. You can try both with `curl -H "Accept: text/markdown"` against their sites.

Content negotiation has one well-known trap: caches. A CDN that stores the Markdown response for the canonical URL and then serves it to a browser has broken your site. The [Vary header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Vary) exists for this. MDN describes it as the header that "describes the parts of the request message (aside from the method and URL) that influenced the content of the response", and it must be sent on **every** response for the URL — the HTML one included — so that the cache keys on `Accept` from the first request onwards. Cloudflare's documentation states that its implementation adds `Accept` to `Vary`, preserving any dimensions the origin already declared, "so that caches store separate variants for Markdown and HTML".

```http
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Link: <https://example.com/guides/serving-markdown-to-llms>; rel="canonical"
```

> **Warning: Parse q-values, not substrings**
>
> Checking `accept.includes("text/markdown")` is not enough. A client may send `text/html, text/markdown;q=0.5`, in which case it wants HTML. Rank the listed types by their `q` value (default 1), then by position, and serve Markdown only when it outranks `text/html`. A `*/*` wildcard should never trigger Markdown.

Agents that do not send the header still need a way to discover the alternative. Vercel's post recommends a `<link rel="alternate" type="text/markdown">` element in the HTML head; pointing it at the page's own `.md` URL is the most precise choice, and the page's `/llms.txt` entry covers the rest.

## Converting CMS richtext to Markdown

Where the Markdown comes from matters as much as how it is served. Converting the rendered HTML back to Markdown works, and it is what an edge converter has to do, but it drags navigation, banners and footers along unless you strip them carefully. If your content lives in a headless CMS you have a better source: the structured fields themselves.

Storyblok, for example, stores a richtext field as ProseMirror-style JSON — a tree of `heading`, `paragraph`, `bullet_list`, `code_block` and `table` nodes with `bold`, `italic`, `code` and `link` marks, plus embedded component bloks. Walking that tree and emitting Markdown is a short, deterministic function. Embedded components need a decision: a callout becomes a blockquote with a bold label, a checklist becomes a task list, a statistic becomes a bold figure with its source link. Render them as readable prose, not as your own directive syntax, because the audience is a model that has never seen your component names.

Wrap the body in YAML front matter carrying the fields a model cannot infer from the text:

```yaml
---
title: "Serving Markdown to LLMs: content negotiation and .md endpoints"
canonical: https://example.com/guides/serving-markdown-to-llms
author: Endrit Krasniqi
author_url: https://example.com/authors/endrit-krasniqi
published: 2026-03-30
updated: 2026-03-30
license: CC BY 4.0
description: How to give language models a plain-text version of every page.
---
```

The canonical URL is what a model will cite. The author and dates are the trust signals a retrieval system checks. The licence tells an operator what they may do with the text; choose one deliberately rather than leaving it implied. Put the short answer immediately after the front matter, then the body, then the FAQ and the sources as a plain list of links, so the whole document reads top-down like the HTML page.

## A Next.js proxy and route handler

In Next.js 16 the `middleware` file convention was renamed to `proxy`; the file lives at `proxy.ts` next to `app` and exports a `proxy` function. The following example handles both patterns: a `.md` suffix is rewritten to a Markdown route handler, and an `Accept` header that prefers Markdown is answered with a `303` redirect to the `.md` URL. A redirect, rather than a rewrite, keeps one URL per representation, so a shared cache can never store the Markdown body under the HTML key — the framework overwrites `Vary` on pre-rendered HTML, which makes the rewrite variant unsafe behind a CDN.

```ts
// src/proxy.ts
import { NextResponse, type NextRequest } from "next/server";

const CONTENT_PATH = /^\/guides\/([a-z0-9-]+)(\.md)?$/;

function prefersMarkdown(accept: string | null): boolean {
  if (!accept) return false;
  const ranked = accept
    .split(",")
    .map((part, index) => {
      const [type, ...params] = part.trim().split(";");
      const q = params.map((p) => p.trim()).find((p) => p.startsWith("q="));
      return { type: type.trim().toLowerCase(), q: q ? Number(q.slice(2)) : 1, index };
    })
    .filter((m) => m.q > 0)
    .sort((a, b) => b.q - a.q || a.index - b.index);
  const md = ranked.find((m) => m.type === "text/markdown");
  if (!md) return false;
  const html = ranked.find((m) => m.type === "text/html");
  return !html || md.q > html.q || (md.q === html.q && md.index < html.index);
}

export function proxy(request: NextRequest) {
  const match = CONTENT_PATH.exec(request.nextUrl.pathname);
  if (!match) return NextResponse.next();
  const [, slug, ext] = match;
  if (ext === ".md") {
    // /guides/<slug>.md → the Markdown route handler, URL unchanged for the client
    const url = request.nextUrl.clone();
    url.pathname = `/api/markdown/${slug}`;
    return NextResponse.rewrite(url);
  }
  if (prefersMarkdown(request.headers.get("accept"))) {
    // Accept: text/markdown on the HTML URL → send the client to the .md URL
    const url = request.nextUrl.clone();
    url.pathname = `/guides/${slug}.md`;
    const res = NextResponse.redirect(url, 303);
    res.headers.set("Vary", "Accept");
    return res;
  }
  return NextResponse.next();
}

export const config = { matcher: ["/guides/:path*"] };
```

The route handler fetches the story from the CMS, converts it and sets the headers discussed above:

```ts
// src/app/api/markdown/[slug]/route.ts
import { getGuide } from "@/lib/cms";
import { guideToMarkdown } from "@/lib/markdown-export";

export async function GET(_req: Request, ctx: { params: Promise<{ slug: string }> }) {
  const { slug } = await ctx.params;
  const guide = await getGuide(slug);
  if (!guide) return new Response("Not found", { status: 404 });

  return new Response(guideToMarkdown(guide), {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "Content-Disposition": "inline",
      Link: `<https://example.com/guides/${slug}>; rel="canonical"`,
      Vary: "Accept",
    },
  });
}
```

Rewriting rather than redirecting is deliberate: the client keeps the canonical URL in its address bar and its citation, and only the representation changes. This site runs exactly this arrangement; `curl -H "Accept: text/markdown"` against any guide URL returns the Markdown version.

## Buttons for people: Copy as Markdown and Open in ChatGPT

Once the endpoint exists, expose it to readers. Documentation platforms have converged on a small menu next to the page title: **Copy page as Markdown**, **View as Markdown**, and **Open in ChatGPT** or **Open in Claude**. Mintlify's [contextual menu](https://mintlify.com/docs/ai/contextual-menu), for instance, offers copy, view, and options that create "a ChatGPT conversation with the current page as context" or "a Claude conversation with the current page as context".

The copy button fetches the `.md` URL and writes the text to the clipboard. The open-in buttons build a URL that opens a new conversation with a prompt asking the assistant to read the page. ChatGPT accepts a `q` query parameter on `https://chatgpt.com/` for this, although OpenAI does not formally document it. Anthropic [documents](https://support.claude.com/en/articles/14729294-open-claude-desktop-with-a-link) a `q` parameter for Claude Desktop deep links of the form `claude://claude.ai/new?q=`, with the prompt URL-encoded and truncated at roughly 14,000 characters; that document covers the desktop app, not the claude.ai website, so test any web link before shipping it. Keep the prompt short and put the Markdown URL in it, not the HTML one.

```ts
const md = `https://example.com/guides/${slug}.md`;
const prompt = encodeURIComponent(`Read ${md} and answer questions about it.`);
const openInChatGPT = `https://chatgpt.com/?q=${prompt}`;
```

**Markdown delivery checklist**

- [ ] A .md URL for every content page, returning Content-Type: text/markdown; charset=utf-8
- [ ] Accept: text/markdown on the canonical URL returns the same bytes, with q-values parsed
- [ ] Vary: Accept on every response for negotiated URLs, including the HTML one
- [ ] Link: \<canonical HTML URL>; rel="canonical" on every Markdown response
- [ ] Content-Disposition: inline so browsers display rather than download
- [ ] Markdown generated from CMS fields, with YAML front matter for canonical, author, dates and licence
- [ ] link rel="alternate" type="text/markdown" in the HTML head, and an entry in /llms.txt
- [ ] .md URLs excluded from the XML sitemap; robots.txt allows the crawlers you want
- [ ] Copy-as-Markdown and Open-in buttons wired to the same endpoint

> **Tip: Where to go next**
>
> Markdown delivery is one of three technical pieces. Read [llms.txt: what it is and how to write one](/guides/llms-txt-what-it-is-and-how-to-write-one) to build the index that points at your `.md` URLs, and [Controlling AI crawlers with robots.txt](/guides/controlling-ai-crawlers-with-robots-txt) to make sure the clients you are serving are allowed to fetch at all.

## Frequently asked questions

### Does serving Markdown create duplicate content problems for SEO?

Not if you mark the relationship. Send a Link header with rel="canonical" pointing at the HTML URL on every Markdown response; Google documents this HTTP header form for non-HTML documents. Keep the Markdown out of your XML sitemap, or list it in a separate Markdown sitemap for agents. The HTML page remains the only URL you want indexed and cited.

### Which is better: a .md suffix or Accept-header content negotiation?

Ship both. The .md suffix is discoverable from llms.txt, easy to test in a browser and cannot be confused by caches. Content negotiation keeps one canonical URL and works for agents that send the header without knowing your URL scheme. Both routes should return byte-identical Markdown from the same generator.

### Do AI crawlers actually send Accept: text/markdown?

Some agents and tools do, and the number is growing now that Cloudflare converts HTML to Markdown for any request carrying that header. Most training crawlers still request HTML by default. That is why the .md suffix, an llms.txt index and a link rel="alternate" tag matter: they give a client three ways to find the Markdown without guessing.

### Can I convert the rendered HTML to Markdown instead of generating it from the CMS?

You can, and edge services do exactly that, but the result carries navigation, cookie banners and footer noise unless you strip them. Generating from CMS fields produces cleaner output, lets you add front matter with author, dates and licence, and guarantees the Markdown never disagrees with the visible text because both come from the same source.

### Does Markdown delivery help a page get cited?

It removes obstacles rather than adding a ranking signal. A model that receives clean Markdown reads the full article within its fetch budget, sees the heading outline and can quote a passage exactly. A model that receives a heavy JavaScript-rendered page may see only a shell. Extraction quality goes up; the decision to cite still depends on the content.

## Sources

1. [Making agent-friendly pages with content negotiation](https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation) — Vercel (2026)
2. [Introducing Markdown for Agents](https://developers.cloudflare.com/changelog/post/2026-02-12-markdown-for-agents/) — Cloudflare (2026)
3. [RFC 7763: The text/markdown Media Type](https://www.rfc-editor.org/rfc/rfc7763) — IETF (2016)
4. [The /llms.txt file](https://llmstxt.org/) — Jeremy Howard, Answer.AI (2024)
5. [Understand JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) — Google Search Central (2025)
6. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — OpenAI (2025)
7. [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) — Cloudflare Docs (2026)
8. [Open Claude Desktop with a link](https://support.claude.com/en/articles/14729294-open-claude-desktop-with-a-link) — Anthropic Help Center (2026)

## Related guides

- [llms.txt: what it is, how to write one, and whether it helps](https://citable.wiki/guides/llms-txt-what-it-is-and-how-to-write-one)
- [Controlling AI crawlers with robots.txt: GPTBot, ClaudeBot, PerplexityBot and friends](https://citable.wiki/guides/controlling-ai-crawlers-with-robots-txt)
- [Modelling content for answer engines in a headless CMS!](https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms)

---

Source: https://citable.wiki/guides/serving-markdown-to-llms
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt