Serving Markdown to LLMs: content negotiation and .md endpoints

Why plain text beats the rendered DOM

A language model does not see your page the way a browser does. Whatever fetches it — a training crawler, a search indexer, or an agent acting for a user — receives bytes, and every byte it keeps costs a token. A modern HTML page is mostly bytes the model has no use for: framework bootstrap scripts, hydration payloads, inline SVG icons, class attributes, cookie-consent markup and navigation repeated on every URL. The article you wrote is a small fraction of the response.

Vercel measured this on the blog post in which it announced Markdown delivery in February 2026: the HTML version of that page was around 500KB and the Markdown version 3KB. Cloudflare's Markdown for Agents documentation describes an x-markdown-tokens header on converted responses, alongside an x-original-tokens header, precisely so operators can see the difference.

Token cost is only the first problem. The second is fetch budgets: a crawler has a limited number of bytes and requests per host, and an agent answering a live question has seconds, not minutes. The third is JavaScript rendering. Google documents a three-phase pipeline for JavaScript sites — crawl, render, index — in which a page "may stay on this queue for a few seconds, but it can take longer than that" before headless Chromium executes it. OpenAI's crawler documentation lists GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot and says nothing about executing JavaScript at all; you should assume a client-rendered page is an empty shell to them.

Markdown solves all three. It is small, it needs no rendering, and its structure — headings, lists, code fences, links — is the structure a model was trained on. It is also a registered media type: RFC 7763 defines text/markdown with a required charset parameter and an optional variant, which is what makes the second delivery pattern below possible.

Pattern one: a .md suffix on the canonical URL

The simplest pattern is a second URL. If the HTML lives at /guides/serving-markdown-to-llms, the Markdown lives at /guides/serving-markdown-to-llms.md. The llms.txt proposal at llmstxt.org recommends exactly this: pages "provide a clean markdown version of those pages at the same URL as the original page, either with .md appended (page.html.md) or with the extension replaced by .md (page.md)". Mintlify's hosted documentation follows the convention and notes that page links in its generated llms.txt "include a .md extension so AI tools can fetch the Markdown version of each page directly".

The advantages are practical. You can open the URL in a browser to check it. Caches never confuse it with the HTML. Any client that has read your /llms.txt knows how to reach it. The response should look like this:

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Content-Disposition: inline
Link: <https://example.com/guides/serving-markdown-to-llms>; rel="canonical"
Cache-Control: public, max-age=0, s-maxage=600, stale-while-revalidate=86400

Two details matter. Content-Disposition: inline stops browsers downloading the file, so people can read it too. The Link header with rel="canonical" tells search engines that the HTML page is the one to index; Google documents this HTTP-header form of canonicalisation for non-HTML documents such as PDFs, and Markdown is no different. Leave .md URLs out of your XML sitemap.

Pattern two: content negotiation on Accept

The second pattern serves Markdown from the canonical URL itself when the client asks for it. In server-driven content negotiation the client's Accept header lists the media types it can process, each with an optional quality value, and the server picks the best representation. A client that prefers Markdown sends:

GET /guides/serving-markdown-to-llms HTTP/1.1
Host: example.com
Accept: text/markdown, text/html;q=0.8

This is the pattern Vercel described in February 2026 for its own documentation, blog and changelog, and it is what Cloudflare's Markdown for Agents implements at the edge for sites that enable it: when a request carries Accept: text/markdown, Cloudflare converts the origin's HTML "on the fly" and returns content-type: text/markdown with a token count header. You can try both with curl -H "Accept: text/markdown" against their sites.

Content negotiation has one well-known trap: caches. A CDN that stores the Markdown response for the canonical URL and then serves it to a browser has broken your site. The Vary header exists for this. MDN describes it as the header that "describes the parts of the request message (aside from the method and URL) that influenced the content of the response", and it must be sent on every response for the URL — the HTML one included — so that the cache keys on Accept from the first request onwards. Cloudflare's documentation states that its implementation adds Accept to Vary, preserving any dimensions the origin already declared, "so that caches store separate variants for Markdown and HTML".

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Link: <https://example.com/guides/serving-markdown-to-llms>; rel="canonical"

Agents that do not send the header still need a way to discover the alternative. Vercel's post recommends a <link rel="alternate" type="text/markdown"> element in the HTML head; pointing it at the page's own .md URL is the most precise choice, and the page's /llms.txt entry covers the rest.

Converting CMS richtext to Markdown

Where the Markdown comes from matters as much as how it is served. Converting the rendered HTML back to Markdown works, and it is what an edge converter has to do, but it drags navigation, banners and footers along unless you strip them carefully. If your content lives in a headless CMS you have a better source: the structured fields themselves.

Storyblok, for example, stores a richtext field as ProseMirror-style JSON — a tree of heading, paragraph, bullet_list, code_block and table nodes with bold, italic, code and link marks, plus embedded component bloks. Walking that tree and emitting Markdown is a short, deterministic function. Embedded components need a decision: a callout becomes a blockquote with a bold label, a checklist becomes a task list, a statistic becomes a bold figure with its source link. Render them as readable prose, not as your own directive syntax, because the audience is a model that has never seen your component names.

Wrap the body in YAML front matter carrying the fields a model cannot infer from the text:

---
title: "Serving Markdown to LLMs: content negotiation and .md endpoints"
canonical: https://example.com/guides/serving-markdown-to-llms
author: Endrit Krasniqi
author_url: https://example.com/authors/endrit-krasniqi
published: 2026-03-30
updated: 2026-03-30
license: CC BY 4.0
description: How to give language models a plain-text version of every page.
---

The canonical URL is what a model will cite. The author and dates are the trust signals a retrieval system checks. The licence tells an operator what they may do with the text; choose one deliberately rather than leaving it implied. Put the short answer immediately after the front matter, then the body, then the FAQ and the sources as a plain list of links, so the whole document reads top-down like the HTML page.

A Next.js proxy and route handler

In Next.js 16 the middleware file convention was renamed to proxy; the file lives at proxy.ts next to app and exports a proxy function. The following example handles both patterns: a .md suffix is rewritten to a Markdown route handler, and an Accept header that prefers Markdown is answered with a 303 redirect to the .md URL. A redirect, rather than a rewrite, keeps one URL per representation, so a shared cache can never store the Markdown body under the HTML key — the framework overwrites Vary on pre-rendered HTML, which makes the rewrite variant unsafe behind a CDN.

// src/proxy.ts
import { NextResponse, type NextRequest } from "next/server";

const CONTENT_PATH = /^\/guides\/([a-z0-9-]+)(\.md)?$/;

function prefersMarkdown(accept: string | null): boolean {
  if (!accept) return false;
  const ranked = accept
    .split(",")
    .map((part, index) => {
      const [type, ...params] = part.trim().split(";");
      const q = params.map((p) => p.trim()).find((p) => p.startsWith("q="));
      return { type: type.trim().toLowerCase(), q: q ? Number(q.slice(2)) : 1, index };
    })
    .filter((m) => m.q > 0)
    .sort((a, b) => b.q - a.q || a.index - b.index);
  const md = ranked.find((m) => m.type === "text/markdown");
  if (!md) return false;
  const html = ranked.find((m) => m.type === "text/html");
  return !html || md.q > html.q || (md.q === html.q && md.index < html.index);
}

export function proxy(request: NextRequest) {
  const match = CONTENT_PATH.exec(request.nextUrl.pathname);
  if (!match) return NextResponse.next();
  const [, slug, ext] = match;
  if (ext === ".md") {
    // /guides/<slug>.md → the Markdown route handler, URL unchanged for the client
    const url = request.nextUrl.clone();
    url.pathname = `/api/markdown/${slug}`;
    return NextResponse.rewrite(url);
  }
  if (prefersMarkdown(request.headers.get("accept"))) {
    // Accept: text/markdown on the HTML URL → send the client to the .md URL
    const url = request.nextUrl.clone();
    url.pathname = `/guides/${slug}.md`;
    const res = NextResponse.redirect(url, 303);
    res.headers.set("Vary", "Accept");
    return res;
  }
  return NextResponse.next();
}

export const config = { matcher: ["/guides/:path*"] };

The route handler fetches the story from the CMS, converts it and sets the headers discussed above:

// src/app/api/markdown/[slug]/route.ts
import { getGuide } from "@/lib/cms";
import { guideToMarkdown } from "@/lib/markdown-export";

export async function GET(_req: Request, ctx: { params: Promise<{ slug: string }> }) {
  const { slug } = await ctx.params;
  const guide = await getGuide(slug);
  if (!guide) return new Response("Not found", { status: 404 });

  return new Response(guideToMarkdown(guide), {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "Content-Disposition": "inline",
      Link: `<https://example.com/guides/${slug}>; rel="canonical"`,
      Vary: "Accept",
    },
  });
}

Rewriting rather than redirecting is deliberate: the client keeps the canonical URL in its address bar and its citation, and only the representation changes. This site runs exactly this arrangement; curl -H "Accept: text/markdown" against any guide URL returns the Markdown version.

Buttons for people: Copy as Markdown and Open in ChatGPT

Once the endpoint exists, expose it to readers. Documentation platforms have converged on a small menu next to the page title: Copy page as Markdown, View as Markdown, and Open in ChatGPT or Open in Claude. Mintlify's contextual menu, for instance, offers copy, view, and options that create "a ChatGPT conversation with the current page as context" or "a Claude conversation with the current page as context".

The copy button fetches the .md URL and writes the text to the clipboard. The open-in buttons build a URL that opens a new conversation with a prompt asking the assistant to read the page. ChatGPT accepts a q query parameter on https://chatgpt.com/ for this, although OpenAI does not formally document it. Anthropic documents a q parameter for Claude Desktop deep links of the form claude://claude.ai/new?q=, with the prompt URL-encoded and truncated at roughly 14,000 characters; that document covers the desktop app, not the claude.ai website, so test any web link before shipping it. Keep the prompt short and put the Markdown URL in it, not the HTML one.

const md = `https://example.com/guides/${slug}.md`;
const prompt = encodeURIComponent(`Read ${md} and answer questions about it.`);
const openInChatGPT = `https://chatgpt.com/?q=${prompt}`;