Skip to content
Citable

Definition

Content negotiation

Also known as: Proactive negotiation, server-driven negotiation

Content negotiation is the HTTP mechanism by which a client states which representations it prefers — using headers such as Accept, Accept-Language and Accept-Encoding — and the server picks the best available one for the same URL. For answer engines it lets one canonical URL return HTML to browsers and Markdown to a crawler that sends Accept: text/markdown.

HTTP defines content negotiation in RFC 9110, section 12. The common form is proactive (server-driven): the client sends Accept headers, optionally weighted with q= values; the server chooses a representation and names the headers it considered in Vary. If nothing acceptable exists it may answer 406 Not Acceptable or fall back to a default. Compression (Accept-Encoding) and language (Accept-Language) have been negotiated this way for decades, so every cache and proxy already understands the mechanism.

The text/markdown media type has been registered since 2016 (RFC 7763), which makes Markdown a legitimate representation to ask for.

An answer engine wants the text, not the chrome. Serving Markdown for Accept: text/markdown gives a crawler a compact representation of the same canonical URL with no navigation, scripts or consent banners to strip. This site does exactly that for every guide and glossary page, and also exposes the same Markdown at a .md suffix:

bash
curl -H "Accept: text/markdown" https://your-site.example/glossary/content-negotiation

The one operational hazard is caching. Set Vary: Accept on both the HTML and the Markdown response, or a shared cache may hand Markdown to a browser. Serving Markdown to LLMs covers the implementation details.

Frequently asked questions

Should I use content negotiation or a .md URL to serve Markdown?

Both, if you can. A .md suffix is visible, linkable and easy to test in a browser; content negotiation keeps a single canonical URL. Whichever route serves Markdown, send Vary: Accept so caches keep the two representations apart, and point back to the HTML page with a canonical link header.

Related terms

Guides that use this term

  • Technical

    Serving Markdown to LLMs: content negotiation and .md endpoints

    Serving Markdown to LLMs means publishing a plain-text representation of each page next to the HTML one, so that AI crawlers and agents can read it without rendering JavaScript or spending tokens on markup. There are two delivery patterns: a .md suffix on the canonical URL, and content negotiation on the Accept: text/markdown request header, with Vary: Accept and a Link rel=canonical header pointing back to the HTML page.

    10 min read

  • Technical

    llms.txt: what it is, how to write one, and whether it helps

    llms.txt is a proposed convention, published by Jeremy Howard of Answer.AI in September 2024, for a Markdown file at /llms.txt that gives language models a curated index of a site: an H1 title, a blockquote summary and H2 sections of links with one-line descriptions. No major AI provider has confirmed reading it, but it costs almost nothing to generate from a CMS and is widely published by documentation sites.

    8 min read