# Citable — full content> Citable is an open field guide to Answer Engine Optimization (AEO), Generative Engine Optimization (GEO) and LLM-readable content. Every guide is written answer-first, marked up with schema.org JSON-LD, and served as clean HTML and Markdown so that search engines, answer engines and language models can quote it accurately.This file contains the complete text of every guide and glossary term on https://citable.wiki/. Each section starts with front matter that includes the canonical URL to cite.

<!-- ---------------------------------------------------------------- -->

---
title: "Modelling content for answer engines in a headless CMS!"
subtitle: "How to turn every AEO signal into a constrained CMS field, and generate JSON-LD, Markdown and llms.txt from those fields, with Storyblok as the worked example."
canonical: https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms
category: technical
author: "Endrit Krasniqi"
date_published: 2026-05-25
date_modified: 2026-05-25
license: CC BY 4.0
---

# Modelling content for answer engines in a headless CMS!

*How to turn every AEO signal into a constrained CMS field, and generate JSON-LD, Markdown and llms.txt from those fields, with Storyblok as the worked example.*

## Short answer

A headless CMS content model for answer engines makes every Answer Engine Optimization signal a constrained field, not an editorial habit: a required short answer with a maximum length, key takeaways, FAQ items, sources with URL validation, the author as a relation, and published and updated dates. JSON-LD, Markdown and llms.txt are then generated from those fields, so no representation can disagree with the page.

## Key takeaways

- Every AEO signal should be a constrained CMS field, not an editorial habit, so it is present on every page by construction.
- Model the short answer, takeaways, FAQ and sources as required fields and nestable bloks with length and URL constraints.
- Author and related terms belong in relation fields, which give JSON-LD stable Person and DefinedTerm identifiers.
- Generate JSON-LD, Markdown and llms.txt from the same fields so no representation can disagree with the visible page.
- In Storyblok, seed the schema with the Management API, preview with Draft Mode and revalidate from publish webhooks.

## Why the content model is the AEO strategy

Most Answer Engine Optimization advice is a list of habits: open with a direct answer, add an FAQ, cite sources, show the author and the date, mark it up. Habits decay. The third writer forgets the short answer, the FAQ becomes body prose, and the JSON-LD template emits a `dateModified` nobody updates.

A headless CMS lets you move each habit into the **content model**. A required field with a maximum length is present on every page by construction, and a front end that derives its structured data, Markdown and `llms.txt` from those fields can never publish a representation that disagrees with the visible text.

The method in one sentence: **one set of fields, many generated representations**. What follows is the field list, the constraints, and the Storyblok plumbing Citable uses.

> **Note: Templates cannot enforce anything**
>
> A template can render a short answer if one exists; it cannot make an editor write one. Constraints have to live where the content is entered. The CMS is the only layer that can refuse to save a guide without an author.

## Signals become fields

Every signal described in [What is Answer Engine Optimization](/guides/what-is-answer-engine-optimization) maps to a field, a constraint and a downstream use.

| Signal | Field | Constraint | Generated from it |
| --- | --- | --- | --- |
| Direct answer | `short_answer` (textarea) | required, 420 characters (about 40–70 words) | Speakable block, JSON-LD `description`, `llms.txt` line |
| Scannable summary | `key_takeaways` (nestable `takeaway`) | max 6, 240 characters each | `ItemList` in JSON-LD, Markdown list |
| Question coverage | `faq` (nestable `faq_item`) | question + answer, 600 characters | `FAQPage` JSON-LD, FAQ section in HTML and Markdown |
| Verifiable sources | `sources` (nestable `source`) | `url` must match `^https?://` | `citation` in JSON-LD, numbered list in Markdown |
| Who wrote it | `author` (single story reference) | required, restricted to `author` stories | `Person` JSON-LD with `sameAs`, `rel="author"` link |
| When | `published_at`, `updated_at` (date) | date only | `datePublished`, `dateModified`, feed and sitemap dates |
| Entities | `related_terms` (multi story reference) | restricted to `glossary_term` | `about` as a `DefinedTerm` list, `keywords`, a "terms used" section |

### The short answer is a separate field, not the first paragraph

A first paragraph is subject to every editorial instinct that produces a warm-up sentence. A required textarea with a character limit gets written as an answer. Citable renders it in a `data-speakable="short-answer"` block and points the [Speakable](/glossary/speakable) selector at it.

### Author and related terms are relations, not strings

A free-text author field cannot produce a stable `Person` identifier. A relation to an `author` story gives every guide the same `@id`, bio and `sameAs` links. Related glossary terms work the same way: the term's `DefinedTerm` `@id` is reused in the guide's `about` property, so a crawler can connect the definition to every guide that relies on it.

### Sources are structured, not a paragraph of links

A `source` blok has `title`, `url`, `publisher` and `published`. The URL pattern check stops a bare domain from being saved as a citation, and the four fields are exactly what a `CreativeWork` citation needs.

## Citable's Storyblok schema

Storyblok distinguishes **content types** (root components that become stories) from **nestable** components (bloks that live inside a field of another component). Citable has five content types and a set of small nestables, defined once in TypeScript and pushed to the space with the Management API, so the CMS and the front-end types cannot drift apart.

### Content types

- **guide** — `title` (max 90), `subtitle` (max 200), `short_answer` (required, max 420), `category` and `difficulty` (single-option), `reading_time` (number, computed when empty), `author` (required single story reference), `cover` (asset), `key_takeaways` (bloks: `takeaway`, max 6), `body` (richtext, required), `faq` (bloks: `faq_item`), `sources` (bloks: `source`), `related_guides` and `related_terms` (multi story references), `seo_title` (max 70), `seo_description` (max 160), `published_at` and `updated_at` (date).
- **glossary\_term** — `term`, `definition` (required, max 320, emitted as `DefinedTerm`), `also_known_as`, `expanded` (richtext), `related_terms`, `related_guides`, `faq`, `seo_description`.
- **author** — `name`, `role`, `bio` (required, becomes the `Person` description), `avatar`, `website`, `links` (bloks: `social_link`, become `sameAs`).
- **page** — `title`, `seo_description`, `body` (bloks: `hero`, `feature_grid`, `guide_collection`, `glossary_preview`, `faq_section`, `principles`, `cta`, `prose_section`).
- **site\_config** — `site_name`, `tagline`, `description`, `nav`, `footer_links`, `ai_policy` (published verbatim in `llms.txt`).

### How a relation field is declared

Story references in Storyblok are `option` (single) or `options` (multiple) fields with `source: "internal_stories"`. Restricting them by content type and folder turns "a reference" into "an author".

```ts
const storyRef = (contentTypes: string[], folder: string, extra = {}) => ({
  type: "option",
  source: "internal_stories",
  use_uuid: true,
  filter_content_type: contentTypes,
  folder_slug: folder,
  ...extra,
});

// in the guide schema
author: storyRef(["author"], "authors/", { required: true }),
short_answer: {
  type: "textarea",
  required: true,
  max_length: 420,
  description: "The direct answer, 40–70 words. This is what answer engines quote.",
},
```

### Rich text with embedded bloks

The guide body is a richtext field. Storyblok stores rich text as JSON nodes (paragraphs, headings, lists, code blocks) rather than HTML, and nestable components can be inserted between them. The `@storyblok/richtext` documentation puts it plainly: components are embedded as `blok` nodes, each carrying an `attrs.body` array that the renderer resolves itself.

Citable uses that for the parts of a guide that need semantics a paragraph cannot carry: `callout` (variant, title, richtext body), `checklist` (a list of `check_item`), `stat` (value, label, source) and `code_block`. Because a callout is a component with fields, the Markdown exporter can render it as a blockquote, the plain-text extractor can count its words, and the JSON-LD builder can ignore it. None of them has to parse HTML.

## Generating every representation from the same fields

Each machine-readable output is then a pure function of a story.

### JSON-LD

Google's Article structured data documentation recommends `author`, `author.name`, `author.url`, `datePublished`, `dateModified`, `headline` and `image`. All of them are fields or derived from fields. The guide builder returns a `TechArticle` (a subtype of `Article` on [schema.org](https://schema.org/TechArticle)) together with `Person`, `FAQPage` and `BreadcrumbList` objects:

```ts
const article = {
  "@context": "https://schema.org",
  "@type": "TechArticle",
  headline: c.title,
  description: c.short_answer,
  datePublished: pubDate(guide),
  dateModified: modDate(guide),
  author: { "@id": `${authorUrl}#person` },
  speakable: { "@type": "SpeakableSpecification", cssSelector: ["[data-speakable='short-answer']"] },
  citation: c.sources.map((s) => ({ "@type": "CreativeWork", name: s.title, url: s.url })),
  about: terms.map((t) => ({ "@type": "DefinedTerm", "@id": `${termUrl(t)}#term`, name: t.content.term })),
};
```

### Markdown and content negotiation

The same story renders to Markdown: front matter with the canonical URL, author and both dates, then the short answer, takeaways, body, FAQ and numbered sources. A request for `/guides/<slug>.md`, or for `/guides/<slug>` with `Accept: text/markdown`, is rewritten to that renderer, and the response carries a `Link: <…>; rel="canonical"` header pointing back at the HTML page.

### llms.txt

The index follows the [llmstxt.org](https://llmstxt.org/) layout: an H1, a blockquote summary, then sections of `- [Title](url): description` links. Every description is a guide's `short_answer` or a term's `definition`, so the file is a faithful index of the site without anyone maintaining it:

```markdown
# Citable

> Citable is an open field guide to Answer Engine Optimization (AEO) …

## Guides: Technical

- [Modelling content for answer engines in a headless CMS](https://…/guides/modeling-content-for-answer-engines-in-a-headless-cms.md): A headless CMS content model …
```

The file's "Policy for AI systems" section is the `ai_policy` field of `site_config`.

## Wiring Storyblok to the site

### Seeding the space with the Management API

Citable's seed script pushes the component definitions and local content into a space. Components are created or updated by name and stories by `full_slug`, so it is idempotent. Relations need two passes: Storyblok assigns story UUIDs on creation, so the script first creates every story with relation fields emptied, then writes the relations with the assigned UUIDs. Storyblok's Management API documentation states a limit of 3 requests per second on the Starter plan and 6 on higher plans, so the script throttles and retries on `429`.

**50** — Maximum number of related stories the Content Delivery API resolves in one request with resolve\_relations. Keep relation lists short and avoid deep chains. (Source: [Storyblok: Retrieve multiple stories](https://www.storyblok.com/docs/api/content-delivery/v2/stories/retrieve-multiple-stories))

### Reading content, published or draft

The front end reads the Content Delivery API with `version=published` and a `resolve_relations` list of `component.field` pairs (`guide.author`, `guide.related_terms` and so on), then merges the returned `rels` array back into each story. Responses are cached with the tag `storyblok`.

Preview uses Next.js Draft Mode. The Storyblok visual editor loads the site in an iframe from a preview URL set under Settings → Visual Editor; Citable points it at `/api/draft?secret=…&slug=`, and Storyblok appends its `_storyblok` parameters. The handler enables Draft Mode and redirects to the story:

```ts
import { draftMode } from "next/headers";
import { redirect } from "next/navigation";

export async function GET(request: Request) {
  const url = new URL(request.url);
  if (url.searchParams.get("secret") !== process.env.DRAFT_MODE_SECRET) {
    return new Response("Invalid token", { status: 401 });
  }
  (await draftMode()).enable();
  redirect(`/${url.searchParams.get("slug") ?? ""}`);
}
```

While Draft Mode is on, the content layer switches to `version=draft` and `cache: "no-store"`, so editors see unpublished changes immediately.

### Webhooks for revalidation

Storyblok's webhook documentation lists `published`, `unpublished`, `deleted` and `moved` story events, each posting a JSON body with `action`, `full_slug`, `story_id` and `space_id`. Citable registers one endpoint for publish and unpublish and invalidates everything tagged `storyblok`:

```ts
import { revalidatePath, revalidateTag } from "next/cache";

export async function POST(request: Request) {
  const url = new URL(request.url);
  if (url.searchParams.get("secret") !== process.env.REVALIDATE_SECRET) {
    return Response.json({ ok: false }, { status: 401 });
  }
  const payload = await request.json().catch(() => ({}));
  revalidateTag("storyblok", "max");
  revalidatePath("/", "layout");
  return Response.json({ ok: true, story: payload.full_slug ?? null });
}
```

Because `llms.txt`, `llms-full.txt`, the sitemap, the feed and every Markdown route read through the same tagged fetches, one publish event refreshes all of them. Storyblok's webhook documentation also describes a `webhook-signature` header that your endpoint can verify against a webhook secret; secrets are only available on paid plans, and that check is stronger than a query-string secret.

## Validating content before it ships

Field constraints catch the crude mistakes; a content check catches the ones that need counting. Citable runs a script over every story before it is seeded.

**Rules the content check enforces**

- [ ] short\_answer is 40–70 words
- [ ] at least three FAQ items, each question ending with a question mark
- [ ] at least two sources, every URL https
- [ ] author, related\_guides and related\_terms resolve to real stories
- [ ] body starts with an H2, has at least three H2 sections and no H1
- [ ] body is at least 700 words and every internal link resolves
- [ ] seo\_title at most 70 characters, seo\_description at most 160

The fixtures the check validates are the ones the seed script pushes, so a guide that passes locally renders correctly in the space. Because the model is data, it can be tested like data.

> **Tip: Where to go next**
>
> [Structured data for AI answers](/guides/structured-data-for-ai-answers) covers the JSON-LD types those fields feed in detail, and [llms.txt: what it is and how to write one](/guides/llms-txt-what-it-is-and-how-to-write-one) shows what the generated index should contain.

## Frequently asked questions

### Why should the short answer be its own field rather than the first paragraph?

Because a first paragraph is subject to every editorial instinct that produces a warm-up sentence, and nothing enforces its length. A required textarea with a character limit is written as an answer, can be rendered in a dedicated Speakable block, and can be reused unchanged as the JSON-LD description and the llms.txt entry for the page.

### Should the author be a text field or a relation to an author entry?

A relation. A free-text name cannot produce a stable schema.org Person identifier, a bio, or sameAs links, and it drifts in spelling across pages. A relation to an author content type gives every guide the same Person @id and lets the author page list its guides without a search.

### Can this approach be used with a CMS other than Storyblok?

Yes. The model needs only required fields, length limits, URL validation, references to other entries and nestable components. Contentful, Sanity, Strapi and Payload all offer those. The Storyblok-specific parts are the Management API seeding, the visual editor preview URL and the webhook payload shape, and each has an equivalent elsewhere.

### How does the site stay current after an editor publishes in Storyblok?

Storyblok sends a webhook on story publish and unpublish events. The site's endpoint verifies a secret, then invalidates every cached Content Delivery API response by tag. Because the HTML pages, Markdown routes, llms.txt, sitemap and feed all read through the same tagged fetches, one publish event refreshes every representation on its next request.

### Do embedded bloks in rich text make the content harder for machines to read?

No. Storyblok stores rich text as JSON nodes, and an embedded component is a typed blok node with named fields. The Markdown exporter renders a callout as a blockquote and a checklist as a list, and the plain-text extractor counts their words. Typed fields are easier to convert than HTML that has to be parsed and guessed at.

## Sources

1. [Management API](https://www.storyblok.com/docs/api/management) — Storyblok
2. [Retrieve multiple stories (Content Delivery API v2)](https://www.storyblok.com/docs/api/content-delivery/v2/stories/retrieve-multiple-stories) — Storyblok
3. [Webhooks](https://www.storyblok.com/docs/concepts/webhooks) — Storyblok
4. [@storyblok/richtext](https://www.storyblok.com/docs/packages/storyblok-richtext) — Storyblok
5. [Article (Article, NewsArticle, BlogPosting) structured data](https://developers.google.com/search/docs/appearance/structured-data/article) — Google Search Central (2025)
6. [draftMode](https://nextjs.org/docs/app/api-reference/functions/draft-mode) — Next.js documentation (Vercel) (2026)

## Related guides

- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)
- [llms.txt: what it is, how to write one, and whether it helps](https://citable.wiki/guides/llms-txt-what-it-is-and-how-to-write-one)
- [Serving Markdown to LLMs: content negotiation and .md endpoints](https://citable.wiki/guides/serving-markdown-to-llms)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)

---

Source: https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt


<!-- ---------------------------------------------------------------- -->

---
title: "FAQ content that answer engines pick up (and why FAQPage rich results went away)"
subtitle: "How to choose real questions, write self-contained answers, keep them in the HTML and mark them up — now that Google has retired the FAQ rich result."
canonical: https://citable.wiki/guides/faq-content-for-answer-engines
category: structured-data
author: "Endrit Krasniqi"
date_published: 2026-05-11
date_modified: 2026-05-11
license: CC BY 4.0
---

# FAQ content that answer engines pick up (and why FAQPage rich results went away)

*How to choose real questions, write self-contained answers, keep them in the HTML and mark them up — now that Google has retired the FAQ rich result.*

## Short answer

FAQ content works for answer engines when each item pairs a question people ask with a self-contained answer of 40–80 words, rendered in the HTML rather than loaded on click, and mirrored in FAQPage JSON-LD from the same fields. Google restricted FAQ rich results in 2023 and withdrew them in May 2026, but FAQPage remains valid schema.org, and question-and-answer pairs are the passages retrieval systems extract.

## Key takeaways

- Google limited FAQ rich results to government and health sites in 2023 and withdrew them entirely in May 2026; FAQPage markup remains valid schema.org.
- Source questions from People Also Ask, forum threads and assistant-style prompts, and keep the user's phrasing in the heading.
- Each answer should stand alone in 40–80 words, naming the entity in its first sentence and never pointing back to earlier text.
- Render answers in the HTML with native details and summary elements; collapsed content stays in the DOM, content fetched on click does not.
- Generate FAQPage JSON-LD from the same CMS fields as the visible questions so markup and page can never disagree.
- Use an in-article FAQ for follow-up questions and a dedicated page when the questions are the topic itself.

## What changed, and what did not

The FAQ rich result — the expandable question-and-answer block that used to appear under a Google listing — went away in two steps.

On 8 August 2023, Google's Search Central blog announced that FAQ rich results would "only be shown for well-known, authoritative government and health websites"; for every other site the rich result "will no longer be shown regularly". The same post said there was no need to remove the markup: "structured data that's not being used does not cause problems for Search, but also has no visible effects in Google Search."

On 8 May 2026, Google's documentation changelog added a deprecation notice stating that the feature "will no longer appear in Google Search starting May 7, 2026". In June 2026 the changelog recorded that the FAQ rich result documentation had been removed altogether.

That is the whole story of the rich result. Three things did not change.

- `FAQPage`** is still a valid schema.org type.** [schema.org/FAQPage](https://schema.org/FAQPage) defines it as "a WebPage presenting one or more 'Frequently asked questions'", a subtype of `WebPage`. The vocabulary is unchanged and anything that parses JSON-LD can still read it.
- **Question-and-answer pairs are still the tidiest passage a retrieval system can extract.** [Answer engines](/glossary/answer-engine) split pages into chunks and score each chunk against a query. A heading that *is* the question, followed by a paragraph that *is* the answer, is a chunk that needs no context from anywhere else on the page.
- **Visible text is still what counts.** Google's documentation for AI features says you "don't need to create new machine readable files, AI text files, or markup to appear in these features" and that "there's also no special schema.org structured data that you need to add". The requirement is that the page is indexed and eligible to be shown with a snippet.

So the job of an FAQ section moved. It used to be a search-result decoration. Now it is a set of quotable passages that also carry a machine-readable label.

> **Note: Rich result versus structured data**
>
> A **rich result** is a presentation feature of Google Search; **structured data** is the markup that describes what is on the page. Google withdrew the first. The second is defined by schema.org and read by every system that parses JSON-LD, including retrieval pipelines that never show a rich result.

## Find the questions people actually ask

An FAQ written from inside the company answers questions nobody asked. An FAQ that gets cited answers the questions people type into a search box or an assistant. There are four cheap places to find them.

1. **People Also Ask.** Search for your main query in Google and expand the People Also Ask box; each expansion reveals further questions, phrased the way searchers phrase them.
2. **Forum phrasing.** Reddit threads, Stack Overflow, GitHub issues and product forums show the question with its constraints attached: "does X still work if I…".
3. **Assistant-style prompts.** Questions put to ChatGPT, Claude or Perplexity are longer and more conversational than search queries, and often carry a situation ("I run a Next.js site and…"). Write at least one question in that register.
4. **Your own data.** Search Console queries containing "how", "why", "can", "does" and "vs"; support tickets; sales call notes.

Keep the phrasing: the heading should read the way the question is asked, not the way a marketing team would rewrite it.

| Written from the inside | Written from the question |
| --- | --- |
| Compatibility | Does FAQPage markup still work after the 2023 change? |
| Pricing and plans | Is there a free plan, and what does it leave out? |
| Our approach to security | Where is my data stored and who can access it? |

One question per item. "How do I install it and what does it cost?" is two chunks fighting over one heading.

## Write answers that stand on their own

An answer engine will lift the answer paragraph out of your page and drop it into a context where the heading, the previous paragraph and the product name may all be missing. Write for that context.

- **Name the entity in the first sentence.** "FAQPage markup is still valid schema.org" travels; "Yes, it still works" does not.
- **Answer, then qualify.** The first sentence is the answer. The second and third add the condition, the exception or the number. Nothing comes before the answer.
- **Stay within 40–80 words.** Long enough for an answer and one qualification, short enough to be quoted whole. An answer that wants 300 words is a section or a page; write the short answer here and link to the long one.
- **Put one verifiable thing in it.** A date, a version, a named source, a number with a unit. The GEO study by Aggarwal et al. found that adding citations, quotations and statistics raised a source's visibility in generated answers substantially.
- **No back-references.** No "as mentioned above", no "this" without a noun, no "the tool" when you mean the product name.

> **Example: Before and after**
>
> **Before.** Yes! As we explained above, it's still fully supported and you don't need to change anything.
> 
> **After.** FAQPage structured data is still a valid schema.org type, and Google has said there is no need to remove it. What ended was the FAQ *rich result*, restricted in 2023 and withdrawn in 2026; the markup still labels each question-and-answer pair for any parser that reads JSON-LD.

## Put the answers in the HTML with native details

The most common way FAQ sections fail is an accordion that renders the question in HTML and fetches the answer only on click. A crawler with a small rendering budget sees questions with nothing under them.

The fix is the platform's own disclosure widget. MDN describes `<details>` as a widget "in which information is visible only when the widget is toggled into an open state", and the content stays in the DOM whether the widget is open or closed. A group of `<details>` elements that share a `name` attribute behaves as an exclusive accordion — "only one of the grouped `<details>` elements can be open at a time" — with no script at all.

```html
<section aria-labelledby="faq-heading">
  <h2 id="faq-heading">Frequently asked questions</h2>

  <details name="faq" open>
    <summary>
      <h3>Should I remove FAQPage markup now that the rich result is gone?</h3>
    </summary>
    <p>No. Google's 2023 announcement said there was no need to remove it,
    and unused structured data does not cause problems for Search…</p>
  </details>

  <details name="faq">
    <summary>
      <h3>How long should an FAQ answer be?</h3>
    </summary>
    <p>Aim for 40–80 words…</p>
  </details>
</section>
```

Three details matter.

- **Open the first item by default.** Google's general structured data guidelines say "don't mark up content that is not visible to readers of the page". An open first answer leaves no room for argument.
- **Keep the heading outline intact.** Each question is an `<h3>` under the FAQ `<h2>`, so a chunker gets one heading-plus-paragraph unit per item.
- **Do not fetch on demand.** If the answer text is not in the initial HTML response, assume it is not in the index.

## Generate FAQPage JSON-LD from the same fields

The shape Google documented while the rich result existed, and which schema.org still defines, is small: a `FAQPage` whose `mainEntity` is an array of `Question`, each `Question` with a `name` and an `acceptedAnswer`, and each `Answer` with `text`. That is the whole schema. Google's general guidelines require markup to describe what the HTML body shows, so generate it from the same fields that render the visible FAQ, never from a second copy.

```ts
// lib/faq-jsonld.ts
export interface FaqItem {
  question: string;
  answer: string; // plain text, or minimal HTML
}

export function faqPageJsonLd(items: FaqItem[]) {
  return {
    "@context": "https://schema.org",
    "@type": "FAQPage",
    mainEntity: items.map((item) => ({
      "@type": "Question",
      name: item.question,
      acceptedAnswer: { "@type": "Answer", text: item.answer },
    })),
  };
}
```

In a Next.js App Router page, pass the same `items` array to the visible `<Faq>` component and to a `<script type="application/ld+json">` tag rendered from `faqPageJsonLd(items)`. One source, two outputs.

Two things not to do:

- **Do not use **`QAPage`** for an FAQ.** Google's `QAPage` documentation is explicit: "Don't use QAPage markup for FAQ pages or pages where there are multiple questions per page." `QAPage` is for a single question with user-submitted answers, such as a forum thread.
- **Do not mark up a question whose answer is on another page.** The `text` of each `Answer` must be the answer, on this page; "see our pricing page" is a link, not an answer.

## A dedicated FAQ page or an FAQ inside the article

Both work; they do different jobs.

| Use an in-article FAQ when… | Use a dedicated FAQ page when… |
| --- | --- |
| The questions qualify the main answer: edge cases, "does this apply if…" | The questions are the topic: pricing, shipping, a product's limits |
| There are three to six of them | There are dozens, and they cluster into groups |
| Each answer fits in 40–80 words | Some answers need their own heading, so the page becomes a hub |
| You want the article's canonical URL to collect the citation | You want a URL that support, sales and the footer can all link to |

Two rules cut across both.

1. **One question per URL when the question is big.** If a question needs more than a paragraph, it has become a guide. Give it a page with the question in the title and H1, and leave a short answer plus a link in the FAQ.
2. **Never duplicate the main answer in the FAQ.** If the article's opening paragraph answers "what is X", the FAQ should not ask "what is X" again. Two near-identical passages on one page compete in retrieval and dilute both.

**FAQ section pre-publish check**

- [ ] Every question is phrased the way a person asks it, and ends with a question mark
- [ ] One question per item, no compound questions
- [ ] Every answer is 40–80 words, names the entity in the first sentence and contains no back-references
- [ ] Answers are in the initial HTML, using details and summary, not fetched on click
- [ ] The first item is open by default and each question is an H3 under the FAQ H2
- [ ] FAQPage JSON-LD is generated from the same fields that render the visible FAQ
- [ ] No QAPage markup, and no question whose answer lives on a different page
- [ ] The FAQ does not repeat the article's own short answer

> **Tip: Where to go next**
>
> The FAQ is one of several answer-first blocks on a page. [How to write answer-first content](/guides/how-to-write-answer-first-content) covers the rest of the page, and [Structured data for AI answers](/guides/structured-data-for-ai-answers) shows how the `FAQPage` block fits alongside `Article`, `Person` and `DefinedTerm` markup generated from the same CMS fields.

## Frequently asked questions

### Should I remove FAQPage structured data now that the rich result is gone?

No. Google's August 2023 announcement said there was no need to proactively remove it, and that unused structured data does not cause problems for Search. Google has since retired the rich result and its documentation, but FAQPage remains a valid schema.org type that labels each question-and-answer pair for any parser that reads JSON-LD. Keep it, generated from the same fields as the visible FAQ.

### How long should an FAQ answer be for answer engines?

Aim for 40–80 words. That is long enough to state the answer, name the entity and add one qualification, and short enough to be quoted whole. If an answer needs 300 words it is not an FAQ item; give it its own heading or its own page, and link to it from a shorter answer.

### Do answer engines read content hidden inside a collapsed details element?

The content of a closed details element is still in the HTML and the DOM, so any crawler that parses HTML can read it. What crawlers cannot reliably read is content that is only fetched or rendered after a click. Use native details and summary, open the first item by default, and avoid loading answers with JavaScript.

### Should I use FAQPage or QAPage markup?

Use FAQPage when you write both the questions and the answers and there is one answer per question. Use QAPage only for a page about a single question where users can submit their own answers, such as a forum thread. Google's documentation says not to use QAPage for FAQ pages or pages with multiple questions.

### Do I need FAQ markup to appear in Google AI Overviews?

No. Google's documentation for AI features says no special schema.org structured data is required, and that a page only needs to be indexed and eligible to be shown with a snippet. The markup helps parsers label the pairs; the retrieval benefit comes from the answer text itself being visible, self-contained and quotable.

## Sources

1. [Changes to HowTo and FAQ rich results](https://developers.google.com/search/blog/2023/08/howto-faq-changes) — Google Search Central Blog (2023)
2. [Latest Google Search documentation updates](https://developers.google.com/search/updates#removing-faq-rich-result) — Google Search Central (2026)
3. [General structured data guidelines](https://developers.google.com/search/docs/appearance/structured-data/sd-policies) — Google Search Central (2026)
4. [Schema for Q&A pages (QAPage)](https://developers.google.com/search/docs/appearance/structured-data/qapage) — Google Search Central (2026)
5. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Google Search Central (2025)
6. [<details>: The Details disclosure element](https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/details) — MDN Web Docs (2026)

## Related guides

- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)
- [How to write answer-first content that LLMs can quote](https://citable.wiki/guides/how-to-write-answer-first-content)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)
- [Modelling content for answer engines in a headless CMS!](https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms)

---

Source: https://citable.wiki/guides/faq-content-for-answer-engines
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt


<!-- ---------------------------------------------------------------- -->

---
title: "E-E-A-T for answer engines: authors, dates and sources that machines can verify"
subtitle: "What Experience, Expertise, Authoritativeness and Trust look like when the reader is a retrieval system, and how to encode them so they can be checked."
canonical: https://citable.wiki/guides/eeat-for-answer-engines
category: content
author: "Endrit Krasniqi"
date_published: 2026-04-27
date_modified: 2026-04-27
license: CC BY 4.0
---

# E-E-A-T for answer engines: authors, dates and sources that machines can verify

*What Experience, Expertise, Authoritativeness and Trust look like when the reader is a retrieval system, and how to encode them so they can be checked.*

## Short answer

E-E-A-T (Experience, Expertise, Authoritativeness, Trustworthiness) is Google's description of a reliable source, and it is not a ranking factor. For answer engines it translates into signals a machine can verify: a named author with a profile page and Person markup, visible published and updated dates that match the structured data, outbound links to primary sources with a citation list, first-hand evidence, and an About page that says who is responsible.

## Key takeaways

- Google states that E-E-A-T is not a specific ranking factor; it describes what its systems try to reward, with trust as the central quality.
- A retrieval system cannot judge quality like a human rater, so it leans on what it can check: author, dates, sources and site identity.
- Name the author, give them a profile page, and connect the two with Person markup that carries url and sameAs.
- Visible dates and datePublished/dateModified must match, and dateModified must come from an editorial field, never from build time.
- Link primary sources where claims are made and emit the same list as schema.org citation; research on generative engines rewards it.
- An About page with Organization markup answers the rater's question of who is responsible, and gives every article a publisher entity.

## What E-E-A-T means when the reader is a retrieval system

E-E-A-T stands for Experience, Expertise, Authoritativeness and Trustworthiness. Google introduced the four-letter form on 15 December 2022, when its Search Central blog announced that "E-A-T is gaining an E for experience" in an update to the Search Quality Rater Guidelines. Those guidelines are the document Google gives to the human raters who evaluate its ranking systems.

Two clarifications from Google's own documentation come first. The "Creating helpful, reliable, people-first content" page states that "E-E-A-T itself isn't a specific ranking factor", and that raters "have no control over how pages rank". The same page says that of the four qualities "trust is most important" and that content "doesn't necessarily have to demonstrate all of them". E-E-A-T is a description of what a good source looks like, not a score you can move.

The description is useful for answer engines precisely because a retrieval system cannot judge quality the way a rater can. It fetches a passage, inspects a few properties of the page around it, and decides whether the passage is safe to quote. The properties it can actually inspect are the machine-readable shadow of E-E-A-T: is there a named author it can resolve to an entity, when was the page published and last changed, what does the page cite, and who is responsible for the site.

Google's [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) documentation says there are "no additional requirements to appear in AI Overviews or AI Mode" and that you "don't need to create new machine readable files, AI text files, or markup" for them. The existing fundamentals apply, including the helpful-content guidance where E-E-A-T is described. The rest of this guide is about making those fundamentals legible to a machine.

> **Definition: E-E-A-T**
>
> Experience, Expertise, Authoritativeness and Trustworthiness: the four qualities Google's Search Quality Rater Guidelines ask human raters to assess when deciding whether a page and its author are a reliable source on the page's topic. Trust is the central member; the other three are evidence for it.

## Authors a machine can resolve

A byline that reads "Staff" or "Admin" is an absent author to a retrieval system. What a machine needs is a **named person, a profile page, and markup that connects the two**.

Google's Article structured data documentation asks that "all the authors that are presented as authors on the web page are also included in markup", that `author.name` should "only specify the name of the author" with no job titles or honorifics, and that `author.url` be "a link to a web page that uniquely identifies the author of the article". It also states that Google understands both `sameAs` and `url` for disambiguating authors. The JSON-LD on an article page therefore looks like this:

```json
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "E-E-A-T for answer engines",
  "datePublished": "2026-04-27",
  "dateModified": "2026-04-27",
  "author": {
    "@type": "Person",
    "@id": "https://example.com/authors/endrit-krasniqi#person",
    "name": "Endrit Krasniqi",
    "url": "https://example.com/authors/endrit-krasniqi",
    "sameAs": ["https://github.com/Endrit-seek"],
    "knowsAbout": ["Answer Engine Optimization", "Structured data", "Next.js"]
  }
}
```

The profile page itself should carry `ProfilePage` markup with the same `Person` as its `mainEntity`. Google's ProfilePage documentation describes the type as intended for sites "where creators (either people or organizations) share first-hand perspectives", and recommends `name`, `sameAs`, `description` and `image` on the entity. Reuse the `@id` so that every article on the site points at one node rather than a fresh anonymous person per page.

schema.org defines `sameAs` as the "URL of a reference Web page that unambiguously indicates the item's identity", such as a Wikipedia page, a Wikidata entry or an official website. An existing GitHub, LinkedIn or ORCID profile is a stronger identity anchor than prose on your own site. `knowsAbout` is a newer schema.org property, defined as indicating "a topic that is known about - suggesting possible expertise but not implying it"; use it to declare subject areas, not to boast.

### Consistent entity names across the site

Language models and knowledge graphs resolve people and organisations by name. If the byline says "E. Krasniqi", the author page says "Endrit K." and the JSON-LD says "Endrit Krasniqi", a machine sees three weak entities where you meant one strong one. Pick the canonical form, store it once in the CMS, and render the byline, the profile page, `Person.name` and the `/llms.txt` entry from that single field. The same rule applies to the site's own name and to any product you write about.

## Dates that are honest

A date is the cheapest trust signal on a page and the easiest to corrupt. Google's [publication dates](https://developers.google.com/search/docs/appearance/publication-dates) guidance asks for a visible, labelled date ("Published February 4, 2019", "Last updated: Feb 14, 2018"), for `datePublished` and `dateModified` in `Article` markup, and that "the date (and optional time and timezone) match between the equivalent user-visible and structured values". It also says not to specify future dates or the date of the action described on the page, and advises removing other dates from the page if Google keeps picking the wrong one.

The common failure is not missing dates but dishonest ones. A `dateModified` equal to the build timestamp tells a retrieval system that every page on the site changed today, which is the same as telling it nothing. A `dateModified` bumped on a cosmetic edit invites a model to prefer a stale page over a genuinely newer one, and to attach the wrong year to a fact it quotes from you.

Derive both dates from CMS fields that an editor has to change deliberately, never from the clock:

```ts
// lib/seo/article-dates.ts
export function articleDates(story: { published_at: string; updated_at?: string }) {
  const datePublished = story.published_at;
  // Fall back to published_at, never to `new Date()`.
  const dateModified = story.updated_at || story.published_at;
  return { datePublished, dateModified };
}
```

Render the same two values into the visible byline and into the JSON-LD from that one function, so they cannot drift. On this site `updated_at` moves only when the substance of a guide changes; correcting a typo does not count.

## Sources a machine can follow

Authoritativeness, to a rater, is reputation. To a retrieval system it is closer to **provenance**: can the claims in this passage be traced somewhere outside it? Two practices make provenance visible.

The first is outbound links to primary sources at the point where a claim is made, rather than a "further reading" list of secondary posts. Link the vendor documentation, the standard, the paper. Quote it where the wording matters, as this guide does, so that a model comparing your passage with its source finds agreement rather than paraphrase.

The second is a citation list the machine can read without parsing prose. schema.org's `citation` property, available on any `CreativeWork`, is defined as "a citation or reference to another creative work, such as another publication, web page, scholarly article, etc." Emit it from the same source list that renders at the bottom of the page:

```json
"citation": [
  {
    "@type": "WebPage",
    "name": "Creating helpful, reliable, people-first content",
    "url": "https://developers.google.com/search/docs/fundamentals/creating-helpful-content",
    "publisher": { "@type": "Organization", "name": "Google Search Central" }
  },
  {
    "@type": "ScholarlyArticle",
    "name": "GEO: Generative Engine Optimization",
    "url": "https://arxiv.org/abs/2311.09735"
  }
]
```

There is evidence that this pays off inside generated answers. The GEO study by Aggarwal et al. tested content modifications against generative engines and reports that adding citations, quotations from sources and statistics were among the best-performing methods; its abstract states that GEO "can boost visibility by up to 40% in generative engine responses". The paper also notes that effectiveness varies by domain, so treat the figure as an upper bound from one benchmark rather than a promise.

**up to 40%** — visibility boost in generative engine responses reported for the best-performing GEO methods, which include adding citations, quotations and statistics (Source: [GEO: Generative Engine Optimization (arXiv)](https://arxiv.org/abs/2311.09735))

### First-hand evidence

Experience is the hardest of the four to encode, because it lives in the content rather than around it. Google's 2022 post describes it as whether content was produced with "actual use of a product, having actually visited a place or communicating what a person experienced". For technical writing that means working code you ran, output you observed, numbers from your own logs and screenshots you took, each labelled as such. A passage that says "on this site, X produced Y" is more quotable and harder to fabricate than one that says "experts recommend X".

> **Warning: Do not manufacture signals**
>
> Invented author personas, borrowed credentials, `dateModified` bumps and citation lists padded with links nobody consulted are detectable by the same consistency checks that make honest signals valuable. Google's helpful-content guidance frames the question as who created the content, how, and why. If the honest answer is unflattering, the fix is the content, not the markup.

## An About page and a site-level entity

The Search Quality Rater Guidelines include a section on finding who is responsible for a website and who created the content on a page, and treat the absence of that information as a negative signal on topics where it matters. A retrieval system runs a cruder version of the same check: does the site declare an organisation, and does that organisation resolve to something beyond the site itself?

Google's Organization structured data documentation recommends placing organisation markup "on your home page, or a single page that describes your organization, for example the about us page", with `name`, `url`, `logo` and `sameAs`. This site's [About page](/about) carries that node, and every guide's `publisher` points at it by `@id`:

```json
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "@id": "https://example.com/#organization",
  "name": "Citable",
  "url": "https://example.com/",
  "logo": "https://example.com/logo.png",
  "sameAs": ["https://github.com/Endrit-seek"]
}
```

Keep the About page in plain HTML: who runs the site, what it covers, how content is produced, and how to reach you. Its job is to answer "who is behind this" for a rater and a model alike.

**E-E-A-T signals a machine can verify**

- [ ] Every page has a named author, a byline linking to a profile page, and Person markup with url and sameAs
- [ ] One canonical spelling of each author, organisation and product name, stored once and rendered everywhere
- [ ] Visible published and updated dates that match datePublished and dateModified exactly
- [ ] dateModified derived from an editorial field, never from build time or trivial edits
- [ ] Outbound links to primary sources at the claim, plus a citation list emitted as schema.org citation
- [ ] First-hand evidence labelled as such: code you ran, data you collected, behaviour you observed
- [ ] An About page with Organization markup, and publisher on every article pointing at it
- [ ] No signal in markup that is not also visible on the page

> **Tip: Where to go next**
>
> The markup in this guide is generated from CMS fields; [Structured data for AI answers](/guides/structured-data-for-ai-answers) shows the full Article, Person and FAQPage graph and how to keep it in sync with the visible text. For the writing side, [How to write answer-first content](/guides/how-to-write-answer-first-content) covers the passage structure that these trust signals wrap around.

## Frequently asked questions

### Is E-E-A-T a ranking factor for AI Overviews or ChatGPT?

No. Google's helpful-content documentation states that E-E-A-T itself is not a specific ranking factor, and its AI features documentation says there are no additional requirements to appear in AI Overviews or AI Mode. Other answer engines publish no equivalent. E-E-A-T is useful as a checklist of the trust signals a retrieval system can verify, not as a score to optimise.

### Do I need a real named author on every page?

For content you want cited, yes. A byline that resolves to a profile page and to Person markup with url and sameAs gives a machine an entity to attach the page to. Pages by an organisation rather than a person can use Organization as the author, but it should still resolve to an About page and external profiles.

### Should I update dateModified whenever I touch a page?

Only when the substance changes. Google asks that visible and structured dates match and warns against dates that do not reflect when the page was published or updated. A dateModified set from the build clock, or bumped for typo fixes, tells a retrieval system that nothing on the site is stable and can attach the wrong date to your facts.

### What counts as a primary source?

The document closest to the fact: vendor documentation, a standard, a published paper, an official announcement, or your own measured data. A blog post summarising one of those is secondary. Link primary sources at the point of the claim, quote them where the wording matters, and list them in a citation block that is also emitted as schema.org citation.

### How do I show Experience on a technical page?

Include evidence that you did the thing: code you ran, output you observed, numbers from your own logs, screenshots you took, each labelled as yours. Google's 2022 update describes Experience as content produced with actual use of a product or first-hand involvement. Passages of the form 'on this site, X produced Y' are both more quotable and harder to fabricate.

## Sources

1. [Creating helpful, reliable, people-first content](https://developers.google.com/search/docs/fundamentals/creating-helpful-content) — Google Search Central (2025)
2. [Our latest update to the quality rater guidelines: E-A-T gets an extra E for Experience](https://developers.google.com/search/blog/2022/12/google-raters-guidelines-e-e-a-t) — Google Search Central Blog (2022)
3. [Search Quality Rater Guidelines (PDF)](https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf) — Google (2025)
4. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Google Search Central (2025)
5. [Article (Article, NewsArticle, BlogPosting) structured data](https://developers.google.com/search/docs/appearance/structured-data/article) — Google Search Central (2025)
6. [GEO: Generative Engine Optimization](https://arxiv.org/abs/2311.09735) — Aggarwal et al., KDD 2024 (arXiv) (2023)

## Related guides

- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)
- [How to write answer-first content that LLMs can quote](https://citable.wiki/guides/how-to-write-answer-first-content)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)

---

Source: https://citable.wiki/guides/eeat-for-answer-engines
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt


<!-- ---------------------------------------------------------------- -->

---
title: "Measuring AI visibility: how to track citations in ChatGPT, Perplexity and AI Overviews"
subtitle: "What you can actually measure about answer-engine citations — referrals, crawler hits, sampled answers and Search Console — and how to define the metrics."
canonical: https://citable.wiki/guides/measuring-ai-visibility-and-citations
category: measurement
author: "Endrit Krasniqi"
date_published: 2026-04-13
date_modified: 2026-04-13
license: CC BY 4.0
---

# Measuring AI visibility: how to track citations in ChatGPT, Perplexity and AI Overviews

*What you can actually measure about answer-engine citations — referrals, crawler hits, sampled answers and Search Console — and how to define the metrics.*

## Short answer

AI visibility is measured from four partial signals, because no answer engine reports citations directly. Referral traffic identifies clicks from chatgpt.com, perplexity.ai, copilot.microsoft.com, claude.ai and gemini.google.com; server logs show which AI crawlers fetch which pages; a fixed panel of prompts run on a schedule gives citation rate and share of answer; and Google Search Console reports AI Overview clicks only blended into Web search totals.

## Key takeaways

- No answer engine publishes a citation report, so AI visibility is triangulated from referrals, crawler logs, sampled prompts and Search Console.
- ChatGPT appends utm_source=chatgpt.com to cited links; other engines are identified by referrer host, and app traffic often arrives with no referrer.
- User-triggered fetchers such as ChatGPT-User, Perplexity-User and Claude-User in your logs are the closest thing to a real-time citation signal.
- A fixed panel of prompts, run weekly with repeated samples, turns citations into two trackable numbers: citation rate and share of answer.
- Google Search Console counts AI Overview and AI Mode clicks inside the Web search type, so they cannot be separated there.
- Commercial tools automate prompt sampling at scale; judge them on panel size, cadence, repeat runs and raw data export.

## Why AI visibility has to be triangulated

Classic search gives you a report: queries, impressions, clicks, position. Answer engines give you nothing of the kind. OpenAI, Perplexity, Anthropic and Microsoft publish no per-site citation data, and Google folds AI Overview traffic into its existing search totals. Measurement therefore means combining four partial signals, each of which sees a different slice of what happens between a question and your page.

| Signal | What it sees | What it misses |
| --- | --- | --- |
| Referral traffic | Citations that produced a click | Citations that did not, and clicks with no referrer |
| Server logs | Which AI crawlers and fetchers requested which URLs | Whether the fetched page was actually used |
| Prompt sampling | Which engines cite you for which questions, and against whom | Every question you did not put in the panel |
| Search Console | Total Google clicks and impressions, AI features included | The split between AI Overviews and ordinary results |

When referrals, user-triggered fetches and sampled citations move in the same direction for a topic, the trend is real. When one moves alone, check the instrument first.

## Referral traffic from answer engines

The cheapest signal is already in your analytics. When someone clicks a citation, the visit arrives with a referrer, a UTM parameter, or both.

OpenAI's Publishers and Developers FAQ states that ChatGPT automatically includes `utm_source=chatgpt.com` in referral URLs so that publishers can track the traffic in tools such as Google Analytics. The other engines are identified by referrer host:

- `chatgpt.com` (and the older `chat.openai.com`) for ChatGPT
- `perplexity.ai` for Perplexity
- `copilot.microsoft.com` for Microsoft Copilot
- `claude.ai` for Claude
- `gemini.google.com` for Gemini

### Grouping them in Google Analytics 4

GA4 files these under Referral by default. Google's custom channel group documentation includes an "AI assistants" example built on a *matches regex* condition against the Source dimension, and notes that custom channel groups apply retroactively. Google's example pattern is deliberately broad; a tighter expression that matches only the hosts above is:

```text
^(www\.)?(chatgpt\.com|chat\.openai\.com|perplexity\.ai|copilot\.microsoft\.com|claude\.ai|gemini\.google\.com)$
```

Place the new channel above Referral so it is evaluated first. A standard GA4 property allows two custom channel groups.

### What referrals undercount

Referral traffic is a lower bound on citations. Answer engines resolve many questions without a click, so a citation often produces no session. Native apps and in-app browsers frequently drop the referrer, so the visit lands in Direct. And some engines put citations in a collapsed source list that few users open. A flat referral line means clicks are rare, which is the normal case, not that you are uncited.

## AI crawler hits in server logs

Your access logs record every fetch, including the ones no analytics script sees. The vendors publish their user agents, and the names mean different things.

OpenAI's crawler documentation lists three relevant agents: `GPTBot` crawls content that may be used for training; `OAI-SearchBot` surfaces websites in ChatGPT's search features; and `ChatGPT-User` handles certain user actions in ChatGPT, where, because the fetch is initiated by a user, robots.txt rules may not apply. Perplexity documents `PerplexityBot` for its search index and `Perplexity-User`, which may visit a page when a user asks a question and which, in Perplexity's words, generally ignores robots.txt. Anthropic documents `ClaudeBot` (training), `Claude-SearchBot` (search quality) and `Claude-User` (fetches on behalf of a user), all of which it says honour robots.txt. Google uses ordinary Googlebot for AI features; `Google-Extended` is a robots.txt control token with no user agent string of its own, so it never appears in a log.

That gives a three-way split worth counting separately:

1. **Training crawlers** (`GPTBot`, `ClaudeBot`, `CCBot`). Volume tells you about training exposure, not about answers.
2. **Search-index crawlers** (`OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`). A fetch means the page is a retrieval candidate.
3. **User-triggered fetchers** (`ChatGPT-User`, `Perplexity-User`, `Claude-User`). A fetch means the page was retrieved while a live question was being answered. This is the closest thing you have to a real-time citation signal.

A short pipeline over an Nginx or Apache combined log gives a daily count per agent and URL:

```bash
grep -E 'GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|Perplexity-User|ClaudeBot|Claude-SearchBot|Claude-User' access.log \
  | awk '{
      match($0, /GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|Perplexity-User|ClaudeBot|Claude-SearchBot|Claude-User/);
      agent = substr($0, RSTART, RLENGTH);
      split($4, d, ":");
      print substr(d[1], 2), agent, $7
    }' \
  | sort | uniq -c | sort -rn | head -50
```

User agent strings are trivially forged, so verify before trusting a spike. OpenAI publishes IP ranges per agent (`openai.com/gptbot.json`, `openai.com/searchbot.json`, `openai.com/chatgpt-user.json`), Perplexity publishes `perplexity.com/perplexitybot.json` and `perplexity.com/perplexity-user.json`, and Anthropic publishes `claude.com/crawling/bots.json`.

> **Warning: Blocking removes the signal**
>
> If you disallow a fetcher in robots.txt or challenge it at the CDN, its hits disappear from your logs along with the citations it would have produced. Decide on access first, using [Controlling AI crawlers with robots.txt](/guides/controlling-ai-crawlers-with-robots-txt), then measure what you allowed.

## Prompt-based citation sampling

Referrals and logs are passive. The only way to learn *which questions* you are cited for, and who is cited instead, is to ask the engines yourself and record the answers.

### Build the panel

Write twenty to fifty questions that your pages are meant to answer, phrased the way users phrase them rather than as keywords. Give each question a stable id, version the panel, and never silently edit a prompt: an edited prompt is a new prompt.

Run each prompt in a fresh, logged-out session on each engine, because answers vary with history, account and location. Generation is non-deterministic, so run each prompt three times and record every run.

### Record the answers

One row per answer:

| Column | Meaning |
| --- | --- |
| `date` | Day of the run |
| `engine` | `chatgpt`, `perplexity`, `copilot`, `claude`, `gemini`, `ai-overviews` |
| `panel_version` | Version of the prompt set, so old rows stay comparable |
| `prompt_id` | Stable id of the question |
| `run` | 1, 2 or 3 |
| `cited` | 1 if your domain appears in the sources, otherwise 0 |
| `position` | Order of your first citation among the sources; blank if not cited |
| `cited_url` | The exact URL cited, to see which passage won |
| `all_domains` | Every cited domain, in order, separated by semicolons |
| `notes` | No sources shown, refusal, wrong entity |

### Define the metrics

- **Citation rate**: answers in which your domain is cited, divided by all answers, per engine.
- **Share of answer**: your citations divided by all citations across the panel's answers, per engine. This is the answer-engine equivalent of share of voice, and the number that falls when a competitor starts winning.
- **Mean citation position**: average of `position` over cited answers. A rising mean with a stable citation rate means you are being pushed down the list.
- **Competitor share**: share of answer computed for every domain in `all_domains`. Keep the whole distribution.
- **Coverage**: answers that showed any sources at all. A coverage drop is not a visibility drop.

The GEO paper by Aggarwal et al. formalises the same idea with two finer measures: *Position-Adjusted Word Count*, which weights the words attributed to a citation by an exponentially decaying function of its position, and *Subjective Impression*, which uses a model to rate how prominent a source looks in the answer. A weekly sheet with `cited` and `position` columns is a coarse approximation of both.

**up to 40%** — increase in visibility in generative engine responses reported for passages that added citations, quotations and statistics (Source: [GEO: Generative Engine Optimization (arXiv)](https://arxiv.org/abs/2311.09735))

> **Example: A weekly readout**
>
> Panel v2, 40 prompts, 3 runs each on Perplexity: 120 answers. Your domain cited in 27 of them, so citation rate is 22.5%. Those 120 answers contained 96 citations in total, 31 of them yours, so share of answer is 32.3%. Mean citation position 1.9. Two competitor domains hold 28% and 17%. Coverage 118 of 120.

## What Google Search Console can and cannot tell you

Search Console is the only first-party source for Google, and its limit is precise. Google's documentation on AI features and your website says that sites appearing in AI features such as AI Overviews and AI Mode "are included in the overall search traffic in Search Console" and are "reported on in the Performance report, within the 'Web' search type". As of this writing there is no filter that isolates them, so an AI Overview click and an ordinary click on the same query are indistinguishable in the export.

What you can still do:

- Export query-level impressions and clicks weekly for the topics in your panel, and watch click-through rate. Impressions holding steady while CTR falls is consistent with an AI Overview absorbing the click, but also with a new competitor or a changed layout.
- Record whether an AI Overview appeared for each query, from a rank tracker that detects it or from your own sampled searches, and whether your URL was among its sources.
- Google's documented controls for AI features (`nosnippet`, `data-nosnippet`, `max-snippet`, `noindex`) are levers, not measurements, but they allow a before-and-after comparison on one section of the site.

## Categories of commercial tools

The market is young and the products overlap. Four categories cover most of it, and none replaces the definitions above.

1. **Prompt-monitoring platforms** automate the panel across engines on a schedule. Ask whether they query the consumer interface or an API (the two can cite differently), how many runs per prompt they take, how they handle regions, and whether raw answers can be exported.
2. **Bot analytics in CDNs and WAFs** classify crawler traffic by user agent and, in the better cases, by verified IP range. Ask whether verification is done or only claimed.
3. **Analytics add-ons and dashboard templates** package an AI channel group and a report. The regex above is most of the product; the value is presentation and alerting.
4. **Rank trackers with AI Overview detection** record whether an AI Overview appeared for tracked keywords and whether your URL was cited in it. Google only, but it fills the Search Console gap.

Whatever you buy, keep the spreadsheet. The metric definitions should be yours, so that changing tools does not reset the series.

**A minimal AI visibility measurement stack**

- [ ] GA4 custom channel group for AI assistants, placed above Referral
- [ ] Weekly count of user-triggered fetches (ChatGPT-User, Perplexity-User, Claude-User) per URL from server logs
- [ ] Crawler user agents verified against the vendors' published IP ranges
- [ ] A versioned prompt panel of 20–50 questions, run three times per engine per week
- [ ] Citation rate and share of answer computed per engine, with competitor domains kept
- [ ] Search Console query-level impressions and CTR exported weekly for the same topics
- [ ] One dashboard row per topic that shows all four signals side by side

> **Tip: Where to go next**
>
> Measurement only tells you where to work. [Controlling AI crawlers with robots.txt](/guides/controlling-ai-crawlers-with-robots-txt) covers the access decisions that determine which fetchers can reach you at all, and [What is Answer Engine Optimization?](/guides/what-is-answer-engine-optimization) explains what to change on a page once the panel shows where you are losing citations.

## Frequently asked questions

### Can I see which prompts led to a ChatGPT citation?

No. ChatGPT sends a referrer and a utm_source=chatgpt.com parameter, but not the conversation or the question. The only way to connect citations to questions is to ask the questions yourself: run a fixed prompt panel, record which URLs each engine cites, and treat the result as a sample rather than a census.

### Why does my AI referral traffic look so small?

Three reasons. Answer engines resolve many questions without a click, so a citation often produces no visit. Native apps and in-app browsers frequently drop the referrer, which files the visit under direct traffic. And referrals count only clicked citations, not the far larger number of times a page was fetched by an engine or quoted without a click.

### Does Google Search Console show AI Overview clicks?

Not as a separate line. Google's documentation states that traffic from AI Overviews and AI Mode is included in the Performance report under the Web search type, mixed with ordinary results. You can watch for impressions holding steady while clicks fall, but that pattern has other causes too, so treat it as a prompt to investigate rather than a measurement.

### How many prompts do I need in a citation panel?

Enough to cover your real topics, and few enough to run consistently. Twenty to fifty questions, each run three times per engine per week, is a practical starting point for one site. Consistency matters more than size: change the panel rarely, version it when you do, and keep the old prompts so that trends stay comparable.

### Is a crawler hit the same as a citation?

No. A training crawler such as GPTBot or ClaudeBot fetching a page says nothing about answers. A search-index crawler fetch means the page is a candidate. A user-triggered fetch by ChatGPT-User, Perplexity-User or Claude-User means the page was retrieved while a live question was being answered, which is the strongest log-based signal, but still not proof that it was cited.

## Sources

1. [Publishers and Developers – FAQ](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq) — OpenAI Help Center (2025)
2. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — OpenAI (2025)
3. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Google Search Central (2025)
4. [[GA4] Custom channel groups](https://support.google.com/analytics/answer/13051316) — Google Analytics Help (2025)
5. [Perplexity crawlers](https://docs.perplexity.ai/guides/bots) — Perplexity (2025)
6. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — Anthropic Help Center (2025)
7. [GEO: Generative Engine Optimization](https://arxiv.org/abs/2311.09735) — Aggarwal et al., KDD 2024 (arXiv) (2023)

## Related guides

- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)
- [Controlling AI crawlers with robots.txt: GPTBot, ClaudeBot, PerplexityBot and friends](https://citable.wiki/guides/controlling-ai-crawlers-with-robots-txt)
- [GEO vs SEO vs AEO: what actually changed](https://citable.wiki/guides/geo-vs-seo-vs-aeo)

---

Source: https://citable.wiki/guides/measuring-ai-visibility-and-citations
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt


<!-- ---------------------------------------------------------------- -->

---
title: "Serving Markdown to LLMs: content negotiation and .md endpoints"
subtitle: "How to give language models a plain-text version of every page, without duplicating content or confusing caches and search engines."
canonical: https://citable.wiki/guides/serving-markdown-to-llms
category: technical
author: "Endrit Krasniqi"
date_published: 2026-03-30
date_modified: 2026-03-30
license: CC BY 4.0
---

# Serving Markdown to LLMs: content negotiation and .md endpoints

*How to give language models a plain-text version of every page, without duplicating content or confusing caches and search engines.*

## Short answer

Serving Markdown to LLMs means publishing a plain-text representation of each page next to the HTML one, so that AI crawlers and agents can read it without rendering JavaScript or spending tokens on markup. There are two delivery patterns: a .md suffix on the canonical URL, and content negotiation on the Accept: text/markdown request header, with Vary: Accept and a Link rel=canonical header pointing back to the HTML page.

## Key takeaways

- A rendered DOM costs a model far more tokens than the same content as Markdown, and many AI fetchers do not execute JavaScript at all.
- The .md suffix pattern is the simplest to ship and to debug; llmstxt.org proposes it and Mintlify-style docs platforms link to it from llms.txt.
- Content negotiation serves Markdown from the canonical URL when a client sends Accept: text/markdown; Vercel and Cloudflare both ship this pattern.
- Every Markdown response needs Content-Type: text/markdown, Vary: Accept and a Link header with rel=canonical so caches and search engines stay consistent.
- Generate Markdown from the same CMS fields as the HTML, with YAML front matter for canonical URL, author, dates and licence.
- Copy-as-Markdown and Open-in-ChatGPT buttons reuse the same endpoint and make the feature visible to people, not only to crawlers.

## Why plain text beats the rendered DOM

A language model does not see your page the way a browser does. Whatever fetches it — a training crawler, a search indexer, or an agent acting for a user — receives bytes, and every byte it keeps costs a token. A modern HTML page is mostly bytes the model has no use for: framework bootstrap scripts, hydration payloads, inline SVG icons, class attributes, cookie-consent markup and navigation repeated on every URL. The article you wrote is a small fraction of the response.

Vercel measured this on the blog post in which it announced Markdown delivery in February 2026: the HTML version of that page was around 500KB and the Markdown version 3KB. Cloudflare's [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) documentation describes an `x-markdown-tokens` header on converted responses, alongside an `x-original-tokens` header, precisely so operators can see the difference.

**3KB** — size of the Markdown version of a Vercel blog post whose HTML response was around 500KB, as measured by Vercel (Source: [Vercel, Making agent-friendly pages with content negotiation](https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation))

Token cost is only the first problem. The second is **fetch budgets**: a crawler has a limited number of bytes and requests per host, and an agent answering a live question has seconds, not minutes. The third is **JavaScript rendering**. Google documents a three-phase pipeline for JavaScript sites — crawl, render, index — in which a page "may stay on this queue for a few seconds, but it can take longer than that" before headless Chromium executes it. OpenAI's crawler documentation lists `GPTBot`, `OAI-SearchBot`, `ChatGPT-User` and `OAI-AdsBot` and says nothing about executing JavaScript at all; you should assume a client-rendered page is an empty shell to them.

Markdown solves all three. It is small, it needs no rendering, and its structure — headings, lists, code fences, links — is the structure a model was trained on. It is also a registered media type: [RFC 7763](https://www.rfc-editor.org/rfc/rfc7763) defines `text/markdown` with a required `charset` parameter and an optional `variant`, which is what makes the second delivery pattern below possible.

> **Note: Markdown is not a ranking signal**
>
> No engine has said that offering Markdown makes a page more likely to be cited. What it does is remove the failure modes — truncation, unexecuted JavaScript, noisy extraction — that stop a good passage from being read at all. Treat it as infrastructure, like a sitemap.

## Pattern one: a .md suffix on the canonical URL

The simplest pattern is a second URL. If the HTML lives at `/guides/serving-markdown-to-llms`, the Markdown lives at `/guides/serving-markdown-to-llms.md`. The llms.txt proposal at [llmstxt.org](https://llmstxt.org/) recommends exactly this: pages "provide a clean markdown version of those pages at the same URL as the original page, either with `.md` appended (`page.html.md`) or with the extension replaced by `.md` (`page.md`)". Mintlify's hosted documentation follows the convention and [notes](https://mintlify.com/docs/ai/llmstxt) that page links in its generated `llms.txt` "include a `.md` extension so AI tools can fetch the Markdown version of each page directly".

The advantages are practical. You can open the URL in a browser to check it. Caches never confuse it with the HTML. Any client that has read your `/llms.txt` knows how to reach it. The response should look like this:

```http
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Content-Disposition: inline
Link: <https://example.com/guides/serving-markdown-to-llms>; rel="canonical"
Cache-Control: public, max-age=0, s-maxage=600, stale-while-revalidate=86400
```

Two details matter. `Content-Disposition: inline` stops browsers downloading the file, so people can read it too. The `Link` header with `rel="canonical"` tells search engines that the HTML page is the one to index; Google [documents this HTTP-header form](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) of canonicalisation for non-HTML documents such as PDFs, and Markdown is no different. Leave `.md` URLs out of your XML sitemap.

## Pattern two: content negotiation on Accept

The second pattern serves Markdown from the canonical URL itself when the client asks for it. In server-driven content negotiation the client's `Accept` header lists the media types it can process, each with an optional quality value, and the server picks the best representation. A client that prefers Markdown sends:

```http
GET /guides/serving-markdown-to-llms HTTP/1.1
Host: example.com
Accept: text/markdown, text/html;q=0.8
```

This is the pattern Vercel described in February 2026 for its own documentation, blog and changelog, and it is what Cloudflare's Markdown for Agents implements at the edge for sites that enable it: when a request carries `Accept: text/markdown`, Cloudflare converts the origin's HTML "on the fly" and returns `content-type: text/markdown` with a token count header. You can try both with `curl -H "Accept: text/markdown"` against their sites.

Content negotiation has one well-known trap: caches. A CDN that stores the Markdown response for the canonical URL and then serves it to a browser has broken your site. The [Vary header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Vary) exists for this. MDN describes it as the header that "describes the parts of the request message (aside from the method and URL) that influenced the content of the response", and it must be sent on **every** response for the URL — the HTML one included — so that the cache keys on `Accept` from the first request onwards. Cloudflare's documentation states that its implementation adds `Accept` to `Vary`, preserving any dimensions the origin already declared, "so that caches store separate variants for Markdown and HTML".

```http
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Link: <https://example.com/guides/serving-markdown-to-llms>; rel="canonical"
```

> **Warning: Parse q-values, not substrings**
>
> Checking `accept.includes("text/markdown")` is not enough. A client may send `text/html, text/markdown;q=0.5`, in which case it wants HTML. Rank the listed types by their `q` value (default 1), then by position, and serve Markdown only when it outranks `text/html`. A `*/*` wildcard should never trigger Markdown.

Agents that do not send the header still need a way to discover the alternative. Vercel's post recommends a `<link rel="alternate" type="text/markdown">` element in the HTML head; pointing it at the page's own `.md` URL is the most precise choice, and the page's `/llms.txt` entry covers the rest.

## Converting CMS richtext to Markdown

Where the Markdown comes from matters as much as how it is served. Converting the rendered HTML back to Markdown works, and it is what an edge converter has to do, but it drags navigation, banners and footers along unless you strip them carefully. If your content lives in a headless CMS you have a better source: the structured fields themselves.

Storyblok, for example, stores a richtext field as ProseMirror-style JSON — a tree of `heading`, `paragraph`, `bullet_list`, `code_block` and `table` nodes with `bold`, `italic`, `code` and `link` marks, plus embedded component bloks. Walking that tree and emitting Markdown is a short, deterministic function. Embedded components need a decision: a callout becomes a blockquote with a bold label, a checklist becomes a task list, a statistic becomes a bold figure with its source link. Render them as readable prose, not as your own directive syntax, because the audience is a model that has never seen your component names.

Wrap the body in YAML front matter carrying the fields a model cannot infer from the text:

```yaml
---
title: "Serving Markdown to LLMs: content negotiation and .md endpoints"
canonical: https://example.com/guides/serving-markdown-to-llms
author: Endrit Krasniqi
author_url: https://example.com/authors/endrit-krasniqi
published: 2026-03-30
updated: 2026-03-30
license: CC BY 4.0
description: How to give language models a plain-text version of every page.
---
```

The canonical URL is what a model will cite. The author and dates are the trust signals a retrieval system checks. The licence tells an operator what they may do with the text; choose one deliberately rather than leaving it implied. Put the short answer immediately after the front matter, then the body, then the FAQ and the sources as a plain list of links, so the whole document reads top-down like the HTML page.

## A Next.js proxy and route handler

In Next.js 16 the `middleware` file convention was renamed to `proxy`; the file lives at `proxy.ts` next to `app` and exports a `proxy` function. The following example handles both patterns: a `.md` suffix is rewritten to a Markdown route handler, and an `Accept` header that prefers Markdown is answered with a `303` redirect to the `.md` URL. A redirect, rather than a rewrite, keeps one URL per representation, so a shared cache can never store the Markdown body under the HTML key — the framework overwrites `Vary` on pre-rendered HTML, which makes the rewrite variant unsafe behind a CDN.

```ts
// src/proxy.ts
import { NextResponse, type NextRequest } from "next/server";

const CONTENT_PATH = /^\/guides\/([a-z0-9-]+)(\.md)?$/;

function prefersMarkdown(accept: string | null): boolean {
  if (!accept) return false;
  const ranked = accept
    .split(",")
    .map((part, index) => {
      const [type, ...params] = part.trim().split(";");
      const q = params.map((p) => p.trim()).find((p) => p.startsWith("q="));
      return { type: type.trim().toLowerCase(), q: q ? Number(q.slice(2)) : 1, index };
    })
    .filter((m) => m.q > 0)
    .sort((a, b) => b.q - a.q || a.index - b.index);
  const md = ranked.find((m) => m.type === "text/markdown");
  if (!md) return false;
  const html = ranked.find((m) => m.type === "text/html");
  return !html || md.q > html.q || (md.q === html.q && md.index < html.index);
}

export function proxy(request: NextRequest) {
  const match = CONTENT_PATH.exec(request.nextUrl.pathname);
  if (!match) return NextResponse.next();
  const [, slug, ext] = match;
  if (ext === ".md") {
    // /guides/<slug>.md → the Markdown route handler, URL unchanged for the client
    const url = request.nextUrl.clone();
    url.pathname = `/api/markdown/${slug}`;
    return NextResponse.rewrite(url);
  }
  if (prefersMarkdown(request.headers.get("accept"))) {
    // Accept: text/markdown on the HTML URL → send the client to the .md URL
    const url = request.nextUrl.clone();
    url.pathname = `/guides/${slug}.md`;
    const res = NextResponse.redirect(url, 303);
    res.headers.set("Vary", "Accept");
    return res;
  }
  return NextResponse.next();
}

export const config = { matcher: ["/guides/:path*"] };
```

The route handler fetches the story from the CMS, converts it and sets the headers discussed above:

```ts
// src/app/api/markdown/[slug]/route.ts
import { getGuide } from "@/lib/cms";
import { guideToMarkdown } from "@/lib/markdown-export";

export async function GET(_req: Request, ctx: { params: Promise<{ slug: string }> }) {
  const { slug } = await ctx.params;
  const guide = await getGuide(slug);
  if (!guide) return new Response("Not found", { status: 404 });

  return new Response(guideToMarkdown(guide), {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "Content-Disposition": "inline",
      Link: `<https://example.com/guides/${slug}>; rel="canonical"`,
      Vary: "Accept",
    },
  });
}
```

Rewriting rather than redirecting is deliberate: the client keeps the canonical URL in its address bar and its citation, and only the representation changes. This site runs exactly this arrangement; `curl -H "Accept: text/markdown"` against any guide URL returns the Markdown version.

## Buttons for people: Copy as Markdown and Open in ChatGPT

Once the endpoint exists, expose it to readers. Documentation platforms have converged on a small menu next to the page title: **Copy page as Markdown**, **View as Markdown**, and **Open in ChatGPT** or **Open in Claude**. Mintlify's [contextual menu](https://mintlify.com/docs/ai/contextual-menu), for instance, offers copy, view, and options that create "a ChatGPT conversation with the current page as context" or "a Claude conversation with the current page as context".

The copy button fetches the `.md` URL and writes the text to the clipboard. The open-in buttons build a URL that opens a new conversation with a prompt asking the assistant to read the page. ChatGPT accepts a `q` query parameter on `https://chatgpt.com/` for this, although OpenAI does not formally document it. Anthropic [documents](https://support.claude.com/en/articles/14729294-open-claude-desktop-with-a-link) a `q` parameter for Claude Desktop deep links of the form `claude://claude.ai/new?q=`, with the prompt URL-encoded and truncated at roughly 14,000 characters; that document covers the desktop app, not the claude.ai website, so test any web link before shipping it. Keep the prompt short and put the Markdown URL in it, not the HTML one.

```ts
const md = `https://example.com/guides/${slug}.md`;
const prompt = encodeURIComponent(`Read ${md} and answer questions about it.`);
const openInChatGPT = `https://chatgpt.com/?q=${prompt}`;
```

**Markdown delivery checklist**

- [ ] A .md URL for every content page, returning Content-Type: text/markdown; charset=utf-8
- [ ] Accept: text/markdown on the canonical URL returns the same bytes, with q-values parsed
- [ ] Vary: Accept on every response for negotiated URLs, including the HTML one
- [ ] Link: \<canonical HTML URL>; rel="canonical" on every Markdown response
- [ ] Content-Disposition: inline so browsers display rather than download
- [ ] Markdown generated from CMS fields, with YAML front matter for canonical, author, dates and licence
- [ ] link rel="alternate" type="text/markdown" in the HTML head, and an entry in /llms.txt
- [ ] .md URLs excluded from the XML sitemap; robots.txt allows the crawlers you want
- [ ] Copy-as-Markdown and Open-in buttons wired to the same endpoint

> **Tip: Where to go next**
>
> Markdown delivery is one of three technical pieces. Read [llms.txt: what it is and how to write one](/guides/llms-txt-what-it-is-and-how-to-write-one) to build the index that points at your `.md` URLs, and [Controlling AI crawlers with robots.txt](/guides/controlling-ai-crawlers-with-robots-txt) to make sure the clients you are serving are allowed to fetch at all.

## Frequently asked questions

### Does serving Markdown create duplicate content problems for SEO?

Not if you mark the relationship. Send a Link header with rel="canonical" pointing at the HTML URL on every Markdown response; Google documents this HTTP header form for non-HTML documents. Keep the Markdown out of your XML sitemap, or list it in a separate Markdown sitemap for agents. The HTML page remains the only URL you want indexed and cited.

### Which is better: a .md suffix or Accept-header content negotiation?

Ship both. The .md suffix is discoverable from llms.txt, easy to test in a browser and cannot be confused by caches. Content negotiation keeps one canonical URL and works for agents that send the header without knowing your URL scheme. Both routes should return byte-identical Markdown from the same generator.

### Do AI crawlers actually send Accept: text/markdown?

Some agents and tools do, and the number is growing now that Cloudflare converts HTML to Markdown for any request carrying that header. Most training crawlers still request HTML by default. That is why the .md suffix, an llms.txt index and a link rel="alternate" tag matter: they give a client three ways to find the Markdown without guessing.

### Can I convert the rendered HTML to Markdown instead of generating it from the CMS?

You can, and edge services do exactly that, but the result carries navigation, cookie banners and footer noise unless you strip them. Generating from CMS fields produces cleaner output, lets you add front matter with author, dates and licence, and guarantees the Markdown never disagrees with the visible text because both come from the same source.

### Does Markdown delivery help a page get cited?

It removes obstacles rather than adding a ranking signal. A model that receives clean Markdown reads the full article within its fetch budget, sees the heading outline and can quote a passage exactly. A model that receives a heavy JavaScript-rendered page may see only a shell. Extraction quality goes up; the decision to cite still depends on the content.

## Sources

1. [Making agent-friendly pages with content negotiation](https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation) — Vercel (2026)
2. [Introducing Markdown for Agents](https://developers.cloudflare.com/changelog/post/2026-02-12-markdown-for-agents/) — Cloudflare (2026)
3. [RFC 7763: The text/markdown Media Type](https://www.rfc-editor.org/rfc/rfc7763) — IETF (2016)
4. [The /llms.txt file](https://llmstxt.org/) — Jeremy Howard, Answer.AI (2024)
5. [Understand JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) — Google Search Central (2025)
6. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — OpenAI (2025)
7. [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) — Cloudflare Docs (2026)
8. [Open Claude Desktop with a link](https://support.claude.com/en/articles/14729294-open-claude-desktop-with-a-link) — Anthropic Help Center (2026)

## Related guides

- [llms.txt: what it is, how to write one, and whether it helps](https://citable.wiki/guides/llms-txt-what-it-is-and-how-to-write-one)
- [Controlling AI crawlers with robots.txt: GPTBot, ClaudeBot, PerplexityBot and friends](https://citable.wiki/guides/controlling-ai-crawlers-with-robots-txt)
- [Modelling content for answer engines in a headless CMS!](https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms)

---

Source: https://citable.wiki/guides/serving-markdown-to-llms
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt


<!-- ---------------------------------------------------------------- -->

---
title: "Controlling AI crawlers with robots.txt: GPTBot, ClaudeBot, PerplexityBot and friends"
subtitle: "Which AI user agents exist, what each one is for, and how to write a robots.txt that keeps you citable without feeding every training run."
canonical: https://citable.wiki/guides/controlling-ai-crawlers-with-robots-txt
category: technical
author: "Endrit Krasniqi"
date_published: 2026-03-16
date_modified: 2026-03-16
license: CC BY 4.0
---

# Controlling AI crawlers with robots.txt: GPTBot, ClaudeBot, PerplexityBot and friends

*Which AI user agents exist, what each one is for, and how to write a robots.txt that keeps you citable without feeding every training run.*

## Short answer

AI crawlers are controlled through robots.txt user-agent groups, and every major vendor runs several agents. OpenAI and Anthropic separate a training crawler (GPTBot, ClaudeBot), a search indexer (OAI-SearchBot, Claude-SearchBot) and a user-triggered fetcher; Perplexity runs PerplexityBot and Perplexity-User. Google's AI Overviews use ordinary Googlebot; Google-Extended only opts out of Gemini training and grounding. Allow the search agents if you want citations, and treat user-triggered fetchers as outside robots.txt.

## Key takeaways

- Each vendor runs separate agents for training, search indexing and user-triggered fetching; block them individually, not as one 'AI' bucket.
- A robots.txt group named after a bot replaces the wildcard group for that bot, so repeat your ordinary Disallow lines in every named group.
- Google's AI Overviews and AI Mode are governed by Googlebot rules; Google-Extended only controls Gemini training and grounding, and does not affect Search.
- OpenAI, Perplexity, Meta and Google all document that user-triggered fetchers may ignore robots.txt because a person requested the page.
- Blocking the training crawlers does not remove you from ChatGPT search or Perplexity, provided OAI-SearchBot and PerplexityBot stay allowed.
- Verify a claimed bot against the vendor's published IP list or reverse DNS before trusting the user-agent string.

## What robots.txt can and cannot do for AI crawlers

robots.txt is the oldest control you have over automated access, and it is the one every AI vendor points to first. The rules are standardised in [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html): a file at `/robots.txt`, groups introduced by `User-agent:` lines, and `Allow:` and `Disallow:` path rules inside each group.

Two properties of the standard matter more for AI crawlers than they ever did for search engines.

First, **a crawler uses only the group that matches its own name**. If GPTBot finds a `User-agent: GPTBot` group, it ignores the `User-agent: *` group entirely. Any path you disallow for everyone must be repeated inside every named group, or the named bot will crawl it.

Second, **robots.txt is a request, not an access control**. Compliant crawlers honour it; nothing forces them to. It cannot remove pages from a training set collected before you added the rule.

There is a third limit that is specific to answer engines. When a person pastes a URL into ChatGPT or asks Perplexity a question, the system fetches the page on that person's behalf. OpenAI, Perplexity, Meta and Google all document that these **user-triggered fetchers** may ignore robots.txt. You can still recognise and rate-limit them by IP, but you cannot opt out of them with a text file.

> **Warning: A named group replaces the wildcard**
>
> Adding `User-agent: OAI-SearchBot` with `Allow: /` and nothing else silently grants that bot access to `/admin/`, `/api/` and anything else your `*` group disallows. Copy the full rule set into each named group.

## The AI user agents, vendor by vendor

Every major vendor runs more than one agent, and most split them the same way: a training crawler, a search indexer and a user-triggered fetcher. The names below are what you put in robots.txt.

### OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

OpenAI's crawler documentation describes **GPTBot** as the crawler for content "that may be used in training our generative AI foundation models", and **OAI-SearchBot** as the agent "used to surface websites in search results in ChatGPT's search features". Both honour robots.txt. **ChatGPT-User** is "used for certain user actions in ChatGPT and Custom GPTs", and because those actions are initiated by a user, OpenAI says robots.txt rules may not apply. IP ranges for each are published at `openai.com/gptbot.json`, `openai.com/searchbot.json` and `openai.com/chatgpt-user.json`.

### Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic's support article follows the same pattern. **ClaudeBot** collects web content "that could potentially contribute to" model training. **Claude-SearchBot** "navigates the web to improve search result quality", and **Claude-User** fetches pages when "individuals ask questions to Claude". Anthropic states that all three honour "do not crawl" directives in robots.txt, and publishes IP ranges at `claude.com/crawling/bots.json`. It also advises against blocking by IP address, because a blocked crawler cannot read your robots.txt and so cannot record the opt-out.

### Perplexity: PerplexityBot and Perplexity-User

Perplexity's documentation lists **PerplexityBot**, "designed to surface and link websites in search results on Perplexity", which respects robots.txt, and **Perplexity-User**, which visits a page when a user asks a question. The docs are explicit: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." IP ranges are published as `perplexitybot.json` and `perplexity-user.json` on perplexity.com. Perplexity states that neither agent is used to collect content for training foundation models, and documents no training crawler.

### Google: Googlebot powers AI Overviews, Google-Extended does not

Google's guidance on AI features says that "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners". AI Overviews and AI Mode draw on the ordinary Search index, so the only way to keep a page out of them is to keep it out of Search, or to limit its snippets with `nosnippet`, `data-nosnippet`, `max-snippet` or `noindex`.

**Google-Extended** is a robots.txt token, not a crawler. Google's common crawlers page describes it as the control publishers use "to manage whether content Google crawls from their sites may be used for training future generations of Gemini models" and for grounding in Gemini apps and Vertex AI, and states that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". Disallowing it costs you nothing in Search or AI Overviews.

Google's separate page on [user-triggered fetchers](https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers), which includes tools such as Google-Agent and Gemini Notebook, says plainly that "because the fetch was requested by a user, these fetchers generally ignore robots.txt rules".

### Apple, Common Crawl, Amazon, Meta and ByteDance

- **Applebot** crawls for Spotlight, Siri and Safari search and, per Apple, also for training its foundation models. **Applebot-Extended** is an opt-out token that "does not crawl webpages"; Apple states that pages which disallow it "can still be included in search results".
- **CCBot** is Common Crawl's crawler, whose open archive is a common raw material for training sets. It honours `User-agent: CCBot` and Common Crawl warns that it is "aware of crawlers falsely identifying themselves as CCBot", publishing its real ranges at `index.commoncrawl.org/ccbot.json`.
- **Amazonbot** is used, in Amazon's words, "to improve our products and services" and "may be used to train Amazon AI models". It respects robots.txt and Amazon publishes its IP addresses.
- **meta-externalagent** crawls "for use cases such as training foundation AI models or improving products by indexing content directly". Meta's crawler page also lists **meta-webindexer** for Meta AI search and **meta-externalfetcher**, which fetches links at a user's request and may bypass robots.txt.
- **Bytespider** is ByteDance's crawler. ByteDance does not publish crawler documentation comparable to the vendors above, so there is no vendor statement on robots.txt compliance or a published IP list to verify against. A robots.txt rule is still worth adding, but treat it as unverified.

| Agent | Vendor | Job | robots.txt |
| --- | --- | --- | --- |
| GPTBot | OpenAI | training | honoured |
| OAI-SearchBot | OpenAI | search index | honoured |
| ChatGPT-User | OpenAI | user fetch | may not apply |
| ClaudeBot | Anthropic | training | honoured |
| Claude-SearchBot | Anthropic | search index | honoured |
| Claude-User | Anthropic | user fetch | honoured |
| PerplexityBot | Perplexity | search index | honoured |
| Perplexity-User | Perplexity | user fetch | generally ignored |
| Googlebot | Google | Search, AI Overviews, AI Mode | honoured |
| Google-Extended | Google | Gemini training and grounding opt-out | token only |
| Applebot-Extended | Apple | training opt-out | token only |
| CCBot | Common Crawl | open crawl archive | honoured |
| meta-externalagent | Meta | training and indexing | honoured |
| Bytespider | ByteDance | training | unverified |

## Training versus citation: the trade-off

The reason the vendors split their agents is that the two uses have opposite economics for you.

A **search indexer** fetches your page so that an answer engine can retrieve it later, quote a passage and attach a [citation](/glossary/citation). Blocking it removes you from that engine's answers, immediately and completely. There is no version of Answer Engine Optimization that works with OAI-SearchBot, Claude-SearchBot or PerplexityBot disallowed.

A **training crawler** fetches your page so that the text can shape model weights. The model may later reproduce your facts and phrasing without any attribution, and you cannot measure it. Whether that is reach or uncompensated use is a policy question, and it is independent of whether you want citations.

The practical position for a site that exists to be cited is therefore: allow every search agent and Googlebot, decide the training agents on policy rather than habit, and accept that user-triggered fetchers will arrive regardless.

> **Note: What this site does**
>
> Citable allows every documented AI agent, including the training crawlers, and names each one explicitly in its [robots.txt](/robots.txt). Naming bots that `*` already allows changes nothing technically; it states intent.

## A robots.txt that allows citation and opts out of training

The file below keeps a site fully visible to answer engines while opting out of training and archive crawlers. Adjust the `Disallow` lines to your own private paths and repeat them in every group.

```txt
# Everyone, including search and answer engines
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

# AI search indexers: same rules as everyone else
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Applebot
Allow: /
Disallow: /api/
Disallow: /admin/

# Training and archive crawlers: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

# Training opt-out tokens that do not crawl
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml
```

Googlebot is intentionally absent from the named groups: it falls through to `*` and inherits the same rules. Amazonbot is absent because Amazon documents it as serving products as well as possible training; put it in whichever group matches your policy.

In a Next.js App Router project the same policy can be generated from `app/robots.ts`, which keeps the bot list in one place next to the rest of your site configuration:

```ts
import type { MetadataRoute } from "next";

const PRIVATE = ["/api/", "/admin/"];
const SEARCH_BOTS = ["OAI-SearchBot", "Claude-SearchBot", "PerplexityBot", "Applebot"];
const TRAINING_BOTS = ["GPTBot", "ClaudeBot", "CCBot", "meta-externalagent", "Bytespider"];
const OPT_OUT_TOKENS = ["Google-Extended", "Applebot-Extended"];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: "*", allow: "/", disallow: PRIVATE },
      { userAgent: SEARCH_BOTS, allow: "/", disallow: PRIVATE },
      { userAgent: [...TRAINING_BOTS, ...OPT_OUT_TOKENS], disallow: "/" },
    ],
    sitemap: "https://example.com/sitemap.xml",
  };
}
```

**Before you ship a robots.txt change**

- [ ] Every named group repeats the Disallow lines from the \`\*\` group
- [ ] OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot can reach every page you want cited
- [ ] Google-Extended and Applebot-Extended are treated as opt-out tokens, not crawlers
- [ ] The Sitemap line points at an absolute URL that returns 200
- [ ] The file is served at exactly \`/robots.txt\` with \`Content-Type: text/plain\`
- [ ] You have fetched it from production and read it back, not just the source
- [ ] Your /llms.txt and sitemap only list URLs the allowed bots can actually fetch

## Verifying that a bot is who it says it is

A user-agent string is a request header; anyone can send `GPTBot/1.4`, and scrapers do because sites whitelist it. Common Crawl's own page notes impersonation of CCBot. Check the source address before you act on a bot's identity, whether that is serving a Markdown variant or exempting it from rate limits.

Every vendor here except ByteDance publishes its crawler addresses as JSON lists of CIDR prefixes; fetch them on a schedule and match client IPs at the edge.

```bash
# Published address lists (all return JSON with CIDR prefixes)
curl -s https://openai.com/searchbot.json
curl -s https://openai.com/gptbot.json
curl -s https://claude.com/crawling/bots.json
curl -s https://www.perplexity.com/perplexitybot.json
curl -s https://index.commoncrawl.org/ccbot.json
curl -s https://developers.google.com/static/crawling/ipranges/common-crawlers.json
```

For Googlebot, Google also documents [reverse DNS verification](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot): look up the PTR record for the IP, confirm the hostname ends in `googlebot.com`, `google.com` or `googleusercontent.com`, then resolve that hostname forward and check it returns the original IP. Google's example, `host 66.249.66.1`, resolves to a `crawl-66-249-66-1.googlebot.com` name.

If a request claims to be a documented bot and its IP is not in the published list, it is not that bot. Log it, rate-limit it, and do not count it in your [measurement](/guides/measuring-ai-visibility-and-citations) of AI crawler traffic.

> **Tip: Where to go next**
>
> Once the right bots can reach you, give them something easy to read: [llms.txt: what it is and how to write one](/guides/llms-txt-what-it-is-and-how-to-write-one) covers the curated index, and [Serving Markdown to LLMs](/guides/serving-markdown-to-llms) covers delivering the same content without the HTML overhead.

## Frequently asked questions

### Does blocking GPTBot stop my site appearing in ChatGPT search?

No. OpenAI documents GPTBot as the crawler for content that may be used to train its models and OAI-SearchBot as the agent that surfaces websites in ChatGPT search. They are separate user agents with separate robots.txt groups, so a Disallow for GPTBot leaves ChatGPT search untouched as long as OAI-SearchBot is still allowed.

### Will Google-Extended keep my pages out of AI Overviews?

No. Google states that AI is built into Search, so the robots.txt rules for Googlebot are the control for AI Overviews and AI Mode. Google-Extended is a separate product token that manages whether crawled content may be used to train future Gemini models or for grounding in Gemini apps; Google says it does not affect inclusion or ranking in Search. Use nosnippet, max-snippet or noindex to limit what appears in AI features.

### Do AI crawlers actually obey robots.txt?

The documented crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, Amazon and Meta state that they honour robots.txt. The exceptions are user-triggered fetchers such as ChatGPT-User, Perplexity-User, meta-externalfetcher and Google's user-triggered fetchers, which the vendors say may ignore robots.txt because a person asked for the page. Bytespider has no comparable public documentation, so treat its compliance as unverified.

### Should I block AI crawlers at all if I want to be cited?

Only the ones that do not produce citations. Search indexers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot are how answer engines find pages to cite; blocking them removes you from those answers. Training crawlers such as GPTBot, ClaudeBot and CCBot feed model weights, which may or may not credit you later. Most sites that want citations allow the search agents and decide the training agents case by case.

### How do I know a request really comes from the bot it claims to be?

Check the source IP, not the user-agent string, which anyone can copy. OpenAI, Anthropic, Perplexity, Google, Common Crawl and Amazon publish IP range lists in JSON; Google also documents reverse DNS verification against googlebot.com, google.com or googleusercontent.com. Common Crawl explicitly warns that other crawlers falsely identify themselves as CCBot.

## Sources

1. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — OpenAI (2025)
2. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — Anthropic (2025)
3. [Perplexity crawlers](https://docs.perplexity.ai/guides/bots) — Perplexity (2025)
4. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Google Search Central (2025)
5. [Google's common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) — Google Search Central (2025)
6. [Google's user-triggered fetchers](https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers) — Google Search Central (2025)
7. [About Applebot](https://support.apple.com/en-us/119829) — Apple (2025)
8. [CCBot](https://commoncrawl.org/ccbot) — Common Crawl (2025)
9. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) — IETF (2022)

## Related guides

- [llms.txt: what it is, how to write one, and whether it helps](https://citable.wiki/guides/llms-txt-what-it-is-and-how-to-write-one)
- [Serving Markdown to LLMs: content negotiation and .md endpoints](https://citable.wiki/guides/serving-markdown-to-llms)
- [Measuring AI visibility: how to track citations in ChatGPT, Perplexity and AI Overviews](https://citable.wiki/guides/measuring-ai-visibility-and-citations)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)

---

Source: https://citable.wiki/guides/controlling-ai-crawlers-with-robots-txt
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt


<!-- ---------------------------------------------------------------- -->

---
title: "llms.txt: what it is, how to write one, and whether it helps"
subtitle: "The exact format of the llmstxt.org proposal, a working example, how to generate the file from a CMS, and an honest account of who reads it."
canonical: https://citable.wiki/guides/llms-txt-what-it-is-and-how-to-write-one
category: technical
author: "Endrit Krasniqi"
date_published: 2026-03-02
date_modified: 2026-03-02
license: CC BY 4.0
---

# llms.txt: what it is, how to write one, and whether it helps

*The exact format of the llmstxt.org proposal, a working example, how to generate the file from a CMS, and an honest account of who reads it.*

## Short answer

llms.txt is a proposed convention, published by Jeremy Howard of Answer.AI in September 2024, for a Markdown file at /llms.txt that gives language models a curated index of a site: an H1 title, a blockquote summary and H2 sections of links with one-line descriptions. No major AI provider has confirmed reading it, but it costs almost nothing to generate from a CMS and is widely published by documentation sites.

## Key takeaways

- llms.txt is a Markdown index at the site root: one H1, a blockquote summary, and H2 sections of links with one-line descriptions.
- Only the H1 is required; everything else in the format is optional, which is why the file is so easy to generate.
- As of early 2026 Google, OpenAI, Anthropic and Perplexity document their crawlers without any mention of reading llms.txt.
- Google's guidance for AI features states that no new machine-readable files or AI text files are needed to appear in them.
- Generate the file from CMS fields rather than writing it by hand, so it can never drift from the published pages.
- The realistic payoff is on-demand use by agents and coding assistants, not ranking or citation gains in consumer answer engines.

## What llms.txt is, and what it is not

**llms.txt** is a plain Markdown file served at the root of a website, at `/llms.txt`, that tells a language model what the site contains and where its most useful pages are. Jeremy Howard of Answer.AI published the proposal at [llmstxt.org](https://llmstxt.org/) on 3 September 2024. It is a community convention with a GitHub repository, not a web standard, and nothing obliges any model or crawler to look for it.

The idea rests on one observation from the proposal: "At the moment the most widely and easily understood format for language models is Markdown. Simply showing where key Markdown files can be found is a great first step." A model with a limited context window cannot read a whole site, and an HTML page carries navigation, scripts and boilerplate that waste that context. A short, curated Markdown index lets the model pick the right page and fetch a clean version of it.

It helps to be precise about what the file is *not*, because the name invites comparisons.

- It is not `robots.txt`. `robots.txt` tells crawlers what they may fetch; `llms.txt` tells a model what is worth fetching. The two are complementary, and `robots.txt` must not block `llms.txt` for the second to be of any use. See [controlling AI crawlers with robots.txt](/guides/controlling-ai-crawlers-with-robots-txt).
- It is not a sitemap. A sitemap lists every URL for a crawler that intends to index everything. `llms.txt` is selective and carries a human-written description per link.
- It is not a consent or policy file. It says nothing about training permissions, and no provider treats it as one.

> **Definition: llms.txt**
>
> A Markdown file at `/llms.txt` with an H1 site name, an optional blockquote summary, optional free-text paragraphs, and H2 sections of links written as `- [Title](url): description`. See the [glossary entry](/glossary/llms-txt).

## The format, line by line

The proposal is short enough to hold in your head. In order, a file contains:

1. **An H1** with the name of the project or site. The proposal calls this the only required section.
2. **A blockquote** with a short summary, "containing key information necessary for understanding the rest of the file".
3. **Zero or more Markdown blocks** of any kind except headings: paragraphs, lists, notes on how to use the site.
4. **Zero or more H2 sections**, each containing a list of links. Every list item is a required Markdown hyperlink `[name](url)`, then optionally a colon and notes about that URL.
5. **An **`## Optional`** section**, by convention last, for secondary links an agent can drop when it needs a shorter context.

Two details matter more than they look. First, the description after the colon is where the value is: a model deciding which page to fetch reads that sentence, not the page, so write it as a direct answer rather than a teaser. Second, the proposal recommends that each linked page also be available as clean Markdown at the same URL with `.md` appended, so a model that follows a link gets text rather than rendered HTML. That is the subject of [serving Markdown to LLMs](/guides/serving-markdown-to-llms).

### A working example

This is an abbreviated version of the file this site generates, with the domain replaced. Every description is the page's own short answer or glossary definition, taken from the same CMS field that renders on the page.

```markdown
# Citable

> Citable is an open field guide to Answer Engine Optimization (AEO), Generative Engine Optimization (GEO) and LLM-readable content.

Each page is available as Markdown by appending `.md` to its URL. The full text of every guide is in https://example.com/llms-full.txt.

## Guides: Fundamentals

- [What is Answer Engine Optimization (AEO)?](https://example.com/guides/what-is-answer-engine-optimization.md): AEO is the practice of structuring content so that AI answer engines can retrieve, understand and cite it.

## Glossary

- [llms.txt](https://example.com/glossary/llms-txt.md): A proposed convention for a Markdown file at the root of a website that gives language models a curated overview of the site.

## Optional

- [Sitemap](https://example.com/sitemap.xml): every URL on the site
- [About](https://example.com/about): how the site is built
```

### llms-full.txt

`llms-full.txt` is a companion convention rather than part of the core proposal. It is a single Markdown document containing the full text of every page the index points to, so that a tool can load an entire documentation set in one request. Mintlify's documentation states that it "automatically hosts an `llms-full.txt` file at the root of your project", alongside the index, for every site on its platform, which is a large part of why the pattern is now common.

The two files divide labour cleanly: `llms.txt` is small and read first; `llms-full.txt` is large and read only when a tool has decided it wants everything. If you generate one, generating the other is a loop over the same content.

## Generating llms.txt from a CMS

Do not write the file by hand. A hand-written index goes stale the first time someone publishes a page without updating it, and a stale index sends a model to the wrong place. Treat `llms.txt` as a rendering of your content model, like a sitemap or an RSS feed.

In a headless CMS the ingredients already exist: a guide has a title, slug, category and short answer; a glossary term has a term and a definition. The index is those fields, formatted. The one thing worth modelling deliberately is the description: give every content type a short, self-contained summary field and the file writes itself, while the same field feeds your meta description and structured data. See [modelling content for answer engines in a headless CMS](/guides/modeling-content-for-answer-engines-in-a-headless-cms).

In Next.js a route handler is enough. This is a simplified version of the handler behind this site's own file:

```ts
// app/llms.txt/route.ts
import { getAllGuides, getSiteConfig } from "@/lib/cms";

export const revalidate = 600; // regenerate at most every ten minutes

export async function GET() {
  const [site, guides] = await Promise.all([getSiteConfig(), getAllGuides()]);

  const lines = [
    `# ${site.name}`,
    "",
    `> ${site.description}`,
    "",
    "## Guides",
    "",
    ...guides.map(
      (g) => `- [${g.title}](${site.url}/guides/${g.slug}.md): ${g.shortAnswer}`,
    ),
  ];

  return new Response(lines.join("\n") + "\n", {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "Cache-Control": "public, s-maxage=600, stale-while-revalidate=86400",
    },
  });
}
```

The same shape works in any framework: read the published stories, build a string, return it with a text content type. If your CMS fires a webhook on publish, use it to purge the cached response so the file updates on change rather than on a timer.

**Before you ship llms.txt**

- [ ] The file is at the site root, returns HTTP 200, and is not blocked by robots.txt or a WAF rule
- [ ] The first line is a single H1 with the site name, followed by a blockquote summary
- [ ] Every link is absolute, points at a public page, and has a one-sentence description after the colon
- [ ] Descriptions are generated from a CMS field, not typed into the file
- [ ] Linked pages have a Markdown version at the same URL with .md appended
- [ ] Secondary links live under an Optional heading
- [ ] llms-full.txt exists if you publish more than a handful of pages, and is generated by the same code
- [ ] The response carries a UTF-8 charset and a cache policy that is purged on publish

## Who actually reads it

This is the part most write-ups skip. As of early 2026, no major AI provider has said that its consumer products read `llms.txt`.

Google is the clearest. Its documentation on AI features says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." In April 2025 Google's John Mueller wrote on Reddit, as reported by Search Engine Journal, that "none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)", and compared the file to the keywords meta tag: a claim a site makes about itself that a reader would still have to verify by reading the site.

OpenAI's crawler documentation describes GPTBot, OAI-SearchBot and ChatGPT-User, among others, and how to control them in `robots.txt`; it says nothing about those crawlers reading `llms.txt` on your site. Anthropic's page on ClaudeBot, Claude-User and Claude-SearchBot is the same: `robots.txt` directives, no mention of `llms.txt`. Perplexity's documentation for PerplexityBot and Perplexity-User follows the same pattern. If these products fetched the file, their crawler pages would be the place to say so.

There is an irony here. The developer documentation sites of OpenAI, Anthropic and Perplexity each publish an `llms.txt` of their own, as does Cloudflare, and llmstxt.org names the OpenAI, Anthropic and Gemini developer docs as examples. The teams building the models find the file useful for their docs; the products they ship have not committed to reading it from yours.

> **Warning: What not to expect**
>
> Publishing `llms.txt` will not change how Google's AI Overviews, ChatGPT search or Perplexity rank or cite your pages. Anyone selling it as a citation or ranking lever is selling something the vendors have not confirmed. Treat any before-and-after statistic with suspicion unless the author can show the file being fetched in server logs.

## Why it is still worth ten minutes

Given all that, the case for the file is modest and practical rather than strategic.

- **It is nearly free.** The generator above is a few dozen lines that read fields you already have, with no ongoing cost once it is wired to publish.
- **Agents fetch it on demand.** The realistic reader is not an indexing crawler but a coding assistant, research agent or retrieval pipeline that has been pointed at your domain and wants a map of it. Documentation tools adopted the file for that use. Your server logs will tell you whether it is happening: look for requests to `/llms.txt` and to `.md` URLs.
- **It forces good hygiene.** A useful index needs a one-sentence description for every page, absolute canonical URLs and a Markdown rendition of each page. All three help every retrieval system, whether or not it reads the index, which is the logic behind [Answer Engine Optimization](/guides/what-is-answer-engine-optimization) generally.
- **It is your own best RAG source.** If you build internal search or a support assistant over your content, `llms-full.txt` is already the corpus and `llms.txt` the routing table.

Publish it, generate it, and do not build a strategy on it. When a provider announces that it reads the file, you will already have one.

> **Tip: Where to go next**
>
> The file only pays off if the pages it links to are clean: read [serving Markdown to LLMs](/guides/serving-markdown-to-llms) for the `.md` URL and content-negotiation side. Then make sure the crawlers you want can actually fetch it in [controlling AI crawlers with robots.txt](/guides/controlling-ai-crawlers-with-robots-txt).

## Frequently asked questions

### Is llms.txt an official web standard?

No. It is a community proposal published at llmstxt.org by Jeremy Howard of Answer.AI in September 2024, with a public GitHub repository for discussion. No standards body has adopted it and no AI provider is obliged to honour it. It sits in the same category as humans.txt or security.txt before RFC 9116: a convention that works only to the extent that readers choose to look for it.

### Does Google read llms.txt?

Google has not said that it does. Its documentation on AI features states that you do not need to create new machine-readable files or AI text files to appear in AI Overviews or AI Mode. In April 2025 Google's John Mueller wrote on Reddit that no AI service had said it uses the file and that server logs show they do not check for it, comparing it to the keywords meta tag.

### What is the difference between llms.txt and llms-full.txt?

llms.txt is an index: a short file of links and one-line descriptions that a model reads first to decide what to fetch. llms-full.txt is a companion convention, popularised by documentation tooling such as Mintlify, that concatenates the full Markdown text of every page into one document so a tool can load an entire site into context in a single request.

### Should llms.txt be served as text/plain or text/markdown?

The proposal describes the contents of the file, not its HTTP headers, so either works in practice. Serve it with an explicit UTF-8 charset, make sure robots.txt does not block it, and do not put it behind JavaScript or a login. The file is only useful if a plain HTTP GET returns the Markdown directly.

### How long should llms.txt be?

The proposal sets no limit, but the purpose of the file is to fit in a model's context window alongside the user's task. Keep the root index to the pages you would hand a new colleague on day one, put secondary links under an Optional heading, and move the full text of pages into llms-full.txt or per-page Markdown URLs.

## Sources

1. [The /llms.txt file](https://llmstxt.org/) — Jeremy Howard, Answer.AI (2024)
2. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Google Search Central (2025)
3. [Google Says LLMs.Txt Comparable To Keywords Meta Tag](https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/) — Search Engine Journal (2025)
4. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — OpenAI (2025)
5. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — Anthropic (2025)
6. [llms.txt](https://www.mintlify.com/docs/ai/llmstxt) — Mintlify (2025)

## Related guides

- [Serving Markdown to LLMs: content negotiation and .md endpoints](https://citable.wiki/guides/serving-markdown-to-llms)
- [Controlling AI crawlers with robots.txt: GPTBot, ClaudeBot, PerplexityBot and friends](https://citable.wiki/guides/controlling-ai-crawlers-with-robots-txt)
- [Modelling content for answer engines in a headless CMS!](https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)

---

Source: https://citable.wiki/guides/llms-txt-what-it-is-and-how-to-write-one
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt


<!-- ---------------------------------------------------------------- -->

---
title: "Structured data for AI answers: the schema.org types that matter"
subtitle: "Which JSON-LD types tell a machine what a passage is — an answer, a definition, an author — and how to generate them so they never disagree with the page."
canonical: https://citable.wiki/guides/structured-data-for-ai-answers
category: structured-data
author: "Endrit Krasniqi"
date_published: 2026-02-16
date_modified: 2026-08-20
license: CC BY 4.0
---

# Structured data for AI answers: the schema.org types that matter

*Which JSON-LD types tell a machine what a passage is — an answer, a definition, an author — and how to generate them so they never disagree with the page.*

## Short answer

Structured data for AI answers is JSON-LD that labels what a page contains: Article or TechArticle for provenance (author, datePublished, dateModified, citation), FAQPage for question–answer pairs, DefinedTerm for definitions, Person with sameAs for authors, BreadcrumbList and WebSite for context, and SpeakableSpecification for the direct answer. Google requires no markup for AI features; its value is removing ambiguity, so generate it from the CMS fields that render the visible text.

## Key takeaways

- Google states that no special schema.org markup is needed for AI Overviews or AI Mode; structured data clarifies content rather than ranking it.
- TechArticle with author, datePublished, dateModified and citation gives a retrieval system provenance it can check without reading the page.
- FAQPage no longer earns rich results for most sites, but it still exposes question–answer pairs in the exact shape answer engines extract.
- DefinedTerm and DefinedTermSet make a glossary machine-readable: one term, one definition, one stable @id per entity.
- Person with sameAs links an author to profiles elsewhere, which is how machines disambiguate names.
- Generate every JSON-LD value from CMS fields so the markup can never disagree with the visible text.

## What structured data can and cannot do for an answer

Google's documentation for AI features is unusually direct. The page *AI features and your website* says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add." It adds that the existing best practices for SEO remain relevant. Structured data is therefore not a lever that ranks a page inside AI Overviews or AI Mode, and no other answer engine vendor has published anything that says otherwise.

What structured data does is narrower and still worth doing. A crawler that reads a page sees text. It has to infer that a paragraph is a definition, that a name in the byline is the author, that a list at the foot of the page is the sources. JSON-LD states those things outright, in a vocabulary that search and AI companies already parse, and removes the inference step where machines make mistakes.

Two constraints shape everything below. Google recommends JSON-LD over microdata and RDFa because it is "the easiest solution for website owners to implement and maintain at scale". And Google's structured data policies say: "Don't mark up content that is not visible to readers of the page." Markup describes the page; it never substitutes for it.

> **Note: A labelling layer, not a ranking layer**
>
> Treat every type in this guide as a way of saying "this passage is an X" about text that is already on the page. If a passage is not good enough to be quoted, marking it up will not change that.

## Article and TechArticle: provenance a machine can check

The guide [What is Answer Engine Optimization](/guides/what-is-answer-engine-optimization) argues that retrieval systems lean on the signals they can verify: who wrote a page, when, and on what basis. `Article` carries all three. `TechArticle` is the schema.org subtype for technical and how-to material, and it is what this site emits for guides.

Google's Article documentation lists `author`, `author.name`, `author.url`, `datePublished`, `dateModified`, `headline` and `image` as recommended properties. It asks for dates "in ISO 8601 format" and recommends including a timezone; without one, Google falls back to "the timezone used by Googlebot". Two further properties matter for answers even though Google does not list them:

- `citation` — schema.org defines it as "a citation or reference to another creative work, such as another publication, web page, scholarly article, etc." It accepts a `CreativeWork` or plain text. List the same sources you show at the foot of the page.
- `about` — the entities the page is about, ideally as `DefinedTerm` references with stable `@id` values, covered below.

```json
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "@id": "https://example.com/guides/structured-data-for-ai-answers#article",
  "mainEntityOfPage": {
    "@type": "WebPage",
    "@id": "https://example.com/guides/structured-data-for-ai-answers"
  },
  "headline": "Structured data for AI answers: the schema.org types that matter",
  "description": "Structured data for AI answers is JSON-LD that labels what a page contains …",
  "datePublished": "2026-02-16T09:00:00+01:00",
  "dateModified": "2026-02-16T09:00:00+01:00",
  "inLanguage": "en",
  "author": { "@id": "https://example.com/authors/endrit-krasniqi#person" },
  "publisher": { "@id": "https://example.com/#organization" },
  "isPartOf": { "@id": "https://example.com/#website" },
  "citation": [
    {
      "@type": "CreativeWork",
      "name": "AI features and your website",
      "url": "https://developers.google.com/search/docs/appearance/ai-features",
      "publisher": { "@type": "Organization", "name": "Google Search Central" }
    }
  ],
  "about": [
    {
      "@type": "DefinedTerm",
      "@id": "https://example.com/glossary/json-ld#term",
      "name": "JSON-LD"
    }
  ]
}
```

Reference the author, publisher and website by `@id` rather than repeating their details in every article, so each node exists once.

**up to 40%** — visibility boost reported in the GEO benchmark, with adding citations, quotations and statistics among the strongest methods (Source: [GEO: Generative Engine Optimization (arXiv)](https://arxiv.org/abs/2311.09735))

The GEO study measured citations visible in the text, not the `citation` property; the property makes the same list legible to a parser and does not replace it.

## FAQPage after Google retired the rich result

In August 2023 Google's Search Central blog announced that "FAQ (from FAQPage structured data) rich results will only be shown for well-known, authoritative government and health websites", and added: "While you can drop this structured data from your site, there's no need to proactively remove it." For most sites, FAQPage stopped earning a search appearance on that day. On 8 May 2026 Google's documentation changelog added a deprecation notice saying the feature "will no longer appear in Google Search starting May 7, 2026", and the FAQ rich result documentation was removed in June 2026. The search appearance is now gone for every site.

It is still worth emitting, for a different reason. A `Question` with an `acceptedAnswer` is the exact unit an answer engine extracts: one question, one self-contained answer, no surrounding context required. Keep the answer text identical to the visible FAQ, and keep each answer between roughly thirty and eighty words so it is quotable on its own.

```json
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "@id": "https://example.com/guides/structured-data-for-ai-answers#faq",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "Is structured data a ranking factor for AI Overviews?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "No. Google's documentation for AI features says there is no special schema.org structured data you need to add …"
      }
    }
  ]
}
```

Do not use FAQPage to carry questions that are not visibly answered on the page. That is precisely what the content guideline prohibits.

## DefinedTerm and DefinedTermSet: one entity, one definition

Language models resolve entities by name, and a name defined in several places, slightly differently each time, is harder to resolve than one defined once. `DefinedTerm` is schema.org's type for "a word, name, acronym, phrase, etc. with a formal definition"; `DefinedTermSet` groups terms into a glossary. Each glossary page on this site emits one `DefinedTerm` with an `@id`, and the glossary index emits the set:

```json
{
  "@context": "https://schema.org",
  "@type": "DefinedTerm",
  "@id": "https://example.com/glossary/speakable#term",
  "name": "Speakable",
  "alternateName": ["SpeakableSpecification"],
  "description": "Speakable is a schema.org property (with the SpeakableSpecification type) that identifies, by CSS selector or XPath, which parts of a page are best suited to be read aloud or quoted verbatim.",
  "url": "https://example.com/glossary/speakable",
  "inDefinedTermSet": { "@id": "https://example.com/glossary#set" }
}
```

The `@id` is what makes this useful beyond the glossary. Any guide that discusses the term references the same `@id` in its `about` array, so a parser can connect a definition, the pages that use it and the pages that link to it without string matching. The `description` is the same field that renders as the visible definition; the two cannot diverge.

## Person, BreadcrumbList and WebSite: context around the page

### Person with sameAs

Author markup only works if the author is unambiguous. `sameAs` takes URLs that identify the same person elsewhere — a GitHub profile, a personal site, an ORCID record — and Google's Article documentation names `sameAs` as an alternative to `author.url` for disambiguating authors. One `Person` node with an `@id`, referenced from every article, is enough.

```json
{
  "@context": "https://schema.org",
  "@type": "Person",
  "@id": "https://example.com/authors/endrit-krasniqi#person",
  "name": "Endrit Krasniqi",
  "url": "https://example.com/authors/endrit-krasniqi",
  "jobTitle": "Front-end engineer",
  "sameAs": ["https://github.com/Endrit-seek"],
  "worksFor": { "@id": "https://example.com/#organization" }
}
```

### BreadcrumbList

Breadcrumbs tell a crawler where a page sits within the site. Google requires `position`, `name` and `item` on each `ListItem`, with `item` optional for the final entry, and at least two items for the rich result. Generate the list from the route rather than by hand: home, section, page.

### WebSite

`WebSite` markup once powered the sitelinks search box. Google's blog announced it would remove that visual element "starting on November 21, 2024", while the `WebSite` type "continues to be supported" for site names. The site-names documentation requires the markup to be on the home page — "the domain or subdomain level root URI" — with `name` and `url`, plus an optional `alternateName`. Give it an `@id` and let every article's `isPartOf` point at it.

```json
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "@id": "https://example.com/#website",
  "url": "https://example.com/",
  "name": "Citable",
  "inLanguage": "en",
  "publisher": { "@id": "https://example.com/#organization" }
}
```

## SpeakableSpecification: pointing at the answer

[Speakable](/glossary/speakable) is the most literal type in this guide: it names the element on the page that contains the passage to read aloud or quote. Google documents it as a beta feature, used by the Google Assistant to answer news queries from publishers writing in English; it recommends "roughly two to three sentences" per section, and says to use either `cssSelector` or `xpath`, not both. No answer engine has confirmed that it uses the property. It is nonetheless nearly free if your template already has a short-answer block:

```json
"speakable": {
  "@type": "SpeakableSpecification",
  "cssSelector": ["[data-speakable='short-answer']", "[data-speakable='headline']"]
}
```

The selector targets a `data-speakable` attribute rather than a class name, so a redesign cannot silently break it.

## Generate markup from CMS fields so it never drifts

The common failure with structured data is not missing markup. It is markup that says something the page no longer says: a `dateModified` nobody updates, an author who has left, an FAQ answer edited in the CMS but not in the JSON-LD. Each of those breaks Google's content guideline, and each is a false statement made to the system you are hoping will quote you.

The fix is architectural. Structured data should be a projection of the same fields that render the visible page, so that nothing is typed twice. On this site the guide template builds its JSON-LD from the Storyblok story in one function, with no literals of its own:

```ts
import { absoluteUrl, pathForSlug } from "@/lib/site";
import type { GuideStory } from "@/lib/storyblok/types";

export function guideJsonLd(guide: GuideStory) {
  const c = guide.content;
  const url = absoluteUrl(pathForSlug(guide.full_slug));
  return {
    "@context": "https://schema.org",
    "@type": "TechArticle",
    "@id": `${url}#article`,
    headline: c.title,
    description: c.short_answer,
    datePublished: c.published_at,
    dateModified: c.updated_at || c.published_at,
    author: { "@id": absoluteUrl(`${pathForSlug(c.author.full_slug)}#person`) },
    citation: c.sources.map((s) => ({
      "@type": "CreativeWork",
      name: s.title,
      url: s.url,
    })),
    speakable: {
      "@type": "SpeakableSpecification",
      cssSelector: ["[data-speakable='short-answer']"],
    },
  };
}
```

The component that renders it escapes `<`, so a string value can never close the script element early:

```tsx
export function JsonLdScript({ data }: { data: object }) {
  const json = JSON.stringify(data).replace(/</g, "\\u003c");
  return <script type="application/ld+json" dangerouslySetInnerHTML={{ __html: json }} />;
}
```

Because the same `short_answer` field feeds the visible block, the `description` property and the speakable selector, editing it in the CMS updates all three at once. That is the property to design for: one source of truth, several projections.

**Structured data review for an answer page**

- [ ] Every value in the JSON-LD is derived from a CMS field, never typed separately
- [ ] datePublished and dateModified are ISO 8601 with a timezone, and dateModified changes on every edit
- [ ] author is a Person node with @id, url and at least one sameAs
- [ ] citation lists the same sources shown on the page
- [ ] FAQPage answers are identical to the visible FAQ text
- [ ] Glossary terms use DefinedTerm with a stable @id, referenced from about on the pages that use them
- [ ] speakable targets a data attribute, not a styling class
- [ ] BreadcrumbList is generated from the route
- [ ] Output checked in Google's Rich Results Test and the Schema Markup Validator after every template change

> **Tip: Where to go next**
>
> [FAQ content for answer engines](/guides/faq-content-for-answer-engines) covers writing the question–answer pairs that FAQPage describes. [Modelling content for answer engines in a headless CMS](/guides/modeling-content-for-answer-engines-in-a-headless-cms) shows how to design the fields this markup is generated from.

## Frequently asked questions

### Is structured data a ranking factor for AI Overviews?

No. Google's documentation for AI features says there is no special schema.org structured data you need to add and no additional requirements beyond those for Google Search. Structured data can make a page's meaning unambiguous — who wrote it, when, what a passage is — but it does not by itself make a page more likely to be cited.

### Should I remove FAQPage markup now that Google's FAQ rich result is gone?

No. Google's 2023 announcement said there is no need to proactively remove it, and its May 2026 deprecation notice did not change that. FAQ rich results were limited to government and health sites in August 2023 and stopped appearing altogether on 7 May 2026, so the search appearance is gone for every site. The markup still describes question–answer pairs in a machine-readable form, which is useful independently of rich results, provided the answers match the visible text exactly.

### Which is better for a guide: Article or TechArticle?

TechArticle is a schema.org subtype of Article intended for technical and how-to material, so it is the more precise label for documentation and guides. Google's Article documentation lists Article, NewsArticle and BlogPosting as the types it supports for its Article feature. If that search appearance matters to you, emit Article; if precision matters more, emit TechArticle. Both carry the same author, date and citation properties.

### Do ChatGPT, Perplexity or Claude read JSON-LD?

Only Google documents how it treats structured data. OpenAI's, Anthropic's and Perplexity's published crawler documentation covers user agents and robots.txt, not markup. Their crawlers fetch the same HTML that contains your JSON-LD, so the data is available to them, but there is no vendor statement that it influences which pages are cited. Treat it as a low-cost clarity layer, not a guaranteed signal.

### Where should the JSON-LD go on the page?

Inside a script element with type application/ld+json, in either the head or the body. Google's documentation says it can read JSON-LD in either location, including when it is injected by JavaScript, but server-rendering it is safer for crawlers with small render budgets. Escape the less-than character in the serialised JSON so a string value can never close the script tag early.

## Sources

1. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Google Search Central (2025)
2. [Introduction to structured data markup in Google Search](https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data) — Google Search Central
3. [Article (Article, NewsArticle, BlogPosting) structured data](https://developers.google.com/search/docs/appearance/structured-data/article) — Google Search Central
4. [Changes to HowTo and FAQ rich results](https://developers.google.com/search/blog/2023/08/howto-faq-changes) — Google Search Central Blog (2023)
5. [Google Search Central documentation changelog (FAQ rich result deprecation, May 2026)](https://developers.google.com/search/updates) — Google Search Central (2026)
6. [Farewell, Sitelinks Search Box](https://developers.google.com/search/blog/2024/10/sitelinks-search-box) — Google Search Central Blog (2024)
7. [General structured data guidelines](https://developers.google.com/search/docs/appearance/structured-data/sd-policies) — Google Search Central
8. [DefinedTerm](https://schema.org/DefinedTerm) — Schema.org

## Related guides

- [FAQ content that answer engines pick up (and why FAQPage rich results went away)](https://citable.wiki/guides/faq-content-for-answer-engines)
- [E-E-A-T for answer engines: authors, dates and sources that machines can verify](https://citable.wiki/guides/eeat-for-answer-engines)
- [Modelling content for answer engines in a headless CMS!](https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)

---

Source: https://citable.wiki/guides/structured-data-for-ai-answers
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt


<!-- ---------------------------------------------------------------- -->

---
title: "How to write answer-first content that LLMs can quote"
subtitle: "A writing method for pages that survive being chunked: a direct answer up front, one question per heading, and passages that make sense on their own."
canonical: https://citable.wiki/guides/how-to-write-answer-first-content
category: content
author: "Endrit Krasniqi"
date_published: 2026-02-02
date_modified: 2026-02-02
license: CC BY 4.0
---

# How to write answer-first content that LLMs can quote

*A writing method for pages that survive being chunked: a direct answer up front, one question per heading, and passages that make sense on their own.*

## Short answer

Answer-first content opens every page and every section with a self-contained answer of roughly 40–70 words, then explains, qualifies and expands. The first sentence names the main entity, each heading carries exactly one question, and no passage depends on a pronoun or a paragraph elsewhere. Written this way, a passage still makes sense after an answer engine splits the page into chunks and retrieves one of them in isolation.

## Key takeaways

- Open each page and each section with a 40–70 word answer that names the entity and resolves the question on its own.
- Give every heading exactly one question, phrased the way people ask it, and answer it in the first sentence beneath.
- Retrieval systems split pages into passages of roughly a hundred words, so a passage that needs context from elsewhere loses its meaning.
- Replace pronouns and back-references with the entity's name; a lifted paragraph cannot see the paragraph above it.
- Use lists for steps and enumerations, prose for definitions and reasoning, and never put the answer only in an image or table.
- Citations, quotations and statistics measurably raise how often a passage is used in generated answers; keyword stuffing does not.

## What answer-first content is

Answer-first content is writing that puts the complete answer at the start of a page and at the start of each section, then uses the rest of the section to explain, qualify and support it. It is the **inverted pyramid** of journalism applied at the level of the passage rather than the article. Nielsen Norman Group describes the inverted pyramid as a structure in which the most important information is presented first, followed by supporting detail in descending order of importance.

Answer engines make that structure necessary, for two reasons.

First, they do not read pages; they read passages. A [retrieval-augmented](/glossary/retrieval-augmented-generation) system splits a page into chunks and scores each chunk against the question on its own. The Dense Passage Retrieval paper by Karpukhin et al. split Wikipedia into "disjoint text blocks of 100 words" as its basic retrieval unit, and many production systems work at a similar scale: a heading plus a paragraph or two. Whatever sits in the first hundred words after a heading is what the retriever sees.

Second, language models use what they retrieve unevenly. The *Lost in the Middle* study by Liu et al. found that model performance "is often highest when relevant information occurs at the beginning or end of the input context" and degrades in the middle. A passage that states its answer up front is easier to use than one that reaches it at the end.

None of this is a vendor requirement. Google's documentation on AI features says there are "no additional requirements to appear in AI Overviews or AI Mode". The method follows from how retrieval and generation work, described in [What is Answer Engine Optimization](/guides/what-is-answer-engine-optimization).

> **Definition: Answer-first content**
>
> Answer-first content is a page or section whose first paragraph fully answers the question in its heading without relying on anything that comes before or after it. Everything below that paragraph is explanation, evidence and qualification.

## The 40–70 word direct answer

The central unit of this method is the **direct answer**: a paragraph of roughly forty to seventy words that sits directly under the page title, with a shorter one- or two-sentence version under each major heading.

Forty words is about the minimum needed to name the entity, state what it is or does, and add one qualifying detail. Seventy words keeps the whole answer, plus its heading, inside a single hundred-word retrieval chunk. Longer than that and a chunker may split the answer in two, or a model may quote half of it.

A direct answer follows a fixed shape:

1. **Sentence one names the entity and answers the question.** "llms.txt is a proposed convention for a Markdown file at the root of a website that gives language models a curated overview of the site."
2. **Sentence two adds the detail that makes the answer usable.** What it contains, how it works, who it is for.
3. **Sentence three states the scope or the limit.** Who proposed it, what it does not do, when it applies.

There is no preamble. The paragraph does not begin with "In this guide", "Many people ask" or "Before we get into it". Those openings spend words the chunk cannot afford.

The length is easy to enforce in CI or in the CMS:

```ts
const words = (s: string) => s.trim().split(/\s+/).filter(Boolean).length;

const n = words(guide.short_answer);
if (n < 40 || n > 70) {
  throw new Error(`short_answer is ${n} words (want 40–70)`);
}
```

The GEO study by Aggarwal et al. tested nine ways of rewriting a source page and measured how much of each generated answer was drawn from it. Its authors report that their "top-performing methods, Cite Sources, Quotation Addition, and Statistics Addition, achieved a relative improvement of 30-40%" on their position-adjusted word count metric, while keyword stuffing offered "little to no improvement".

**30–40%** — relative visibility gain for source pages that added citations, quotations or statistics, on the GEO study's position-adjusted word count metric (Source: [GEO: Generative Engine Optimization (Aggarwal et al.)](https://arxiv.org/abs/2311.09735))

Put a source, a quotation from a named authority or a concrete figure in the direct answer or in the sentence immediately after it, not three sections down.

## One question per heading

Each H2 and H3 should name exactly one question, and the first sentence under it should answer that question. The heading can be phrased as a question ("How long should the direct answer be?") or as the noun phrase people would type ("Direct answer length"). What matters is that a reader, or a retriever, can tell from the heading alone which question the section resolves.

Two failure modes are common: the clever heading ("Less is more") that tells a retriever nothing about the content beneath it, and the heading that covers several questions ("Length, tone and formatting"), which dilutes the answer to each. Split the second kind into one heading per question.

Inside a section, repeat the inverted pyramid: answer, then reasons, then examples, then edge cases. If a section grows past three or four paragraphs, it usually contains a second question that deserves its own heading.

In Markdown, the pattern looks like this:

```markdown
## How long should a direct answer be?

A direct answer should be 40–70 words: long enough to name the entity,
state the answer and add one qualification, and short enough to fit
inside a single retrieval chunk together with its heading.

Forty words is roughly the floor because ...

### What happens if the answer is longer?

...
```

## Name entities; do not lean on pronouns

Every paragraph should survive being lifted out of the page. That means naming the entity rather than pointing at it with "it", "this", "the tool" or "as described above".

Anthropic's engineering post on contextual retrieval gives the example. A chunk that reads "The company's revenue grew by 3% over the previous quarter" is accurate inside its document and useless outside it: the chunk does not say which company or which quarter, so a retriever cannot match it to a question about either. Anthropic's fix is to prepend a generated sentence of context to every chunk before indexing it. You cannot rely on every answer engine doing that, so write the context in yourself.

The rules that follow:

- Use the full name on first mention in each section, with the abbreviation in parentheses: "Generative Engine Optimization (GEO)". After that, use either form consistently.
- Do not start a section with a pronoun. "It also supports…" at the top of a chunk has no referent.
- Replace "above", "below", "earlier" and "as we saw" with the thing itself: "the 40–70 word rule", not "the rule above".
- Keep the entity name in the heading and in the first sentence beneath it, so that both heading-based and paragraph-based chunkers see it.

## Sentence length, lists and prose

Short declarative sentences are easier to quote than long ones. A sentence a model can lift verbatim states one claim, has one subject, and does not depend on a subordinate clause three lines back. Around twenty words is a comfortable ceiling for a sentence that carries the answer; sentences that explain or qualify can run longer.

Lists and prose do different jobs:

| Content | Form | Reason |
| --- | --- | --- |
| Steps, sequences, options | Numbered or bullet list | Reproduced faithfully by models; easy to scan |
| Definitions, reasoning, trade-offs | Prose | Bullets drop the logical connectives |
| Comparisons across several attributes | Table | Each cell has an explicit subject |
| The direct answer itself | One paragraph of prose | A list cannot be quoted as a single passage |

Two cautions. Make each list item a complete sentence or a parallel fragment, so that an item quoted on its own still reads correctly. And never put the answer *only* in a table, an image or a diagram: Google's AI features documentation asks that "important content is available in textual form", and text is the only form every crawler can read.

## A worked rewrite

Below is a typical blog-post opening, then the same information rewritten answer-first.

> **Example: Before**
>
> So you've been hearing about llms.txt and you're wondering whether your site needs one. It's been getting a lot of attention lately, and for good reason. Before we get into the details, it's worth stepping back and thinking about what it's actually trying to solve. Spoiler: it's simpler than you think. Later in this article we'll look at what goes into it, but first, some background.

That is 66 words without an answer in them. The entity is named once, in the first sentence, and referred to as "it" five times afterwards. Nothing in the passage could be quoted in response to "what is llms.txt".

> **Example: After**
>
> llms.txt is a proposed convention for a Markdown file at the root of a website that gives language models a curated overview of the site: a title, a short summary and links to its important pages. Jeremy Howard of Answer.AI published the proposal at llmstxt.org in September 2024. It is a community convention, not a web standard, and Google says no such file is needed for its AI features.

The rewrite is 69 words. It names the entity in the first sentence, answers the question, and adds two qualifications. The only pronoun, "It", refers to llms.txt inside the same passage, so the chunk stays self-contained.

The same discipline applies deeper in a page:

> **Before:** It also works with the full-text version. This can get large, so you'll want to think about that.
> 
> **After:** The optional companion file, `llms-full.txt`, is a convention for putting the full text of the linked pages in one document. On a large site it grows substantially, so include only the pages worth a model's context window.

## Editing checklist

Run this pass before publishing and after every update.

**Answer-first editing pass**

- [ ] The first paragraph under the title is 40–70 words and names the main entity in its first sentence
- [ ] The first paragraph could be pasted into a blank document and still be true and complete
- [ ] Every H2 and H3 names one question, and the first sentence beneath it answers that question
- [ ] No section starts with a pronoun or with "above", "below", "as mentioned" or "in this guide"
- [ ] Every entity is written out in full on first mention in each section
- [ ] Steps and options are lists; definitions and reasoning are prose; comparisons are tables
- [ ] Nothing important exists only in an image, a diagram or a table cell
- [ ] A source, a quotation from a named authority or a concrete figure appears in or directly after the direct answer

> **Tip: Where to go next**
>
> Answer-first writing is the input; two other guides make it machine-readable. Read [FAQ content for answer engines](/guides/faq-content-for-answer-engines) to turn recurring questions into self-contained question-and-answer blocks, and [Structured data for AI answers](/guides/structured-data-for-ai-answers) to mark the direct answer up with `Article`, `FAQPage` and `speakable`.

## Frequently asked questions

### How long should the direct answer be?

Aim for 40 to 70 words. Forty is about the minimum needed to name the entity, answer the question and add one qualifying detail. Seventy keeps the answer, together with its heading, inside a single retrieval chunk of roughly a hundred words, so an answer engine is unlikely to split it or quote only half of it.

### Should headings be written as questions?

They can be, but they do not have to be. What matters is that each heading names exactly one question and that the first sentence beneath it answers that question. A heading phrased as the noun phrase people search for, such as "Direct answer length", works as well as "How long should the direct answer be?" A heading that covers two questions, or a clever heading that names none, is the thing to avoid.

### Does answer-first writing make pages worse for human readers?

Usually the opposite. Answer-first is the inverted pyramid that journalists and technical writers have used for decades, and usability research recommends it for web reading because most people scan and leave. Readers who want the reasoning keep reading; readers who only wanted the answer get it in the first paragraph instead of the last.

### Is answer-first content the same as optimising for featured snippets?

The two overlap but are not identical. A featured snippet is one passage Google elevates above its results, and Google's documentation says site owners cannot mark a page as a featured snippet. Answer-first writing produces the kind of self-contained passage that featured snippets, AI Overviews, Perplexity and ChatGPT search all look for, so it serves both without targeting either specifically.

### Can I still use lists and tables?

Yes, and you should, for the right content. Steps, options and enumerations belong in lists, because models reproduce them faithfully and readers scan them. Comparisons belong in tables. The direct answer itself should be one paragraph of prose, because a list cannot be quoted as a single passage, and nothing important should exist only in a table cell or an image.

## Sources

1. [GEO: Generative Engine Optimization](https://arxiv.org/abs/2311.09735) — Aggarwal et al., KDD 2024 (arXiv) (2023)
2. [Introducing Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval) — Anthropic (2024)
3. [Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172) — Liu et al. (arXiv) (2023)
4. [Dense Passage Retrieval for Open-Domain Question Answering](https://arxiv.org/abs/2004.04906) — Karpukhin et al., EMNLP 2020 (arXiv) (2020)
5. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Google Search Central (2025)
6. [Inverted Pyramid: Writing for Comprehension](https://www.nngroup.com/articles/inverted-pyramid/) — Nielsen Norman Group (2018)

## Related guides

- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)
- [FAQ content that answer engines pick up (and why FAQPage rich results went away)](https://citable.wiki/guides/faq-content-for-answer-engines)
- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)
- [E-E-A-T for answer engines: authors, dates and sources that machines can verify](https://citable.wiki/guides/eeat-for-answer-engines)

---

Source: https://citable.wiki/guides/how-to-write-answer-first-content
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt


<!-- ---------------------------------------------------------------- -->

---
title: "GEO vs SEO vs AEO: what actually changed"
subtitle: "Where the three terms come from, what they share, and the three things that genuinely differ when the reader is a language model."
canonical: https://citable.wiki/guides/geo-vs-seo-vs-aeo
category: fundamentals
author: "Endrit Krasniqi"
date_published: 2026-01-19
date_modified: 2026-01-19
license: CC BY 4.0
---

# GEO vs SEO vs AEO: what actually changed

*Where the three terms come from, what they share, and the three things that genuinely differ when the reader is a language model.*

## Short answer

Search Engine Optimization (SEO), Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO) describe three stages of the same job. SEO optimises a page to rank in a list. AEO optimises a passage to be extracted as a direct answer. GEO, from a 2023 paper, optimises a source to be cited inside an answer written by a language model. Crawlability and quality carry over; the unit, reward and pipeline change.

## Key takeaways

- SEO, AEO and GEO share one foundation: a crawlable, indexable page with verifiable quality signals. None of the newer terms removes that requirement.
- GEO was named in a November 2023 research paper by Aggarwal et al.; AEO is an older industry term with no single origin.
- The unit of optimisation moved from the page to the passage, and the reward moved from a click to a citation.
- Answer engines run a retrieval pipeline — crawl, chunk, retrieve, ground, generate — and each stage rewards different properties of your content.
- Google, OpenAI, Anthropic and Perplexity all document separate crawlers for search, user requests and model training; allow them separately in robots.txt.
- Prioritise SEO fundamentals first, AEO structure second, and GEO evidence tactics third; the order is fixed, the effort split is not.

## Three names for one shift

Search Engine Optimization, Answer Engine Optimization and Generative Engine Optimization are usually presented as rivals. They are better read as a timeline. Each term was coined when the system on the other end of the query changed, and each inherits almost everything from the one before it.

The short version: **SEO** optimises a page to rank in a list, **AEO** optimises a passage to be extracted as an answer, and **GEO** optimises a source to be cited inside an answer that a language model has written. This guide places each term, separates what carries over from what does not, and ends with a decision guide for where to spend effort.

## Where each term came from

### SEO: the page and the ranking

Search Engine Optimization dates from the mid-1990s and has a stable meaning: make a URL discoverable, indexable and competitive for a query so that it ranks high in a list of results. Its signals are well documented — crawlability, relevance, links, page experience — and Google publishes most of the guidance that defines the field on Search Central. The reader is a human who scans ten links and picks one.

### AEO: the passage and the extraction

Answer Engine Optimization is an industry term, not an academic one, and it has no single origin. SEO practitioners were using it before generative AI to describe the work of winning featured snippets, "People also ask" boxes and voice-assistant answers — all systems that return one passage instead of a list. After ChatGPT's release in late 2022 the term was widely adopted for AI assistants as well. The [glossary entry](/glossary/answer-engine-optimization) gives the working definition this site uses, and [What is Answer Engine Optimization?](/guides/what-is-answer-engine-optimization) walks through the tactics.

### GEO: the source and the citation

Generative Engine Optimization has a precise origin. The paper *GEO: Generative Engine Optimization* by Pranjal Aggarwal, Vishvak Murahari and colleagues was posted to [arXiv on 16 November 2023](https://arxiv.org/abs/2311.09735) and later presented at KDD 2024. It defines a **generative engine** as a system that synthesises an answer from multiple retrieved sources with a large language model, proposes a benchmark called GEO-bench, and measures which edits to a source increase its share of the generated answer.

**Up to 40%** — increase in a source's visibility in generative engine responses from the best-performing content edits, as reported in the paper's abstract (Source: [GEO: Generative Engine Optimization (Aggarwal et al., 2023)](https://arxiv.org/abs/2311.09735))

Its most repeated finding is that edits which added citations, quotations from authorities and statistics improved visibility, while stuffing in extra keywords did not help. That result is the practical dividing line between GEO and SEO: evidence outperforms keyword coverage when the reader is a model.

> **Definition: Working definitions**
>
> - **SEO** — optimising a page to rank in a list of search results for a query.
> - **AEO** — optimising a passage so that any system which returns a direct answer can extract and use it.
> - **GEO** — optimising a source so that a language model, writing an answer from retrieved sources, uses and cites it.

## What stays the same

The newer terms attract a lot of claims that the fundamentals no longer apply. The vendors say otherwise.

Google's documentation on AI features states that "there are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary", and that no new machine-readable files or special schema.org types are needed. The same Googlebot that indexes for Search feeds AI Overviews and AI Mode, and the same `noindex`, `nosnippet`, `data-nosnippet` and `max-snippet` controls limit what those features can show.

Three things therefore carry over unchanged:

1. **Crawlability and indexing.** A page that cannot be fetched, or that is blocked for the relevant user agent, cannot be retrieved. OpenAI, Anthropic and Perplexity each publish the crawler names their search features use, and each says those crawlers respect `robots.txt`.
2. **Quality and E-E-A-T.** Google's guidance on helpful content says E-E-A-T "isn't a specific ranking factor", but that its systems use a mix of signals to identify experience, expertise, authoritativeness and trust. Answer engines lean on the subset of those signals a machine can check without reading for quality: a named author, visible dates, and links to primary sources.
3. **Semantic HTML and clean rendering.** Server-rendered HTML with a logical heading outline was good practice for Googlebot and is now the difference between a passage that chunks cleanly and one that does not.

If a page does not rank at all, AEO and GEO tactics have nothing to work with. That is the single most useful thing to know about the three terms.

## What actually changed

Three things did change, and they are structural rather than cosmetic.

### The unit is the passage

Classic ranking scores a URL. An [answer engine](/glossary/answer-engine) splits the page into chunks — usually by heading and paragraph, sometimes by a fixed token window — and scores each chunk against the question. A passage that opens mid-argument, or that refers to "it" without naming the entity, loses at this stage regardless of how strong the page is. Every chunk has to survive on its own.

### The reward is the citation

A ranking produces a click. A generative answer produces a **citation**: a footnote or inline link to the source the model used. The citation may or may not produce a click, and some engines send far less referral traffic than others. That changes what you measure — share of citations for a set of questions, not only sessions from organic search — and it changes what you write for, because the model is choosing which passage to attribute, not which page to list.

### The pipeline is retrieval, not ranking

Answer engines are built on [retrieval-augmented generation](/glossary/retrieval-augmented-generation): fetch candidate passages, ground the model in them, generate, attribute. Each stage rewards something different — being fetchable, chunking cleanly, matching the question's wording and entities, containing verifiable evidence, and being the answer rather than discussing it. SEO tactics map onto the first stage only; AEO and GEO tactics map onto the rest.

The pipeline also changed the crawler landscape. OpenAI's crawler documentation lists `OAI-SearchBot` for ChatGPT's search features, `ChatGPT-User` for user-initiated fetches and `GPTBot` for model training. Anthropic's support documentation lists `Claude-SearchBot`, `Claude-User` and `ClaudeBot` with the same three roles, and Perplexity documents `PerplexityBot` for search and `Perplexity-User` for user actions. Google uses Googlebot for AI features; `Google-Extended` is a separate robots.txt token that limits AI training and grounding in some of its other systems. Being cited and being trained on are now separate decisions, and `robots.txt` can express them separately:

```text
# Allow search and user-initiated fetches; decline model training.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
```

Two caveats from the vendor pages. OpenAI and Perplexity both note that their user-initiated fetchers (`ChatGPT-User`, `Perplexity-User`) act on behalf of a person and may not apply `robots.txt` rules. And `Google-Extended` does not affect AI Overviews or AI Mode; those follow the ordinary Googlebot rules.

## Side by side

| Aspect | SEO | AEO | GEO |
| --- | --- | --- | --- |
| Origin | Industry, mid-1990s | Industry, no single origin; pre-dates generative AI | Aggarwal et al., arXiv, November 2023 |
| Target system | Ranked list of links | Anything that returns one direct answer: snippets, voice, AI chat | Engines that generate an answer with an LLM and cite sources |
| Unit optimised | Page (URL) | Passage | Source, as used within a generated answer |
| Reward | Ranking position and click | Extraction as the answer | Citation and share of the answer |
| Core signals | Crawlability, relevance, links, page experience | Answer-first structure, one question per URL, entity clarity, structured data | Citations, quotations, statistics, verifiable authorship |
| Measured by | Rank, impressions, clicks | Snippet and answer ownership | Citation frequency across a question set |
| Still needs | — | Everything SEO needs | Everything SEO and AEO need |

## Which one to prioritise

The terms stack, so the order is fixed even though the effort split is not. Work through them in sequence and stop at the first gap.

1. **Start with SEO if the page is not indexed or does not rank for its core query.** No answer engine will retrieve a passage from a page it cannot fetch or does not trust. Fix crawl access, rendering, canonicals and thin content first.
2. **Move to AEO once the page ranks but is paraphrased rather than quoted.** Put a 40–70 word direct answer at the top, give each question its own URL and heading, name entities in full, and emit `Article`, `FAQPage` and `Person` markup from the same fields as the visible text.
3. **Move to GEO once passages are extracted but rarely cited.** Add what the GEO paper found generative engines reward: outbound citations to primary sources, quotations from named authorities and concrete statistics with a source. Check crawler access for the search-mode user agents above.
4. **Measure at every stage in the unit that stage rewards.** Rank for SEO, snippet or answer ownership for AEO, and citation share across a fixed question set for GEO.

**Decision guide: which term is your bottleneck**

- [ ] The page is not indexed, or is blocked for Googlebot, OAI-SearchBot, Claude-SearchBot or PerplexityBot: SEO and crawler access
- [ ] The page ranks, but no section answers a specific question in its first 70 words: AEO
- [ ] Sections answer questions, but headings do not match how people ask them: AEO
- [ ] Passages appear in AI answers paraphrased but uncredited: GEO
- [ ] Passages are cited by one engine but not others: check per-engine crawler rules, then GEO
- [ ] Author, dates and sources are missing or not marked up: E-E-A-T, which all three share

The names matter less than the sequence. A team that ships crawlable pages, answer-first passages and cited evidence is doing SEO, AEO and GEO at once, whatever it calls the work.

> **Tip: Where to go next**
>
> [How to write answer-first content](/guides/how-to-write-answer-first-content) covers the AEO layer — the passage structure that survives chunking. [Controlling AI crawlers with robots.txt](/guides/controlling-ai-crawlers-with-robots-txt) covers the access layer, including the per-engine split between search, user and training crawlers.

## Frequently asked questions

### Is GEO just a new name for SEO?

No, but it depends on the same foundation. SEO optimises a page to rank in a list of links. GEO optimises a source so that a generative engine uses and cites it when writing an answer. The crawlability, indexing and quality work is shared; what differs is that GEO rewards self-contained passages with citations, quotations and statistics rather than keyword coverage and link equity.

### Where does the term GEO come from?

From the paper GEO: Generative Engine Optimization by Pranjal Aggarwal and colleagues, first posted to arXiv on 16 November 2023 and later presented at KDD 2024. It defined generative engines as systems that synthesise an answer from multiple sources with a language model, introduced the GEO-bench benchmark, and reported that some content edits raised a source's visibility by up to 40%.

### What is the difference between AEO and GEO?

AEO is the broader, older term: optimising for any system that returns a direct answer, including featured snippets, voice assistants and AI chat. GEO is narrower and academic: optimising for engines that generate the answer with a language model and cite sources. In practice the tactics overlap almost completely, and most teams use whichever term their stakeholders recognise.

### Should I stop doing SEO and switch to AEO or GEO?

No. Every answer engine that cites the web discovers pages through crawling and filters them with quality signals that classic SEO already covers. Google states there are no additional requirements to appear in AI Overviews or AI Mode. Treat AEO and GEO as a layer added to a page that already ranks, not as a replacement for the work that got it there.

### Does E-E-A-T still matter for answer engines?

Yes, and arguably more. Google's helpful-content documentation says E-E-A-T is not itself a ranking factor, but that its systems look for signals of experience, expertise, authoritativeness and trust. A retrieval system cannot judge quality by reading, so it leans on what it can verify: a named author, visible dates, and outbound links to primary sources.

## Sources

1. [GEO: Generative Engine Optimization](https://arxiv.org/abs/2311.09735) — Aggarwal et al., KDD 2024 (arXiv) (2023)
2. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Google Search Central (2025)
3. [Creating helpful, reliable, people-first content](https://developers.google.com/search/docs/fundamentals/creating-helpful-content) — Google Search Central (2025)
4. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — OpenAI (2025)
5. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — Anthropic (2025)
6. [Perplexity crawlers](https://docs.perplexity.ai/guides/bots) — Perplexity (2025)

## Related guides

- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)
- [How to write answer-first content that LLMs can quote](https://citable.wiki/guides/how-to-write-answer-first-content)
- [Controlling AI crawlers with robots.txt: GPTBot, ClaudeBot, PerplexityBot and friends](https://citable.wiki/guides/controlling-ai-crawlers-with-robots-txt)
- [E-E-A-T for answer engines: authors, dates and sources that machines can verify](https://citable.wiki/guides/eeat-for-answer-engines)

---

Source: https://citable.wiki/guides/geo-vs-seo-vs-aeo
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt


<!-- ---------------------------------------------------------------- -->

---
title: "What is Answer Engine Optimization (AEO)?"
subtitle: "How answer engines choose what to quote, and what that changes about the way you write and structure pages."
canonical: https://citable.wiki/guides/what-is-answer-engine-optimization
category: fundamentals
author: "Endrit Krasniqi"
date_published: 2026-01-12
date_modified: 2026-08-20
license: CC BY 4.0
---

# What is Answer Engine Optimization (AEO)?

*How answer engines choose what to quote, and what that changes about the way you write and structure pages.*

## Short answer

Answer Engine Optimization (AEO) is the practice of structuring content so that AI-powered answer engines — ChatGPT, Perplexity, Claude, Copilot and Google's AI Overviews — can retrieve it, understand it and cite it inside a generated answer. Where classic SEO competes for a ranking position, AEO competes for extraction: a self-contained passage that resolves one question, from a page with a clear author, date and sources.

## Key takeaways

- Answer engines retrieve passages, not pages: they look for a chunk of text that fully answers the question on its own.
- AEO overlaps with SEO on crawlability and quality, but adds answer-first structure, entity clarity and machine-readable formats.
- Citations, quotations and concrete statistics measurably increase how often a source is used in generated answers.
- Author, date and sources are the quality signals a retrieval system can verify without reading the whole page.
- Serve the same content as clean HTML and as Markdown; what is cheap to parse gets parsed.

## Why a new acronym was needed

For twenty years the unit of search was the page, and the prize was a position in a list. You optimised a URL to rank, and a human decided whether to click it.

Answer engines changed the unit. When someone asks Perplexity, ChatGPT search, Claude or Google's AI Overviews a question, the system does not hand back ten links. It **retrieves a handful of passages**, judges which of them actually resolve the question, writes an answer, and attaches citations to the passages it used. The page is no longer the thing being ranked — the passage is. And the reward is no longer a click but a citation, which may or may not produce a click.

*Answer Engine Optimization* is the name for adapting to that unit. It asks a narrower question than SEO: **is there a passage on this page that a machine could lift out and use, on its own, to answer a specific question — and can it verify that the passage is trustworthy?**

> **Definition: Answer engine**
>
> An answer engine is any system that responds to a query with a synthesised answer rather than a list of results. Today that includes Google AI Overviews and AI Mode, Perplexity, ChatGPT search, Claude with web search, Microsoft Copilot and voice assistants. Most of them are built on retrieval-augmented generation (RAG): fetch candidate passages, then generate with a language model.

## How an answer engine decides what to cite

The details differ between systems, but the pipeline is broadly the same, and each stage rewards different properties of a page.

1. **Discovery and crawling.** The engine has to know the page exists and be allowed to fetch it. This is classic technical SEO: a crawlable URL, a sitemap, no `noindex`, and a `robots.txt` that does not block the engine's user agent. OpenAI, Anthropic and Perplexity each publish their crawler names; Google uses Googlebot for AI features and `Google-Extended` only as a training opt-out.
2. **Chunking.** The page is split into passages — typically by heading and paragraph, sometimes by a fixed token window. A passage that starts mid-thought, or that depends on a definition three sections earlier, loses most of its value here.
3. **Retrieval.** For a given question, the engine scores passages by relevance, usually with a mix of embeddings and keyword signals. Passages whose wording resembles the question, and that contain the entities in the question, score higher.
4. **Selection and grounding.** From the top candidates, the model picks the ones it will actually use. Research on generative engines found that passages containing **citations, quotations from authorities, and concrete statistics** were used substantially more often than the same content without them — up to 40% more visibility in the Princeton/IIT Delhi GEO study.
5. **Generation and attribution.** The model writes the answer and attaches sources. A passage that *is* the answer tends to be cited; a passage that merely *discusses* the answer tends to be paraphrased without attribution.

Every AEO tactic maps onto one of those stages.

## What AEO actually changes

### Write the answer first

The single highest-leverage change is structural: open every page, and every major section, with a self-contained answer. Forty to seventy words, in plain declarative sentences, with the key entity named explicitly rather than referred to as "it". Then explain, qualify and expand.

This is the opposite of the "hook, context, reveal" structure that human editors like, and it is the reason AEO sometimes feels like writing an encyclopedia rather than a magazine. The compromise that works: a short-answer block at the top (this page's *Short answer* box), an ordinary article underneath, and a heading for every question you expect someone to ask.

### One question per URL

Answer engines match questions to passages, and passages inherit trust from their page. A page that answers "what is AEO" *and* "how do I measure AEO" *and* "what tools exist for AEO" has diluted signals for all three. Give each question its own URL, title and canonical, and link them.

### Make entities unambiguous

Language models resolve entities — people, products, concepts — by name. Use the full name on first mention, add the abbreviation in parentheses, and be consistent afterwards. A glossary with one page per term, marked up as `DefinedTerm`, is one of the cheapest ways to make a whole site easier to understand.

### Show who, when and why

Retrieval systems cannot read your page for quality the way an editor can, so they lean on signals they *can* check: a named author with a profile, visible published and updated dates, and outbound links to primary sources. Google's own documentation for AI features points at the same E-E-A-T guidance it uses for search.

### Mark it up

Structured data does not make a bad passage good, but it removes ambiguity about what a passage *is*. `Article` (with `author`, `datePublished`, `dateModified`, `citation`), `FAQPage`, `DefinedTerm`, `Person` and `SpeakableSpecification` are the types that matter for answers. Generate them from your CMS fields so the markup can never disagree with the visible text.

### Ship a machine-readable version

Rendering a page with JavaScript, then expecting a crawler with a small fetch budget to execute it, is a bet against yourself. Pre-render HTML on the server, keep the DOM semantic, and offer the same content as Markdown — a `.md` suffix, or content negotiation on `Accept: text/markdown`. An `/llms.txt` file gives models a curated index of what you have.

**The minimum viable AEO page**

- [ ] A 40–70 word direct answer at the top, with the main entity named in the first sentence
- [ ] One question per URL, with the question in the title and H1
- [ ] A named author, a visible published date and an updated date
- [ ] Outbound links to primary sources, listed at the end
- [ ] Article + FAQPage + Person JSON-LD generated from the same fields as the visible text
- [ ] Server-rendered HTML with a logical heading outline
- [ ] A Markdown representation and an entry in /llms.txt
- [ ] robots.txt that allows the crawlers you want to be cited by

## What AEO does not change

It is worth being clear about the limits, because the acronym attracts hype.

- **It does not bypass quality.** Every engine that cites sources still filters on the same trust signals as search. Thin, derivative pages with perfect markup are still thin, derivative pages.
- **It does not guarantee traffic.** A citation is a mention, not a visit. Some answer engines send meaningful referral traffic, others almost none. Measure both.
- **It is not a trick.** The tactics above are the same things that make a page easier for a hurried human to use: say the answer, say who wrote it, show your sources.

> **Tip: Where to go next**
>
> If you are new to this, read *GEO vs SEO vs AEO* to place the terms, then *How to write answer-first content* for the writing method. If you already run a site, start with the technical guides on `llms.txt`, `robots.txt` and Markdown delivery — they can be shipped in an afternoon.

## Frequently asked questions

### Is AEO the same as GEO?

They overlap heavily. AEO usually describes optimising for any system that returns a direct answer, including featured snippets and voice assistants. GEO (Generative Engine Optimization) specifically targets systems that generate an answer with a language model and cite sources. In practice the tactics are the same: answer-first passages, clear entities, sources, structured data and machine-readable delivery.

### Does AEO replace SEO?

No. Answer engines still discover pages through crawling and still use quality signals that classic SEO covers. AEO is a layer on top: once a page can be crawled and trusted, AEO makes its passages extractable and quotable.

### How long does it take to see results from AEO?

Systems with live retrieval, such as Perplexity or ChatGPT search, can pick up a new page within days of it being crawled. Model training data updates on a much slower cycle, so being cited for a topic in general takes months. Track both with the methods in the measurement guides.

### Do I need structured data for AEO?

It is not mandatory, but it removes ambiguity. Article, FAQPage, DefinedTerm and Person markup tell a crawler what a passage is — a definition, an answer, an author — rather than leaving it to inference. This site emits JSON-LD on every page for exactly that reason.

## Sources

1. [GEO: Generative Engine Optimization](https://arxiv.org/abs/2311.09735) — Aggarwal et al., KDD 2024 (arXiv) (2023)
2. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Google Search Central (2025)
3. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — OpenAI (2025)
4. [The /llms.txt file](https://llmstxt.org/) — Jeremy Howard, Answer.AI (2024)

## Related guides

- [GEO vs SEO vs AEO: what actually changed](https://citable.wiki/guides/geo-vs-seo-vs-aeo)
- [How to write answer-first content that LLMs can quote](https://citable.wiki/guides/how-to-write-answer-first-content)
- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)

---

Source: https://citable.wiki/guides/what-is-answer-engine-optimization
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt


<!-- ---------------------------------------------------------------- -->

---
term: "AI crawler"
canonical: https://citable.wiki/glossary/ai-crawler
also_known_as: "AI bot, LLM crawler"
license: CC BY 4.0
---

# AI crawler

**Definition:** An AI crawler is an automated agent that fetches web pages on behalf of an AI system and identifies itself with its user-agent string, such as GPTBot, ClaudeBot or PerplexityBot. Vendors separate crawlers by purpose — collecting training data, building a search index, or fetching a page a user asked about — so that training and search access can be allowed or blocked independently in robots.txt.

The first AI crawlers, such as Common Crawl's `CCBot` and OpenAI's `GPTBot`, existed to gather training data. As answer engines moved to live retrieval, vendors split their traffic into three roles, each with its own user agent:

| Role | OpenAI | Anthropic | Perplexity |
| --- | --- | --- | --- |
| Training data | `GPTBot` | `ClaudeBot` | — |
| Search index | `OAI-SearchBot` | `Claude-SearchBot` | `PerplexityBot` |
| User-triggered fetch | `ChatGPT-User` | `Claude-User` | `Perplexity-User` |

Training and search crawlers honour `robots.txt`. User-triggered fetchers are different: [OpenAI](https://developers.openai.com/api/docs/bots) and [Perplexity](https://docs.perplexity.ai/guides/bots) both state that robots.txt rules may not apply when a person initiated the request, while [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) says `Claude-User` does respect them.

Google is the exception to the pattern. AI Overviews and AI Mode are served from the ordinary Googlebot index, and `Google-Extended` is a robots.txt token with no user agent of its own; it controls whether crawled content is used for Gemini training and grounding, and does not affect inclusion in Search ([Google crawler documentation](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)).

The decision for a publisher is therefore which roles to allow. A common compromise is to block training crawlers and allow search crawlers, which keeps pages citable without contributing to model weights:

```text
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
```

Blocking everything makes citation impossible. [Controlling AI crawlers with robots.txt](/guides/controlling-ai-crawlers-with-robots-txt) lists the current user agents and their documented behaviour.

## Frequently asked questions

### Does blocking AI crawlers stop my content appearing in AI answers?

Only for the crawlers you block. Blocking GPTBot keeps pages out of OpenAI's training data but not out of ChatGPT search, which uses OAI-SearchBot. Google's AI Overviews are built from Googlebot's index, and the Google-Extended token only controls training and grounding in other Google systems, not inclusion in Search.

## Related guides

- [Controlling AI crawlers with robots.txt: GPTBot, ClaudeBot, PerplexityBot and friends](https://citable.wiki/guides/controlling-ai-crawlers-with-robots-txt)
- [Serving Markdown to LLMs: content negotiation and .md endpoints](https://citable.wiki/guides/serving-markdown-to-llms)

---

Source: https://citable.wiki/glossary/ai-crawler


<!-- ---------------------------------------------------------------- -->

---
term: "AI Overviews"
canonical: https://citable.wiki/glossary/ai-overviews
also_known_as: "Google AI Overviews, formerly Search Generative Experience (SGE)"
license: CC BY 4.0
---

# AI Overviews

**Definition:** AI Overviews is a Google Search feature that shows an AI-generated summary at the top of some results pages, with links to the web pages the summary drew on. It launched to all US users in May 2024 after a year as the Search Generative Experience (SGE) experiment in Search Labs, and is fed by the same crawling and indexing as ordinary Search.

AI Overviews grew out of the **Search Generative Experience (SGE)**, which Google opened as a Search Labs experiment in the United States on 10 May 2023. At Google I/O in May 2024 the feature was renamed AI Overviews and [rolled out to everyone in the US](https://blog.google/products/search/generative-ai-google-search-may-2024/), with other countries following. **AI Mode**, a separate conversational tab introduced in 2025, uses the same underlying approach.

Google's [documentation for AI features](https://developers.google.com/search/docs/appearance/ai-features) describes both as using "query fan-out": the system issues several related searches, retrieves passages from the results, and composes a summary that links to the pages it used. There is no separate index and no separate crawler. Googlebot fetches the page, and the usual indexing and snippet rules decide what may be shown.

That has two practical consequences. First, the ordinary AEO work — an answer-first passage per question, clear entities, a visible author and date — is the whole optimisation; there is nothing extra to submit. Second, control is coarse. `Google-Extended` governs model training, not Search, so the only levers are `nosnippet`, `max-snippet`, `data-nosnippet` and `noindex`, which also change how the page appears in classic results. See [controlling AI crawlers with robots.txt](/guides/controlling-ai-crawlers-with-robots-txt) for the trade-offs, and [zero-click search](/glossary/zero-click-search) for why a citation in an overview may not produce a visit.

## Frequently asked questions

### Do I need special markup to appear in AI Overviews?

No. Google's documentation states that there are no additional requirements or special optimisations for AI Overviews or AI Mode, and no need for new machine-readable files or markup. A page has to be indexed and eligible for a snippet; the same content-quality guidance as ordinary Search applies.

### Can I block AI Overviews without leaving Google Search?

Only partially. Google-Extended is a training opt-out for Gemini models and does not change how a page appears in Search. To limit what AI Overviews shows from a page, Google points to the standard preview controls — nosnippet, max-snippet, data-nosnippet — or noindex, all of which also affect ordinary results.

## Related guides

- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)
- [Controlling AI crawlers with robots.txt: GPTBot, ClaudeBot, PerplexityBot and friends](https://citable.wiki/guides/controlling-ai-crawlers-with-robots-txt)
- [Measuring AI visibility: how to track citations in ChatGPT, Perplexity and AI Overviews](https://citable.wiki/guides/measuring-ai-visibility-and-citations)

---

Source: https://citable.wiki/glossary/ai-overviews


<!-- ---------------------------------------------------------------- -->

---
term: "Answer engine"
canonical: https://citable.wiki/glossary/answer-engine
license: CC BY 4.0
---

# Answer engine

**Definition:** An answer engine is a system that responds to a question with a synthesised answer rather than a ranked list of links. Modern answer engines — Google AI Overviews and AI Mode, Perplexity, ChatGPT search, Claude with web search and Microsoft Copilot — retrieve passages from the web and generate the answer with a large language model, usually with citations to the sources used.

The phrase predates large language models: featured snippets and voice assistants were the first answer engines, returning a single passage instead of a list. What changed after 2022 is that the answer is now *generated* — composed from several passages — and that the system attaches citations, which makes "being cited" a measurable outcome.

## Related guides

- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)

---

Source: https://citable.wiki/glossary/answer-engine


<!-- ---------------------------------------------------------------- -->

---
term: "Answer Engine Optimization"
canonical: https://citable.wiki/glossary/answer-engine-optimization
also_known_as: "AEO"
license: CC BY 4.0
---

# Answer Engine Optimization

**Definition:** Answer Engine Optimization (AEO) is the practice of structuring web content so that systems which return a direct answer — AI assistants, AI-powered search and voice assistants — can retrieve, understand and cite it. It prioritises self-contained answer passages, clear entities, verifiable authorship and machine-readable formats over ranking position.

AEO treats the **passage**, not the page, as the unit of optimisation. An answer engine splits pages into chunks, scores the chunks against a question, and quotes or cites the ones that resolve it on their own.

The practical consequences are a small number of structural habits: put a 40–70 word direct answer at the top of every page; give each question its own URL; name entities explicitly; show a named author, dates and sources; emit schema.org markup from the same fields as the visible text; and serve the content as clean HTML and Markdown.

## Frequently asked questions

### Who coined the term Answer Engine Optimization?

The term has no single origin. It emerged in the SEO industry around 2017–2019 to describe optimising for featured snippets and voice assistants, and was widely adopted for AI assistants after ChatGPT's release in late 2022.

## Related guides

- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)
- [GEO vs SEO vs AEO: what actually changed](https://citable.wiki/guides/geo-vs-seo-vs-aeo)

---

Source: https://citable.wiki/glossary/answer-engine-optimization


<!-- ---------------------------------------------------------------- -->

---
term: "Canonical URL"
canonical: https://citable.wiki/glossary/canonical-url
also_known_as: "rel=canonical, canonical link"
license: CC BY 4.0
---

# Canonical URL

**Definition:** A canonical URL is the single address a site declares as the authoritative version of a page when the same content is reachable at several URLs — with and without a trailing slash, with tracking parameters, as a Markdown or syndicated copy. It is declared with a link element whose rel attribute is canonical, or the equivalent HTTP Link header, and tells crawlers where to consolidate signals and attribution.

Google, Yahoo and Microsoft announced joint support for `rel="canonical"` in February 2009, and the link relation was later written up as RFC 6596 (April 2012), an informational RFC rather than a standards-track one. Google's documentation lists four ways to indicate a preferred URL — a redirect, a `rel="canonical"` link element, the same relation in an HTTP `Link` header, and inclusion in a sitemap — and treats all of them as hints it combines rather than orders it obeys.

Answer engines make the canonical question sharper than classic search did, because the unit of reward is a [citation](/glossary/citation) attached to a specific URL. A guide that exists as `/guides/x`, `/guides/x/`, `/guides/x?utm_source=newsletter` and `/guides/x.md` can be retrieved and cited under any of them. Each variant then shows up separately in whatever you use to track citations, and a syndicated copy on another domain can be cited instead of the original.

The fix is the same as for search, applied to every representation:

```http
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Link: <https://example.com/guides/serving-markdown-to-llms>; rel="canonical"
```

Whether each AI crawler honours canonical declarations when choosing which URL to cite is not documented by any vendor, so a consistent canonical is necessary rather than sufficient. Pair it with a single indexable HTML URL per question and with the delivery pattern in [Serving Markdown to LLMs](/guides/serving-markdown-to-llms).

## Frequently asked questions

### Should the Markdown version of a page have its own canonical URL?

No. Point it at the HTML page. A Markdown file has no head element, so send the declaration as an HTTP header: Link: <https://example.com/guides/x>; rel="canonical". The Markdown copy then contributes to the HTML page's signals instead of competing with it, and any citation lands on the URL you want to measure.

### Is rel=canonical a directive?

Google treats it as a strong hint, not a command: it weighs the declaration alongside redirects, sitemap inclusion and internal links, and can pick a different canonical if the signals disagree. A permanent redirect is the only mechanism that removes the duplicate rather than merely labelling it.

## Related guides

- [Serving Markdown to LLMs: content negotiation and .md endpoints](https://citable.wiki/guides/serving-markdown-to-llms)
- [Measuring AI visibility: how to track citations in ChatGPT, Perplexity and AI Overviews](https://citable.wiki/guides/measuring-ai-visibility-and-citations)

---

Source: https://citable.wiki/glossary/canonical-url


<!-- ---------------------------------------------------------------- -->

---
term: "Chunking"
canonical: https://citable.wiki/glossary/chunking
also_known_as: "Text splitting, passage segmentation"
license: CC BY 4.0
---

# Chunking

**Definition:** Chunking is the step in a retrieval pipeline that splits a page or document into smaller passages — typically a few hundred tokens each, cut at headings, paragraphs or a fixed window — so that each passage can be embedded, indexed and retrieved on its own. Because answer engines retrieve and cite chunks rather than whole pages, chunk boundaries decide what a model actually sees.

You do not control how an answer engine chunks your page, but you control how well the page survives it. Common strategies are fixed-size windows (a few hundred tokens, often with some overlap), recursive splitting on paragraph and then sentence boundaries, and structure-aware splitting on Markdown or HTML headings. Structure-aware splitters are the friendliest, and they are what a clean heading outline, [semantic HTML](/glossary/semantic-html) and a [Markdown version of the page](/guides/serving-markdown-to-llms) make possible.

The failure mode is context loss. Anthropic's [contextual retrieval](https://www.anthropic.com/news/contextual-retrieval) write-up from September 2024 gives the canonical example: a chunk reading "The company's revenue grew by 3% over the previous quarter" is useless on its own, because it does not say which company or which quarter. Their fix — prepending a short generated note on where the chunk sits in the document — cut top-20 retrieval failures by 49% when combined with BM25. You can get much of the same effect at source by writing each section so that it names its own subject.

**Writing for the chunker**

- [ ] One question per H2, phrased as the question
- [ ] The first sentence of each section names the entity and states the answer
- [ ] No "as mentioned above" or unresolved "it" at the start of a section
- [ ] Tables and code blocks short enough to sit inside one chunk with their explanation

## Frequently asked questions

### How long should a section be so that it survives chunking?

Aim for sections that make sense on their own at a few hundred words and open with a self-contained answer. Chunkers differ, but nearly all respect headings and paragraph breaks, so a heading that states the question followed by a 40–70 word direct answer will usually land intact in one chunk.

## Related guides

- [How to write answer-first content that LLMs can quote](https://citable.wiki/guides/how-to-write-answer-first-content)
- [Serving Markdown to LLMs: content negotiation and .md endpoints](https://citable.wiki/guides/serving-markdown-to-llms)
- [Modelling content for answer engines in a headless CMS!](https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms)

---

Source: https://citable.wiki/glossary/chunking


<!-- ---------------------------------------------------------------- -->

---
term: "Citation"
canonical: https://citable.wiki/glossary/citation
also_known_as: "Source attribution, reference"
license: CC BY 4.0
---

# Citation

**Definition:** A citation, in the context of answer engines, is the explicit link or reference that a generated answer attaches to the source passage it drew on — the numbered footnote in Perplexity, the linked sources beside a Google AI Overview, or the source list under a ChatGPT search answer. Citations are the unit of visibility that Answer Engine Optimization is measured by.

The word carries two meanings on this site, and both matter.

**Citations a page receives.** When an answer engine uses a passage, it usually attaches a link back to the page. Whether a passage is cited rather than silently paraphrased depends largely on whether it *is* the answer: a self-contained sentence or two that resolves the question, from a page with a visible author and date. Counting those links, per engine and per query set, is the core of [measuring AI visibility](/guides/measuring-ai-visibility-and-citations).

**Citations a page makes.** Outbound references to primary sources are one of the few quality signals a machine can verify cheaply. The [GEO paper](https://arxiv.org/abs/2311.09735) reported that adding citations, quotations and statistics to content improved its visibility in generative engine responses by up to 40%, while keyword-oriented edits did not help.

Both can be made explicit in markup. Schema.org defines a [`citation`](https://schema.org/citation) property on `CreativeWork` — a reference to another creative work such as a publication or web page — and this site fills it from each guide's sources list:

```json
{
  "@type": "TechArticle",
  "citation": [
    {
      "@type": "CreativeWork",
      "name": "GEO: Generative Engine Optimization",
      "url": "https://arxiv.org/abs/2311.09735"
    }
  ]
}
```

## Frequently asked questions

### Does citing sources on my own page make it more likely to be cited?

The evidence says yes. The GEO study by Aggarwal et al. found that adding citations, quotations from authorities and statistics to a page increased its visibility in generated answers by up to 40% in their metric, whereas adding keywords did not. Outbound citations are also a trust signal a retrieval system can check without reading the whole page.

## Related guides

- [Measuring AI visibility: how to track citations in ChatGPT, Perplexity and AI Overviews](https://citable.wiki/guides/measuring-ai-visibility-and-citations)
- [E-E-A-T for answer engines: authors, dates and sources that machines can verify](https://citable.wiki/guides/eeat-for-answer-engines)
- [How to write answer-first content that LLMs can quote](https://citable.wiki/guides/how-to-write-answer-first-content)

---

Source: https://citable.wiki/glossary/citation


<!-- ---------------------------------------------------------------- -->

---
term: "Content negotiation"
canonical: https://citable.wiki/glossary/content-negotiation
also_known_as: "Proactive negotiation, server-driven negotiation"
license: CC BY 4.0
---

# Content negotiation

**Definition:** Content negotiation is the HTTP mechanism by which a client states which representations it prefers — using headers such as Accept, Accept-Language and Accept-Encoding — and the server picks the best available one for the same URL. For answer engines it lets one canonical URL return HTML to browsers and Markdown to a crawler that sends Accept: text/markdown.

HTTP defines content negotiation in [RFC 9110, section 12](https://www.rfc-editor.org/rfc/rfc9110#section-12). The common form is **proactive** (server-driven): the client sends `Accept` headers, optionally weighted with `q=` values; the server chooses a representation and names the headers it considered in `Vary`. If nothing acceptable exists it may answer `406 Not Acceptable` or fall back to a default. Compression (`Accept-Encoding`) and language (`Accept-Language`) have been negotiated this way for decades, so every cache and proxy already understands the mechanism.

The `text/markdown` media type has been registered since 2016 ([RFC 7763](https://www.rfc-editor.org/rfc/rfc7763)), which makes Markdown a legitimate representation to ask for.

An answer engine wants the text, not the chrome. Serving Markdown for `Accept: text/markdown` gives a crawler a compact representation of the same canonical URL with no navigation, scripts or consent banners to strip. This site does exactly that for every guide and glossary page, and also exposes the same Markdown at a `.md` suffix:

```bash
curl -H "Accept: text/markdown" https://your-site.example/glossary/content-negotiation
```

The one operational hazard is caching. Set `Vary: Accept` on both the HTML and the Markdown response, or a shared cache may hand Markdown to a browser. [Serving Markdown to LLMs](/guides/serving-markdown-to-llms) covers the implementation details.

## Frequently asked questions

### Should I use content negotiation or a .md URL to serve Markdown?

Both, if you can. A .md suffix is visible, linkable and easy to test in a browser; content negotiation keeps a single canonical URL. Whichever route serves Markdown, send Vary: Accept so caches keep the two representations apart, and point back to the HTML page with a canonical link header.

## Related guides

- [Serving Markdown to LLMs: content negotiation and .md endpoints](https://citable.wiki/guides/serving-markdown-to-llms)
- [llms.txt: what it is, how to write one, and whether it helps](https://citable.wiki/guides/llms-txt-what-it-is-and-how-to-write-one)

---

Source: https://citable.wiki/glossary/content-negotiation


<!-- ---------------------------------------------------------------- -->

---
term: "DefinedTerm"
canonical: https://citable.wiki/glossary/defined-term
also_known_as: "schema.org/DefinedTerm, DefinedTermSet"
license: CC BY 4.0
---

# DefinedTerm

**Definition:** DefinedTerm is a Schema.org type for a word, name, acronym or phrase that has a formal definition. It carries a name, a description that holds the definition, an optional termCode, and an inDefinedTermSet link to the DefinedTermSet — the glossary or classification — it belongs to. It is the natural markup for a glossary entry and tells a machine that a passage is a definition.

`DefinedTerm` sits under `Intangible` in the Schema.org hierarchy. Schema.org describes it as a word, name, acronym or phrase with a formal definition, and lists category systems, glossaries, dictionaries and product classifications as typical uses. A `DefinedTermSet` groups terms; it is itself a `CreativeWork`, so a whole glossary can have an author and a URL. Several properties elsewhere in the vocabulary, including `keywords`, accept a `DefinedTerm` as a value, which lets an article point at a controlled term rather than a free-text tag.

Definitions are among the most quoted passage types, and a definition marked as `DefinedTerm` with a stable `@id` becomes an entity other pages can reference. Every entry in this glossary emits markup in this shape:

```json
{
  "@context": "https://schema.org",
  "@type": "DefinedTerm",
  "@id": "https://example.com/glossary/llms-txt",
  "name": "llms.txt",
  "description": "llms.txt is a proposed convention for a Markdown file …",
  "inDefinedTermSet": { "@type": "DefinedTermSet", "name": "Citable glossary" }
}
```

Keep the `description` identical to the visible definition, and keep both self-contained: name the term in the first few words so the sentence still makes sense when quoted alone. The [Schema.org](/glossary/schema-org) entry places the type among its neighbours, and [Structured data for AI answers](/guides/structured-data-for-ai-answers) shows how to generate it from CMS fields.

## Frequently asked questions

### Should a glossary page use DefinedTerm or FAQPage?

Use DefinedTerm for the definition itself and FAQPage only for genuine question-and-answer pairs on the same page. A definition is not an answer to a question, and marking it as one misdescribes it. The two types can coexist: this site emits DefinedTerm for every glossary entry and adds FAQPage when the entry has FAQ items.

## Related guides

- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)
- [Modelling content for answer engines in a headless CMS!](https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms)

---

Source: https://citable.wiki/glossary/defined-term


<!-- ---------------------------------------------------------------- -->

---
term: "E-E-A-T"
canonical: https://citable.wiki/glossary/e-e-a-t
also_known_as: "Experience, Expertise, Authoritativeness, Trustworthiness; formerly E-A-T"
license: CC BY 4.0
---

# E-E-A-T

**Definition:** E-E-A-T stands for Experience, Expertise, Authoritativeness and Trustworthiness: the framework Google's Search Quality Rater Guidelines use to describe whether a page and its author deserve to be believed. It is not a direct ranking signal but a description of the qualities Google's ranking systems try to reward, with trust named as the most important of the four.

Google's quality rater guidelines used **E-A-T** for years before the extra E. Experience was added in December 2022 to distinguish first-hand knowledge — having actually used the product, visited the place or run the test — from expertise learned second-hand ([Google Search Central](https://developers.google.com/search/blog/2022/12/google-raters-guidelines-e-e-a-t)). Google's guidance on [helpful, reliable, people-first content](https://developers.google.com/search/docs/fundamentals/creating-helpful-content) adds that a page need not demonstrate all four qualities, that trust matters most, and that the bar is higher for topics affecting health, money or safety.

For answer engines the framework matters because a retrieval system cannot judge expertise directly. It relies on proxies it can verify: a named author with a profile page, visible published and updated dates, outbound links to primary sources, and a consistent identity for the author and organisation across the web. Google states there are no additional requirements for appearing in AI Overviews beyond the same people-first guidance, so E-E-A-T is the trust layer underneath every other tactic in [E-E-A-T for answer engines](/guides/eeat-for-answer-engines).

The cheapest machine-checkable signal is `Person` markup attached to the `Article`:

```json
"author": {
  "@type": "Person",
  "name": "Jane Doe",
  "url": "https://example.com/authors/jane-doe",
  "jobTitle": "Technical writer",
  "sameAs": ["https://www.linkedin.com/in/jane-doe"]
}
```

## Frequently asked questions

### Is E-E-A-T a ranking factor?

Not directly. Google describes E-E-A-T as the framework its quality raters use to evaluate results, and says its ranking systems use a mix of signals that tend to identify content with strong E-E-A-T. For a page, the practical reading is the same either way: show who wrote it, what qualifies them, and where the claims come from.

## Related guides

- [E-E-A-T for answer engines: authors, dates and sources that machines can verify](https://citable.wiki/guides/eeat-for-answer-engines)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)
- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)

---

Source: https://citable.wiki/glossary/e-e-a-t


<!-- ---------------------------------------------------------------- -->

---
term: "Embedding"
canonical: https://citable.wiki/glossary/embedding
also_known_as: "Vector embedding, text embedding"
license: CC BY 4.0
---

# Embedding

**Definition:** An embedding is a fixed-length list of numbers (a vector) that a neural model produces to represent the meaning of a piece of text, so that texts with similar meaning end up close together in the vector space. Answer engines embed both the question and candidate passages, then retrieve the passages whose vectors are nearest to the question's.

Embedding models take text in and return a vector. OpenAI's `text-embedding-3-small` returns 1,536 numbers by default and `text-embedding-3-large` returns 3,072 ([OpenAI documentation](https://developers.openai.com/api/docs/guides/embeddings)). Similarity is usually measured as cosine similarity: vectors pointing the same way score near 1, unrelated ones near 0. A retrieval system computes the vector for each [chunk](/glossary/chunking) of a page once, stores it in a vector index, and at query time compares the question's vector against millions of stored ones.

Two properties of embeddings shape how you should write.

**They capture meaning, not spelling.** "How do I stop OpenAI from crawling my site" and a passage about `GPTBot` in `robots.txt` land close together even though they share almost no words. Synonyms and paraphrase are handled for you; what must still be present is the concept and the named entity.

**They are computed per passage, in isolation.** The model never sees the heading three sections up or the antecedent of a pronoun. A chunk that begins "It is also worth noting that this approach…" embeds as vague filler; one that begins "Answer-first content places the direct answer…" embeds as a strong match for the question it answers. Production retrievers usually pair embeddings with a keyword scorer such as BM25, so exact terms still count, but the vector side is where [answer-first structure](/guides/how-to-write-answer-first-content) pays off.

## Frequently asked questions

### Do I need to understand embeddings to do AEO?

Only the consequence: retrieval matches meaning, not exact strings. A passage is retrievable for a question when it covers the same concepts and names the same entities, which is why answer-first passages that restate the question's terms in their own first sentence do well without keyword repetition.

## Related guides

- [How to write answer-first content that LLMs can quote](https://citable.wiki/guides/how-to-write-answer-first-content)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)

---

Source: https://citable.wiki/glossary/embedding


<!-- ---------------------------------------------------------------- -->

---
term: "Entity"
canonical: https://citable.wiki/glossary/entity
license: CC BY 4.0
---

# Entity

**Definition:** An entity is a uniquely identifiable thing — a person, organisation, product, place or concept — that exists independently of the words used to name it. Search engines and language models resolve text to entities so that 'Apple', 'Apple Inc.' and 'AAPL' map to one node, and they judge a page partly by which entities it mentions and how unambiguously it names them.

The word entered search vocabulary in May 2012, when Google announced its [Knowledge Graph](/glossary/knowledge-graph) with the phrase "things, not strings". The point was that a query is about a thing in the world, not a sequence of characters, and that a search engine should model the thing.

Answer engines inherit that model twice over. At retrieval time, a passage that contains the entities in the question — named the way the question names them — scores higher than one that refers to them obliquely. At generation time, the language model has to decide *which* Mercury, *which* Cambridge, *which* Apple a passage is about; if it cannot, it either skips the passage or attributes your claim to the wrong thing.

Structured data is the cheapest way to remove that doubt. This site describes its author and publisher as entities with stable identifiers, following the pattern in [Structured data for AI answers](/guides/structured-data-for-ai-answers):

```json
{
  "@type": "Organization",
  "@id": "https://example.com/#organization",
  "name": "Example Ltd",
  "sameAs": [
    "https://www.wikidata.org/wiki/Q000000",
    "https://www.linkedin.com/company/example"
  ]
}
```

Every glossary term here is emitted as a `DefinedTerm` for the same reason: one page, one entity, one unambiguous name.

## Frequently asked questions

### How do I make an entity unambiguous on a page?

Use the full name on first mention, put the abbreviation in parentheses, and stay consistent afterwards instead of switching to 'it' or a nickname. Then give the entity machine-readable identity: an Organization, Person or DefinedTerm object in JSON-LD with a stable @id and sameAs links to the same thing on Wikidata, Wikipedia or an official profile.

## Related guides

- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)
- [How to write answer-first content that LLMs can quote](https://citable.wiki/guides/how-to-write-answer-first-content)

---

Source: https://citable.wiki/glossary/entity


<!-- ---------------------------------------------------------------- -->

---
term: "Generative Engine Optimization"
canonical: https://citable.wiki/glossary/generative-engine-optimization
also_known_as: "GEO"
license: CC BY 4.0
---

# Generative Engine Optimization

**Definition:** Generative Engine Optimization (GEO) is the practice of increasing how often, and how favourably, a source is used and cited by generative engines — search systems that answer with text produced by a large language model, such as Google AI Overviews, Perplexity or ChatGPT search. The term was introduced in a 2023 research paper by Aggarwal et al.

GEO is the academic name for the problem AEO practitioners were already working on. The original paper, *GEO: Generative Engine Optimization* ([arXiv:2311.09735](https://arxiv.org/abs/2311.09735)), defined "generative engines" as systems that synthesise an answer from multiple sources with a language model, proposed a benchmark (GEO-bench) and measured which content edits increased a source's share of the generated answer.

Its headline finding is the one most often repeated: **evidence beats keywords**. Content edits that added citations, quotations and statistics improved visibility; edits that added keywords did not.

## Frequently asked questions

### What did the GEO paper find?

The study (Aggarwal et al., 2023, later published at KDD 2024) tested nine content changes on a benchmark of queries. Adding citations to sources, adding quotations from authorities and adding statistics increased a source's visibility in generated answers by up to 40% in their metric, while keyword stuffing had little or negative effect.

## Related guides

- [GEO vs SEO vs AEO: what actually changed](https://citable.wiki/guides/geo-vs-seo-vs-aeo)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)

---

Source: https://citable.wiki/glossary/generative-engine-optimization


<!-- ---------------------------------------------------------------- -->

---
term: "Grounding"
canonical: https://citable.wiki/glossary/grounding
license: CC BY 4.0
---

# Grounding

**Definition:** Grounding is the practice of tying a language model's output to specific, verifiable sources — retrieved web pages, documents or a knowledge graph — supplied at generation time, so that each claim in the answer can be traced to evidence rather than to the model's training data. In answer engines, grounding is what turns a retrieved passage into a cited one.

The term comes from the model vendors' own APIs. Google's Gemini API offers [Grounding with Google Search](https://ai.google.dev/gemini-api/docs/grounding), which it describes as connecting the model to real-time web content so it can give more accurate answers and cite verifiable sources beyond its knowledge cutoff; the response carries grounding metadata that maps spans of the answer to source URLs. Anthropic's web search tool and OpenAI's web search tool return the same kind of span-to-source annotations. Consumer answer engines expose them as footnotes.

For a content owner, grounding is the stage where being *retrieved* becomes being *cited*. The model is handed a handful of passages and asked to answer using them. A passage that states the answer plainly, with the entity named and a figure or date attached, is easy to ground on: the model can quote or closely paraphrase it and point to it. A passage that hedges, buries the claim, or depends on context elsewhere on the page gets read for background and dropped from the attributions.

That is why [answer-first writing](/guides/how-to-write-answer-first-content) and [retrieval-augmented generation](/glossary/retrieval-augmented-generation) belong together: retrieval decides whether a model sees your passage, grounding decides whether it credits it. Grounding is also the main defence against [hallucination](/glossary/hallucination), which is why engines prefer sources whose claims they can check against one another.

## Frequently asked questions

### Is grounding the same as retrieval-augmented generation?

They overlap but are not identical. RAG is the architecture — retrieve passages, then generate. Grounding is the goal that architecture serves: an answer whose claims are anchored to, and attributable to, the supplied sources. A RAG system can still produce ungrounded statements if the model ignores or contradicts what was retrieved.

## Related guides

- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)
- [How to write answer-first content that LLMs can quote](https://citable.wiki/guides/how-to-write-answer-first-content)

---

Source: https://citable.wiki/glossary/grounding


<!-- ---------------------------------------------------------------- -->

---
term: "Hallucination"
canonical: https://citable.wiki/glossary/hallucination
also_known_as: "Confabulation, fabrication"
license: CC BY 4.0
---

# Hallucination

**Definition:** A hallucination is output from a language model that is fluent and confident but false or unsupported: an invented statistic, a citation to a paper that does not exist, a misattributed quote, or a plausible-sounding date. It arises because the model predicts likely text rather than checking facts, and it is the main failure that answer engines use retrieval and citations to contain.

The term was borrowed from psychology and became mainstream in 2023, when a New York federal court sanctioned lawyers who had filed a brief containing case citations that ChatGPT had invented. The fabricated citations looked exactly like real ones and were checkable in minutes.

Hallucination is not a bug that a patch removes. A 2025 paper from OpenAI researchers, [Why Language Models Hallucinate](https://arxiv.org/abs/2509.04664), argues that it persists partly because training and benchmarks reward a confident guess over an admission of uncertainty. The engineering response is to change what the model is asked to do: an answer engine retrieves passages and generates from them — [retrieval-augmented generation](/glossary/retrieval-augmented-generation) with [grounding](/glossary/grounding) and visible citations, so that a reader can verify each claim against a source.

> **Warning: Why this matters for your pages**
>
> Retrieval reduces hallucination only when the retrieved passage carries the fact. A passage that hedges, relies on a definition three sections earlier, or leaves the key figure in an image forces the model to guess — and the guess is published under your name.

The defence is editorial, not technical: state the number, the unit, the date and the full entity name inside the same paragraph, and keep each answer self-contained. The method is in [How to write answer-first content](/guides/how-to-write-answer-first-content).

## Frequently asked questions

### Can the way I write cause a hallucination about my own content?

Yes. When a retrieved passage is vague — a number without a unit, a claim without a date, a product referred to as 'it' — the model fills the gap with whatever is statistically likely, and the invented detail is then attributed to you. Self-contained passages with explicit names, figures, units and dates give the model nothing to guess.

## Related guides

- [E-E-A-T for answer engines: authors, dates and sources that machines can verify](https://citable.wiki/guides/eeat-for-answer-engines)
- [How to write answer-first content that LLMs can quote](https://citable.wiki/guides/how-to-write-answer-first-content)

---

Source: https://citable.wiki/glossary/hallucination


<!-- ---------------------------------------------------------------- -->

---
term: "Headless CMS"
canonical: https://citable.wiki/glossary/headless-cms
also_known_as: "API-first CMS, decoupled content platform"
license: CC BY 4.0
---

# Headless CMS

**Definition:** A headless CMS is a content management system that stores content as structured fields and delivers it over an API, leaving the presentation layer — website, app, feed, Markdown export — to be built separately. For answer engines it matters because the same fields that render the page can also generate JSON-LD, Markdown and llms.txt without the outputs drifting apart.

The "head" is the presentation layer that a traditional CMS bundles together with its editing interface and database. A headless system keeps the editor and the content store and drops the head: content is fetched through a REST or GraphQL API by whatever needs it, whether a website, a mobile app or a build script. Storyblok, Contentful and Sanity are typical examples. This site runs on Storyblok with a Next.js front end.

An answer engine only ever sees the output, but the quality of that output is decided by the content model. When the short answer, the FAQ items, the sources and the author are fields with validation rules, a page cannot be published without them, and the `FAQPage` and `Person` JSON-LD, the Markdown representation and the [/llms.txt](/llms.txt) index are all generated from the same values as the visible text. Nothing is kept in sync by hand.

The guide content type on this site, abbreviated:

```yaml
guide:
  title: text
  short_answer: textarea         # 40–70 words; the Speakable target
  body: richtext
  key_takeaways: bloks[takeaway]
  faq: bloks[faq_item]           # emitted as FAQPage
  sources: bloks[source]         # emitted as Article.citation
  author: reference[author]      # emitted as Person
  related_terms: references[glossary_term]
```

The guide on [modelling content for answer engines in a headless CMS](/guides/modeling-content-for-answer-engines-in-a-headless-cms) walks through the model field by field.

## Frequently asked questions

### Why does a headless CMS help with AEO?

Because it forces content into named fields. A short answer, a list of FAQ items, a sources list and an author reference are separate fields rather than paragraphs inside one rich-text blob, so the build step can emit them as visible HTML, as schema.org markup and as Markdown from a single source. A traditional page builder makes that separation optional, and in practice it erodes.

## Related guides

- [Modelling content for answer engines in a headless CMS!](https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms)
- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)

---

Source: https://citable.wiki/glossary/headless-cms


<!-- ---------------------------------------------------------------- -->

---
term: "JSON-LD"
canonical: https://citable.wiki/glossary/json-ld
also_known_as: "JSON for Linking Data, application/ld+json"
license: CC BY 4.0
---

# JSON-LD

**Definition:** JSON-LD (JSON for Linking Data) is a W3C standard for expressing linked data as ordinary JSON. On the web it is the format Google recommends for Schema.org structured data: a single script block of type application/ld+json that describes the page's entities — article, author, dates, FAQ, definitions — without touching the visible HTML.

JSON-LD 1.0 became a W3C Recommendation in January 2014; the current version, JSON-LD 1.1, followed in July 2020. The design goal was to let ordinary JSON become linked data by adding a `@context` that maps plain keys to globally defined terms, so a `"name"` in one document means the same thing as a `"name"` in another.

That separation from the HTML is what makes JSON-LD useful for answer engines. A crawler can read one JSON object and learn what the page is (`@type`), who wrote it (`author`), when it was published and updated, what it cites, and which selectors hold the direct answer. None of that requires parsing prose or guessing at layout. Three keys carry most of the weight: `@context`, `@type` and `@id`. Giving an entity an `@id` gives it a stable URL, so the same author or the same [glossary term](/glossary/defined-term) can be referenced consistently from many pages.

```json
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "What is Answer Engine Optimization (AEO)?",
  "author": { "@type": "Person", "@id": "https://example.com/about#author" },
  "datePublished": "2026-01-12",
  "dateModified": "2026-08-20"
}
```

The main risk is drift: markup that says one thing while the visible text says another. Generate the JSON-LD from the same fields that render the page, as described in [Structured data for AI answers](/guides/structured-data-for-ai-answers).

## Frequently asked questions

### Is JSON-LD better than Microdata or RDFa for answer engines?

For most sites, yes. All three carry the same Schema.org vocabulary, but JSON-LD sits in one script block that can be generated from CMS fields, whereas Microdata and RDFa are spread across HTML attributes and break when templates change. Google recommends JSON-LD for the same reason: it is the easiest to implement and maintain at scale.

## Related guides

- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)
- [Modelling content for answer engines in a headless CMS!](https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms)

---

Source: https://citable.wiki/glossary/json-ld


<!-- ---------------------------------------------------------------- -->

---
term: "Knowledge graph"
canonical: https://citable.wiki/glossary/knowledge-graph
license: CC BY 4.0
---

# Knowledge graph

**Definition:** A knowledge graph is a structured database of entities — people, organisations, places, products, concepts — and the typed relationships between them, such as founded by, located in or is a subtype of. Search engines and answer engines use one to disambiguate names, verify facts and assemble panels; Google's Knowledge Graph, introduced in 2012, and the open Wikidata project are the best-known examples.

Google announced its Knowledge Graph on 16 May 2012 with the slogan [things, not strings](https://blog.google/products-and-platforms/products/search/introducing-knowledge-graph-things-not/), seeded from Freebase, Wikipedia and the CIA World Factbook. Wikidata, the open knowledge graph maintained by the Wikimedia community, launched later the same year and is now the identifier most other systems link to. Google also exposes a [Knowledge Graph Search API](https://developers.google.com/knowledge-graph) that returns entities as schema.org JSON-LD, which is a useful way to check whether a name resolves to anything at all.

For answer engines the graph is a source of corroboration. A language model that can resolve your company, your author and your product to known [entities](/glossary/entity) can attach your claims to them, judge whether you are a plausible authority, and avoid confusing you with a namesake. A name that resolves to nothing has to be taken on trust, which retrieval systems are designed not to do.

Sites do not write to a knowledge graph directly; they make it easy for one to read them. The pattern in [Structured data for AI answers](/guides/structured-data-for-ai-answers) is a `Person` or `Organization` object with a stable `@id` and `sameAs` links:

```json
{
  "@type": "Person",
  "@id": "https://example.com/authors/jane-doe#person",
  "name": "Jane Doe",
  "sameAs": ["https://www.wikidata.org/wiki/Q000000", "https://github.com/janedoe"]
}
```

## Frequently asked questions

### How does an organisation or author get into a knowledge graph?

Mostly by being consistently described in sources the graph already trusts: a Wikipedia article and Wikidata item where the notability threshold is met, official profiles, and a home page whose Organization or Person JSON-LD links all of those with sameAs. Consistency of name, logo, founding date and description across those sources is what lets the graph merge them into one entity.

## Related guides

- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)
- [E-E-A-T for answer engines: authors, dates and sources that machines can verify](https://citable.wiki/guides/eeat-for-answer-engines)

---

Source: https://citable.wiki/glossary/knowledge-graph


<!-- ---------------------------------------------------------------- -->

---
term: "llms.txt"
canonical: https://citable.wiki/glossary/llms-txt
license: CC BY 4.0
---

# llms.txt

**Definition:** llms.txt is a proposed convention for a Markdown file at the root of a website (/llms.txt) that gives large language models a curated, plain-text overview of the site: a title, a short summary, and sections of links to the most important pages with one-line descriptions. A companion file, llms-full.txt, contains the full text of those pages in one document.

The format is deliberately simple so that a model can read it without a parser: an H1 with the site name, a blockquote with a summary, optional free-text paragraphs, and H2 sections containing Markdown link lists of the form `- [Title](url): description`. An `## Optional` section marks links that can be skipped when context is limited.

This site publishes both [/llms.txt](/llms.txt) and [/llms-full.txt](/llms-full.txt), generated from the CMS on every content change.

## Frequently asked questions

### Who proposed llms.txt?

Jeremy Howard of Answer.AI published the proposal at llmstxt.org in September 2024. It is a community convention, not a web standard, and no major AI provider has publicly committed to reading it.

### Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers what they may fetch; llms.txt tells a model what your site contains and where the best pages are. They are complementary.

## Related guides

- [llms.txt: what it is, how to write one, and whether it helps](https://citable.wiki/guides/llms-txt-what-it-is-and-how-to-write-one)

---

Source: https://citable.wiki/glossary/llms-txt


<!-- ---------------------------------------------------------------- -->

---
term: "Passage retrieval"
canonical: https://citable.wiki/glossary/passage-retrieval
license: CC BY 4.0
---

# Passage retrieval

**Definition:** Passage retrieval is the search technique of scoring and returning short spans of text — a paragraph, a section, a window of tokens — rather than whole documents, so that a system can answer a question from the most relevant fragment. It is the retrieval step in most answer engines, and it is why a page is cited passage by passage rather than as a unit.

Passage retrieval comes from open-domain question answering, where the task is to find the few sentences in a large corpus that answer a question. Dense Passage Retrieval ([Karpukhin et al., 2020](https://arxiv.org/abs/2004.04906)) showed that embedding passages and questions into one vector space could beat the classic BM25 keyword baseline; that method, usually combined with keyword matching and a reranker, sits inside most [retrieval-augmented generation](/glossary/retrieval-augmented-generation) pipelines today. Google announced passage-level ranking for ordinary web search at its [Search On event in October 2020](https://blog.google/products-and-platforms/products/search/search-on/).

The practical consequence is that your page is never judged whole. It is cut into [chunks](/glossary/chunking), each is scored against the question, and only the winners reach the model. The heading above a paragraph, its opening sentence and whether it names its subject explicitly decide whether it is a candidate at all.

**Making a passage retrievable**

- [ ] One question per section, with the question or its key phrase in the heading
- [ ] The answer in the first sentence, then the qualification
- [ ] The full entity name in the passage, not a pronoun pointing at an earlier section
- [ ] Figures, units and dates inside the same paragraph as the claim they support
- [ ] Plain HTML paragraphs and lists, so the chunk boundaries fall where the meaning does

Modelling content so that every chunk is self-contained is covered in [Modelling content for answer engines in a headless CMS](/guides/modeling-content-for-answer-engines-in-a-headless-cms).

## Frequently asked questions

### Why does passage retrieval favour answer-first writing?

Because the retriever scores each passage on its own, without the rest of the page. A paragraph that opens with the answer, names the entity in full and needs no earlier context resembles the question closely and is complete when lifted out. A paragraph that builds up to the answer scores lower and, if selected, gives the model a fragment that stops mid-thought.

## Related guides

- [How to write answer-first content that LLMs can quote](https://citable.wiki/guides/how-to-write-answer-first-content)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)
- [Modelling content for answer engines in a headless CMS!](https://citable.wiki/guides/modeling-content-for-answer-engines-in-a-headless-cms)

---

Source: https://citable.wiki/glossary/passage-retrieval


<!-- ---------------------------------------------------------------- -->

---
term: "Retrieval-augmented generation"
canonical: https://citable.wiki/glossary/retrieval-augmented-generation
also_known_as: "RAG"
license: CC BY 4.0
---

# Retrieval-augmented generation

**Definition:** Retrieval-augmented generation (RAG) is an architecture in which a language model is given relevant documents or passages retrieved at query time — from a search index, a vector database or the live web — and asked to answer using them. It grounds the model's output in current, citable sources instead of relying only on what it memorised during training.

Most answer engines are RAG systems: a retriever fetches candidate passages, a reranker orders them, and the generator writes an answer while attributing claims to the passages it used. The term was introduced by Lewis et al. in 2020 ([arXiv:2005.11401](https://arxiv.org/abs/2005.11401)).

For AEO, the retrieval step is the gate. Content that is not in the index, that is chunked badly, or that does not resemble the question in wording and entities never reaches the model at all.

## Frequently asked questions

### Why does RAG matter for content creators?

Because it is the mechanism by which your page can be cited. In a RAG pipeline the model only sees the passages that retrieval returned, so a page must first be retrievable — crawlable, chunkable, relevant — before it can be quoted.

## Related guides

- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)

---

Source: https://citable.wiki/glossary/retrieval-augmented-generation


<!-- ---------------------------------------------------------------- -->

---
term: "Schema.org"
canonical: https://citable.wiki/glossary/schema-org
license: CC BY 4.0
---

# Schema.org

**Definition:** Schema.org is a shared vocabulary of types and properties — Article, Person, FAQPage, DefinedTerm and hundreds of others — for describing what a web page is about in a form machines can read. Founded in 2011 by Google, Microsoft, Yahoo and Yandex, it is maintained in the open through a W3C community group and can be written as JSON-LD, Microdata or RDFa.

Before Schema.org, each search engine published its own markup guidance and site owners had to pick between competing vocabularies. The launch in June 2011 gave the major engines one vocabulary to point at. Since April 2015 changes have been proposed and discussed in the W3C Schema.org Community Group, and new releases appear several times a year.

The vocabulary is a hierarchy: every type descends from `Thing`, so a `TechArticle` is also an `Article` and a `CreativeWork`, and inherits their properties. The types that matter for being cited by an answer engine are a short list:

- `Article` and `TechArticle`, with `author`, `datePublished`, `dateModified` and `citation`
- `Person` and `Organization`, with `sameAs` links to profiles that establish who the author is
- `FAQPage` with `Question` and `Answer` pairs
- [`DefinedTerm`](/glossary/defined-term) and `DefinedTermSet` for glossary entries
- `SpeakableSpecification` to point at the passage that is the answer

Schema.org defines the meaning; it says nothing about how the markup is encoded. Google supports [JSON-LD](/glossary/json-ld), Microdata and RDFa and recommends JSON-LD. Property definitions live at predictable URLs (for example `https://schema.org/citation`), which is the fastest way to check what a property expects before you emit it. The guide on [structured data for AI answers](/guides/structured-data-for-ai-answers) covers which properties to fill and how to keep them in sync with the visible page.

## Frequently asked questions

### Does Schema.org markup improve rankings or citations?

Google documents structured data as a way to become eligible for rich results and to help Search understand a page, not as a general ranking factor. No AI vendor has published how its retrieval system weights it. Treat Schema.org as disambiguation — it removes doubt about what a passage is and who wrote it — rather than as a lever that moves a bad page up.

## Related guides

- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)

---

Source: https://citable.wiki/glossary/schema-org


<!-- ---------------------------------------------------------------- -->

---
term: "Semantic HTML"
canonical: https://citable.wiki/glossary/semantic-html
license: CC BY 4.0
---

# Semantic HTML

**Definition:** Semantic HTML is markup that uses elements for what content is rather than how it should look: article, section, nav, h1–h6, dl, table, time, blockquote and cite, instead of generic div and span. Because crawlers and text extractors read structure rather than styling, semantic markup determines how a page is split into passages and which parts survive extraction.

The sectioning and heading elements are defined in the [WHATWG HTML Living Standard](https://html.spec.whatwg.org/multipage/sections.html). MDN's glossary entry on [semantics](https://developer.mozilla.org/en-US/docs/Glossary/Semantics) states the principle plainly: write HTML to represent the data, not its default presentation.

Answer engine pipelines typically convert a page to plain text or Markdown before chunking it, and that conversion is only as good as the markup. Headings become section boundaries, `<p>` becomes a paragraph, `<table>` becomes a table, `<dl>` becomes term and definition pairs. A page built from nested `<div>` elements with visual class names converts into one undifferentiated block, and the answer-first passage at the top merges with the navigation around it.

The rules that matter most for extraction:

- one `<h1>`, and a heading outline that does not skip levels
- `<main>` around the content; `<nav>`, `<header>`, `<footer>` and `<aside>` around everything that is not the content
- `<time datetime="…">` on published and updated dates
- real `<ul>`, `<ol>`, `<table>` and `<dl>` elements for list-like, tabular and definitional data

```html
<article>
  <h1>What is content negotiation?</h1>
  <p data-speakable="short-answer">Content negotiation is the HTTP mechanism…</p>
  <p>Published <time datetime="2026-01-12">12 January 2026</time></p>
  <h2>How it works</h2>
  <dl>
    <dt>Accept</dt>
    <dd>The media types the client prefers, in order.</dd>
  </dl>
</article>
```

A page marked up this way converts cleanly into the Markdown that [this site serves to crawlers](/guides/serving-markdown-to-llms).

## Frequently asked questions

### Is semantic HTML the same as structured data?

No. Semantic HTML describes the role of content within the page — this is an article, this is its heading, this is a definition list. Structured data (JSON-LD using schema.org types) describes the things the page is about — a person, an organisation, a defined term — in a vocabulary machines share across sites. Good pages use both, generated from the same source.

## Related guides

- [Serving Markdown to LLMs: content negotiation and .md endpoints](https://citable.wiki/guides/serving-markdown-to-llms)
- [How to write answer-first content that LLMs can quote](https://citable.wiki/guides/how-to-write-answer-first-content)

---

Source: https://citable.wiki/glossary/semantic-html


<!-- ---------------------------------------------------------------- -->

---
term: "Speakable"
canonical: https://citable.wiki/glossary/speakable
also_known_as: "SpeakableSpecification"
license: CC BY 4.0
---

# Speakable

**Definition:** Speakable is a schema.org property (with the SpeakableSpecification type) that identifies, by CSS selector or XPath, which parts of a page are best suited to be read aloud or quoted verbatim — typically a headline and a short summary. It was introduced for voice assistants and remains the most direct way to tell a machine which passage is the answer.

Google documents `speakable` as a beta feature for news publishers on Google Assistant, and it is not a confirmed ranking or citation signal for AI answer features. It is nonetheless cheap, harmless and precise: it points at the exact element that contains your direct answer.

Every guide on this site marks its short-answer block with `data-speakable="short-answer"` and references that selector from the `TechArticle` JSON-LD:

```json
"speakable": {
  "@type": "SpeakableSpecification",
  "cssSelector": ["[data-speakable='short-answer']", "[data-speakable='headline']"]
}
```

## Related guides

- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)

---

Source: https://citable.wiki/glossary/speakable


<!-- ---------------------------------------------------------------- -->

---
term: "Structured data"
canonical: https://citable.wiki/glossary/structured-data
also_known_as: "schema markup, semantic markup"
license: CC BY 4.0
---

# Structured data

**Definition:** Structured data, in the web context, is machine-readable markup embedded in a page that states explicitly what the page contains — an article, its author, a publication date, a question and its answer — using a shared vocabulary such as Schema.org. It is written as JSON-LD, Microdata or RDFa and lets crawlers and answer engines identify entities and relationships without inferring them from prose.

The phrase covers three encodings of the same idea. **JSON-LD** puts a JSON object in a `<script type="application/ld+json">` block. **Microdata** and **RDFa** attach attributes such as `itemprop` or `property` to the HTML elements that already hold the text. Google supports all three and recommends JSON-LD; its guidelines also require that marked-up content be visible on the page and representative of it, so markup is never a place to say things the reader cannot see.

For answer engines the value is in what survives chunking. When a page is split into passages, a paragraph in the middle of the article no longer knows who wrote the page or when it was updated. The JSON-LD block does: it carries `author`, `datePublished`, `dateModified` and `citation` as explicit fields, and an `FAQPage` block states that a given question and answer belong together.

```json
{
  "@type": "FAQPage",
  "mainEntity": [{
    "@type": "Question",
    "name": "Is AEO the same as GEO?",
    "acceptedAnswer": { "@type": "Answer", "text": "They overlap heavily. …" }
  }]
}
```

The practical rule is one source of truth: generate the markup from the same CMS fields that render the visible text, so the two can never disagree. [Structured data for AI answers](/guides/structured-data-for-ai-answers) walks through the types worth emitting and the [Schema.org](/glossary/schema-org) entry lists them.

## Frequently asked questions

### Is structured data required to be cited by an answer engine?

No. Answer engines cite pages that have no markup at all, and Google's guidance for its AI features points at the same practices it recommends for Search rather than any extra markup. Structured data is an accuracy layer: it removes ambiguity about what a passage is, who wrote it and when, which are the things a retrieval system otherwise has to guess.

## Related guides

- [Structured data for AI answers: the schema.org types that matter](https://citable.wiki/guides/structured-data-for-ai-answers)
- [FAQ content that answer engines pick up (and why FAQPage rich results went away)](https://citable.wiki/guides/faq-content-for-answer-engines)

---

Source: https://citable.wiki/glossary/structured-data


<!-- ---------------------------------------------------------------- -->

---
term: "Zero-click search"
canonical: https://citable.wiki/glossary/zero-click-search
also_known_as: "No-click search"
license: CC BY 4.0
---

# Zero-click search

**Definition:** A zero-click search is a query that ends on the results page without the searcher clicking through to any website, because the answer was shown directly — in a featured snippet, knowledge panel, AI Overview or a similar module. The term describes a measured outcome, not a feature: the searcher got what they needed, or gave up, and the source page received no visit.

The term was popularised by Rand Fishkin, whose clickstream studies from 2019 onward measured how many Google searches ended without a click to the open web. The most recent large study, run with the panel provider Datos, put the figure well above half.

**58.5%** — of US Google searches in 2024 ended without a click to any website, according to clickstream data from roughly 130,000 devices (Source: [SparkToro / Datos, 2024 Zero-Click Search Study](https://sparktoro.com/blog/2024-zero-click-search-study-for-every-1000-us-google-searches-only-374-clicks-go-to-the-open-web-in-the-eu-its-360/))

Answer engines make zero-click the default rather than the exception. Google's AI Overviews, Perplexity, ChatGPT search and Claude with web search all synthesise an answer first and attach sources second; most people who read the answer never open a source. The unit of reward shifts from the visit to the [citation](/glossary/citation), which is why this site treats being quoted as the primary outcome and traffic as a secondary one.

Two consequences follow for anyone publishing content:

- Standard analytics count visits, so a citation without a click is invisible unless you look for it. The methods are in [Measuring AI visibility and citations](/guides/measuring-ai-visibility-and-citations).
- Attribution matters more than position. A passage that *is* the answer tends to be cited by name; a passage that merely discusses the answer tends to be paraphrased without one.

## Frequently asked questions

### Is a zero-click search bad for a publisher?

It is bad for a publisher whose only measure of success is a session. The passage that answered the query still did its job, and if it was attributed, the publisher gained a citation and a brand mention. The practical response is to measure citations and mentions alongside visits, and to write for the query types that still send clicks.

## Related guides

- [Measuring AI visibility: how to track citations in ChatGPT, Perplexity and AI Overviews](https://citable.wiki/guides/measuring-ai-visibility-and-citations)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)

---

Source: https://citable.wiki/glossary/zero-click-search
