---
title: "Controlling AI crawlers with robots.txt: GPTBot, ClaudeBot, PerplexityBot and friends"
subtitle: "Which AI user agents exist, what each one is for, and how to write a robots.txt that keeps you citable without feeding every training run."
canonical: https://citable.wiki/guides/controlling-ai-crawlers-with-robots-txt
category: technical
author: "Endrit Krasniqi"
date_published: 2026-03-16
date_modified: 2026-03-16
license: CC BY 4.0
---

# Controlling AI crawlers with robots.txt: GPTBot, ClaudeBot, PerplexityBot and friends

*Which AI user agents exist, what each one is for, and how to write a robots.txt that keeps you citable without feeding every training run.*

## Short answer

AI crawlers are controlled through robots.txt user-agent groups, and every major vendor runs several agents. OpenAI and Anthropic separate a training crawler (GPTBot, ClaudeBot), a search indexer (OAI-SearchBot, Claude-SearchBot) and a user-triggered fetcher; Perplexity runs PerplexityBot and Perplexity-User. Google's AI Overviews use ordinary Googlebot; Google-Extended only opts out of Gemini training and grounding. Allow the search agents if you want citations, and treat user-triggered fetchers as outside robots.txt.

## Key takeaways

- Each vendor runs separate agents for training, search indexing and user-triggered fetching; block them individually, not as one 'AI' bucket.
- A robots.txt group named after a bot replaces the wildcard group for that bot, so repeat your ordinary Disallow lines in every named group.
- Google's AI Overviews and AI Mode are governed by Googlebot rules; Google-Extended only controls Gemini training and grounding, and does not affect Search.
- OpenAI, Perplexity, Meta and Google all document that user-triggered fetchers may ignore robots.txt because a person requested the page.
- Blocking the training crawlers does not remove you from ChatGPT search or Perplexity, provided OAI-SearchBot and PerplexityBot stay allowed.
- Verify a claimed bot against the vendor's published IP list or reverse DNS before trusting the user-agent string.

## What robots.txt can and cannot do for AI crawlers

robots.txt is the oldest control you have over automated access, and it is the one every AI vendor points to first. The rules are standardised in [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html): a file at `/robots.txt`, groups introduced by `User-agent:` lines, and `Allow:` and `Disallow:` path rules inside each group.

Two properties of the standard matter more for AI crawlers than they ever did for search engines.

First, **a crawler uses only the group that matches its own name**. If GPTBot finds a `User-agent: GPTBot` group, it ignores the `User-agent: *` group entirely. Any path you disallow for everyone must be repeated inside every named group, or the named bot will crawl it.

Second, **robots.txt is a request, not an access control**. Compliant crawlers honour it; nothing forces them to. It cannot remove pages from a training set collected before you added the rule.

There is a third limit that is specific to answer engines. When a person pastes a URL into ChatGPT or asks Perplexity a question, the system fetches the page on that person's behalf. OpenAI, Perplexity, Meta and Google all document that these **user-triggered fetchers** may ignore robots.txt. You can still recognise and rate-limit them by IP, but you cannot opt out of them with a text file.

> **Warning: A named group replaces the wildcard**
>
> Adding `User-agent: OAI-SearchBot` with `Allow: /` and nothing else silently grants that bot access to `/admin/`, `/api/` and anything else your `*` group disallows. Copy the full rule set into each named group.

## The AI user agents, vendor by vendor

Every major vendor runs more than one agent, and most split them the same way: a training crawler, a search indexer and a user-triggered fetcher. The names below are what you put in robots.txt.

### OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

OpenAI's crawler documentation describes **GPTBot** as the crawler for content "that may be used in training our generative AI foundation models", and **OAI-SearchBot** as the agent "used to surface websites in search results in ChatGPT's search features". Both honour robots.txt. **ChatGPT-User** is "used for certain user actions in ChatGPT and Custom GPTs", and because those actions are initiated by a user, OpenAI says robots.txt rules may not apply. IP ranges for each are published at `openai.com/gptbot.json`, `openai.com/searchbot.json` and `openai.com/chatgpt-user.json`.

### Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic's support article follows the same pattern. **ClaudeBot** collects web content "that could potentially contribute to" model training. **Claude-SearchBot** "navigates the web to improve search result quality", and **Claude-User** fetches pages when "individuals ask questions to Claude". Anthropic states that all three honour "do not crawl" directives in robots.txt, and publishes IP ranges at `claude.com/crawling/bots.json`. It also advises against blocking by IP address, because a blocked crawler cannot read your robots.txt and so cannot record the opt-out.

### Perplexity: PerplexityBot and Perplexity-User

Perplexity's documentation lists **PerplexityBot**, "designed to surface and link websites in search results on Perplexity", which respects robots.txt, and **Perplexity-User**, which visits a page when a user asks a question. The docs are explicit: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." IP ranges are published as `perplexitybot.json` and `perplexity-user.json` on perplexity.com. Perplexity states that neither agent is used to collect content for training foundation models, and documents no training crawler.

### Google: Googlebot powers AI Overviews, Google-Extended does not

Google's guidance on AI features says that "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners". AI Overviews and AI Mode draw on the ordinary Search index, so the only way to keep a page out of them is to keep it out of Search, or to limit its snippets with `nosnippet`, `data-nosnippet`, `max-snippet` or `noindex`.

**Google-Extended** is a robots.txt token, not a crawler. Google's common crawlers page describes it as the control publishers use "to manage whether content Google crawls from their sites may be used for training future generations of Gemini models" and for grounding in Gemini apps and Vertex AI, and states that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". Disallowing it costs you nothing in Search or AI Overviews.

Google's separate page on [user-triggered fetchers](https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers), which includes tools such as Google-Agent and Gemini Notebook, says plainly that "because the fetch was requested by a user, these fetchers generally ignore robots.txt rules".

### Apple, Common Crawl, Amazon, Meta and ByteDance

- **Applebot** crawls for Spotlight, Siri and Safari search and, per Apple, also for training its foundation models. **Applebot-Extended** is an opt-out token that "does not crawl webpages"; Apple states that pages which disallow it "can still be included in search results".
- **CCBot** is Common Crawl's crawler, whose open archive is a common raw material for training sets. It honours `User-agent: CCBot` and Common Crawl warns that it is "aware of crawlers falsely identifying themselves as CCBot", publishing its real ranges at `index.commoncrawl.org/ccbot.json`.
- **Amazonbot** is used, in Amazon's words, "to improve our products and services" and "may be used to train Amazon AI models". It respects robots.txt and Amazon publishes its IP addresses.
- **meta-externalagent** crawls "for use cases such as training foundation AI models or improving products by indexing content directly". Meta's crawler page also lists **meta-webindexer** for Meta AI search and **meta-externalfetcher**, which fetches links at a user's request and may bypass robots.txt.
- **Bytespider** is ByteDance's crawler. ByteDance does not publish crawler documentation comparable to the vendors above, so there is no vendor statement on robots.txt compliance or a published IP list to verify against. A robots.txt rule is still worth adding, but treat it as unverified.

| Agent | Vendor | Job | robots.txt |
| --- | --- | --- | --- |
| GPTBot | OpenAI | training | honoured |
| OAI-SearchBot | OpenAI | search index | honoured |
| ChatGPT-User | OpenAI | user fetch | may not apply |
| ClaudeBot | Anthropic | training | honoured |
| Claude-SearchBot | Anthropic | search index | honoured |
| Claude-User | Anthropic | user fetch | honoured |
| PerplexityBot | Perplexity | search index | honoured |
| Perplexity-User | Perplexity | user fetch | generally ignored |
| Googlebot | Google | Search, AI Overviews, AI Mode | honoured |
| Google-Extended | Google | Gemini training and grounding opt-out | token only |
| Applebot-Extended | Apple | training opt-out | token only |
| CCBot | Common Crawl | open crawl archive | honoured |
| meta-externalagent | Meta | training and indexing | honoured |
| Bytespider | ByteDance | training | unverified |

## Training versus citation: the trade-off

The reason the vendors split their agents is that the two uses have opposite economics for you.

A **search indexer** fetches your page so that an answer engine can retrieve it later, quote a passage and attach a [citation](/glossary/citation). Blocking it removes you from that engine's answers, immediately and completely. There is no version of Answer Engine Optimization that works with OAI-SearchBot, Claude-SearchBot or PerplexityBot disallowed.

A **training crawler** fetches your page so that the text can shape model weights. The model may later reproduce your facts and phrasing without any attribution, and you cannot measure it. Whether that is reach or uncompensated use is a policy question, and it is independent of whether you want citations.

The practical position for a site that exists to be cited is therefore: allow every search agent and Googlebot, decide the training agents on policy rather than habit, and accept that user-triggered fetchers will arrive regardless.

> **Note: What this site does**
>
> Citable allows every documented AI agent, including the training crawlers, and names each one explicitly in its [robots.txt](/robots.txt). Naming bots that `*` already allows changes nothing technically; it states intent.

## A robots.txt that allows citation and opts out of training

The file below keeps a site fully visible to answer engines while opting out of training and archive crawlers. Adjust the `Disallow` lines to your own private paths and repeat them in every group.

```txt
# Everyone, including search and answer engines
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

# AI search indexers: same rules as everyone else
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Applebot
Allow: /
Disallow: /api/
Disallow: /admin/

# Training and archive crawlers: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

# Training opt-out tokens that do not crawl
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml
```

Googlebot is intentionally absent from the named groups: it falls through to `*` and inherits the same rules. Amazonbot is absent because Amazon documents it as serving products as well as possible training; put it in whichever group matches your policy.

In a Next.js App Router project the same policy can be generated from `app/robots.ts`, which keeps the bot list in one place next to the rest of your site configuration:

```ts
import type { MetadataRoute } from "next";

const PRIVATE = ["/api/", "/admin/"];
const SEARCH_BOTS = ["OAI-SearchBot", "Claude-SearchBot", "PerplexityBot", "Applebot"];
const TRAINING_BOTS = ["GPTBot", "ClaudeBot", "CCBot", "meta-externalagent", "Bytespider"];
const OPT_OUT_TOKENS = ["Google-Extended", "Applebot-Extended"];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: "*", allow: "/", disallow: PRIVATE },
      { userAgent: SEARCH_BOTS, allow: "/", disallow: PRIVATE },
      { userAgent: [...TRAINING_BOTS, ...OPT_OUT_TOKENS], disallow: "/" },
    ],
    sitemap: "https://example.com/sitemap.xml",
  };
}
```

**Before you ship a robots.txt change**

- [ ] Every named group repeats the Disallow lines from the \`\*\` group
- [ ] OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot can reach every page you want cited
- [ ] Google-Extended and Applebot-Extended are treated as opt-out tokens, not crawlers
- [ ] The Sitemap line points at an absolute URL that returns 200
- [ ] The file is served at exactly \`/robots.txt\` with \`Content-Type: text/plain\`
- [ ] You have fetched it from production and read it back, not just the source
- [ ] Your /llms.txt and sitemap only list URLs the allowed bots can actually fetch

## Verifying that a bot is who it says it is

A user-agent string is a request header; anyone can send `GPTBot/1.4`, and scrapers do because sites whitelist it. Common Crawl's own page notes impersonation of CCBot. Check the source address before you act on a bot's identity, whether that is serving a Markdown variant or exempting it from rate limits.

Every vendor here except ByteDance publishes its crawler addresses as JSON lists of CIDR prefixes; fetch them on a schedule and match client IPs at the edge.

```bash
# Published address lists (all return JSON with CIDR prefixes)
curl -s https://openai.com/searchbot.json
curl -s https://openai.com/gptbot.json
curl -s https://claude.com/crawling/bots.json
curl -s https://www.perplexity.com/perplexitybot.json
curl -s https://index.commoncrawl.org/ccbot.json
curl -s https://developers.google.com/static/crawling/ipranges/common-crawlers.json
```

For Googlebot, Google also documents [reverse DNS verification](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot): look up the PTR record for the IP, confirm the hostname ends in `googlebot.com`, `google.com` or `googleusercontent.com`, then resolve that hostname forward and check it returns the original IP. Google's example, `host 66.249.66.1`, resolves to a `crawl-66-249-66-1.googlebot.com` name.

If a request claims to be a documented bot and its IP is not in the published list, it is not that bot. Log it, rate-limit it, and do not count it in your [measurement](/guides/measuring-ai-visibility-and-citations) of AI crawler traffic.

> **Tip: Where to go next**
>
> Once the right bots can reach you, give them something easy to read: [llms.txt: what it is and how to write one](/guides/llms-txt-what-it-is-and-how-to-write-one) covers the curated index, and [Serving Markdown to LLMs](/guides/serving-markdown-to-llms) covers delivering the same content without the HTML overhead.

## Frequently asked questions

### Does blocking GPTBot stop my site appearing in ChatGPT search?

No. OpenAI documents GPTBot as the crawler for content that may be used to train its models and OAI-SearchBot as the agent that surfaces websites in ChatGPT search. They are separate user agents with separate robots.txt groups, so a Disallow for GPTBot leaves ChatGPT search untouched as long as OAI-SearchBot is still allowed.

### Will Google-Extended keep my pages out of AI Overviews?

No. Google states that AI is built into Search, so the robots.txt rules for Googlebot are the control for AI Overviews and AI Mode. Google-Extended is a separate product token that manages whether crawled content may be used to train future Gemini models or for grounding in Gemini apps; Google says it does not affect inclusion or ranking in Search. Use nosnippet, max-snippet or noindex to limit what appears in AI features.

### Do AI crawlers actually obey robots.txt?

The documented crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, Amazon and Meta state that they honour robots.txt. The exceptions are user-triggered fetchers such as ChatGPT-User, Perplexity-User, meta-externalfetcher and Google's user-triggered fetchers, which the vendors say may ignore robots.txt because a person asked for the page. Bytespider has no comparable public documentation, so treat its compliance as unverified.

### Should I block AI crawlers at all if I want to be cited?

Only the ones that do not produce citations. Search indexers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot are how answer engines find pages to cite; blocking them removes you from those answers. Training crawlers such as GPTBot, ClaudeBot and CCBot feed model weights, which may or may not credit you later. Most sites that want citations allow the search agents and decide the training agents case by case.

### How do I know a request really comes from the bot it claims to be?

Check the source IP, not the user-agent string, which anyone can copy. OpenAI, Anthropic, Perplexity, Google, Common Crawl and Amazon publish IP range lists in JSON; Google also documents reverse DNS verification against googlebot.com, google.com or googleusercontent.com. Common Crawl explicitly warns that other crawlers falsely identify themselves as CCBot.

## Sources

1. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — OpenAI (2025)
2. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — Anthropic (2025)
3. [Perplexity crawlers](https://docs.perplexity.ai/guides/bots) — Perplexity (2025)
4. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Google Search Central (2025)
5. [Google's common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) — Google Search Central (2025)
6. [Google's user-triggered fetchers](https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers) — Google Search Central (2025)
7. [About Applebot](https://support.apple.com/en-us/119829) — Apple (2025)
8. [CCBot](https://commoncrawl.org/ccbot) — Common Crawl (2025)
9. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) — IETF (2022)

## Related guides

- [llms.txt: what it is, how to write one, and whether it helps](https://citable.wiki/guides/llms-txt-what-it-is-and-how-to-write-one)
- [Serving Markdown to LLMs: content negotiation and .md endpoints](https://citable.wiki/guides/serving-markdown-to-llms)
- [Measuring AI visibility: how to track citations in ChatGPT, Perplexity and AI Overviews](https://citable.wiki/guides/measuring-ai-visibility-and-citations)
- [What is Answer Engine Optimization (AEO)?](https://citable.wiki/guides/what-is-answer-engine-optimization)

---

Source: https://citable.wiki/guides/controlling-ai-crawlers-with-robots-txt
Author: Endrit Krasniqi
More: https://citable.wiki/llms.txt