Definition
AI crawler
Also known as: AI bot, LLM crawler
An AI crawler is an automated agent that fetches web pages on behalf of an AI system and identifies itself with its user-agent string, such as GPTBot, ClaudeBot or PerplexityBot. Vendors separate crawlers by purpose — collecting training data, building a search index, or fetching a page a user asked about — so that training and search access can be allowed or blocked independently in robots.txt.
The first AI crawlers, such as Common Crawl's CCBot and OpenAI's GPTBot, existed to gather training data. As answer engines moved to live retrieval, vendors split their traffic into three roles, each with its own user agent:
| Role | OpenAI | Anthropic | Perplexity |
|---|---|---|---|
| Training data | GPTBot | ClaudeBot | — |
| Search index | OAI-SearchBot | Claude-SearchBot | PerplexityBot |
| User-triggered fetch | ChatGPT-User | Claude-User | Perplexity-User |
Training and search crawlers honour robots.txt. User-triggered fetchers are different: OpenAI and Perplexity both state that robots.txt rules may not apply when a person initiated the request, while Anthropic says Claude-User does respect them.
Google is the exception to the pattern. AI Overviews and AI Mode are served from the ordinary Googlebot index, and Google-Extended is a robots.txt token with no user agent of its own; it controls whether crawled content is used for Gemini training and grounding, and does not affect inclusion in Search (Google crawler documentation).
The decision for a publisher is therefore which roles to allow. A common compromise is to block training crawlers and allow search crawlers, which keeps pages citable without contributing to model weights:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Blocking everything makes citation impossible. Controlling AI crawlers with robots.txt lists the current user agents and their documented behaviour.
Frequently asked questions
Does blocking AI crawlers stop my content appearing in AI answers?
Only for the crawlers you block. Blocking GPTBot keeps pages out of OpenAI's training data but not out of ChatGPT search, which uses OAI-SearchBot. Google's AI Overviews are built from Googlebot's index, and the Google-Extended token only controls training and grounding in other Google systems, not inclusion in Search.