AI Search Crawlers vs Training Crawlers

AI search crawlers and model-training crawlers are not one “AI bot” setting. Providers also document user-triggered retrieval agents, ordinary search crawlers, and product-level data-use controls. The correct policy depends on both the token’s purpose and the provider’s documented control semantics.

The main categories

Search discovery

These crawlers find public pages for search or answer features. Current examples include OAI-SearchBot, Claude-SearchBot, and PerplexityBot.

Allowing a search crawler removes one documented policy barrier. It does not guarantee indexing, selection, quotation, or citation.

Model development or training

These crawlers collect public content that may contribute to model training under the provider’s stated terms and controls. Current examples include GPTBot and ClaudeBot.

A publisher can intentionally restrict these while allowing separate search-discovery crawlers.

User-triggered retrieval

These agents fetch content because a person asked a product to retrieve or use a page. Current examples include ChatGPT-User, Claude-User, and Perplexity-User.

This category is where a generic robots.txt rule becomes unsafe. The providers currently document different behavior:

• OpenAI: ChatGPT-User is user-triggered and OpenAI says robots.txt rules may not apply.
• Anthropic: Anthropic says its named bots honor robots.txt, including Claude-User.
• Perplexity: Perplexity says Perplexity-User generally ignores robots.txt because the user requested the fetch.

So “user-triggered” describes purpose, not one shared access-control model.

Ordinary search crawling

Traditional crawlers such as Googlebot remain important because AI answer features can depend on ordinary search systems and indexes. Blocking a dedicated training crawler does not necessarily affect conventional Search discovery.

Data-use controls

Some tokens express how content found through another crawler may be used rather than identifying a separate HTTP crawler.

Google-Extended is the clearest example. Google documents it as a product token controlling specified Gemini training and grounding uses. It has no separate HTTP user-agent and does not determine Google Search inclusion or ranking.

Current provider-purpose matrix

Provider	Search discovery	Training/model development	User-triggered retrieval	Important control note
OpenAI	OAI-SearchBot	GPTBot	ChatGPT-User	ChatGPT-User robots rules may not apply
Anthropic	Claude-SearchBot	ClaudeBot	Claude-User	Anthropic says all named bots honor robots.txt
Perplexity	PerplexityBot	No foundation-model training purpose documented for PerplexityBot	Perplexity-User	Perplexity-User generally ignores robots.txt
Google	Googlebot supports ordinary Search crawling	Google-Extended is not a separate crawler	Product-specific fetchers may exist	Google-Extended is a data-use token, not an HTTP user agent

This is a dated summary. Re-check the primary provider source before changing production policy.

Why “allow AI” is too imprecise

A useful policy asks separate questions:

• Should public pages be discoverable for AI-assisted search?
• May a provider crawl public content for model training?
• What control does the provider document for user-requested retrieval?
• Should ordinary search indexing remain available?
• Does a data-use token affect Search or only downstream use?
• Are some paths public while others require real authentication?

A readiness tool should preserve those distinctions rather than reward the maximum number of Allow directives.

A balanced automatic-crawler example

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /

This is an example of one policy goal, not a universal recommendation.

Do not automatically append equivalent rules for ChatGPT-User, Claude-User, and Perplexity-User; review each provider’s documented behavior first.

OpenAI

OpenAI currently documents:

• OAI-SearchBot for ChatGPT Search;
• GPTBot for content that may be used to train its generative AI foundation models;
• ChatGPT-User for certain user-triggered actions.

OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT Search answers, though they may still appear as navigational links. It also says robots.txt rules may not apply to ChatGPT-User because those actions are initiated by a user.

See GPTBot vs OAI-SearchBot.

Anthropic

Anthropic currently documents:

• Claude-SearchBot for search relevance and accuracy;
• ClaudeBot for content that could contribute to model training;
• Claude-User for retrieval at a user’s direction.

Anthropic says its named bots honor industry-standard robots.txt directives and also supports the non-standard Crawl-delay extension.

Perplexity

Perplexity currently documents:

• PerplexityBot for surfacing and linking websites in search results and says it is not used to crawl content for foundation-model training;
• Perplexity-User for user-requested retrieval and says it generally ignores robots.txt.

Perplexity publishes separate IP-range references for legitimate requests. Treat those ranges as time-sensitive source data.

Google

Google documents Googlebot for Search crawling. Google-Extended is a separate product token that can control specified Gemini training and grounding uses of content discovered by Google.

Important boundaries:

• Google-Extended has no separate HTTP user-agent;
• it should not be monitored as though it independently crawls pages;
• Google says it does not affect Search inclusion or ranking.

Policy is not delivery

Even when robots.txt applies and allows a token, the requested page may still return a WAF challenge, 403, 429, redirect loop, timeout, empty shell, or other unusable response.

Inspect policy and delivery as separate evidence layers.

Policy is not security

Robots.txt is public and voluntary. Use authentication, authorization, and appropriate network/application controls for private data, paid content, unpublished material, account pages, or secrets.

Decision checklist

For each provider and path, record:

1. exact token;
2. documented purpose;
3. whether robots.txt is documented as applicable to that token;
4. primary source and verification date;
5. desired business policy;
6. effective rule where robots.txt applies;
7. actual public response behavior;
8. owner and review date;
9. fresh post-change result.

What a good audit should report

A good audit separates:

• provider-documented purpose and control behavior;
• observed robots.txt content;
• parsed rule result;
• simulated delivery evidence;
• site-owner intent;
• limitations and unknowns.

It should never convert a user-triggered token into a confident “allowed” or “blocked” provider verdict when the provider says robots.txt may not apply or is generally ignored.

Run the OpenForBots audit to review documented crawler purposes and deployed policy separately.