Concepts / AI Search Crawlers vs Training Crawlers

AI Search Crawlers vs Training Crawlers

Why AI search, model training, user-triggered retrieval, and data-use controls are separate website-policy decisions.

AI search crawlers and model-training crawlers perform different jobs and should not be treated as one “AI bot” setting. Search crawlers help a provider discover content for search or answer experiences. Training crawlers collect content that may contribute to future model development. User-triggered fetchers retrieve pages at a person’s request, while some tokens—such as Google-Extended—act as data-use controls rather than separate crawlers.

Why the distinction matters

A site owner may reasonably want to:

allow AI-assisted search discovery;
block potential model-training collection;
allow public pages to be retrieved when a user supplies a link;
keep private or paid content behind real authentication;
inspect the current policy without receiving a recommendation.

Those are different goals. A single “allow AI” or “block AI” label hides the actual choice.

Four policy categories

Category	Typical purpose	Examples	Main policy question
Search or discovery crawler	Finds and indexes public pages for a provider’s search experience	OAI-SearchBot, Claude-SearchBot, PerplexityBot	Should this provider discover public content for search?
Model-training crawler	Collects public content that may contribute to model development	GPTBot, ClaudeBot	Should this provider crawl future content for training-related use?
User-triggered retrieval	Fetches a page in response to a user’s request	ChatGPT-User, Claude-User, Perplexity-User	Should users be able to retrieve this public page through the product?
Data-use control	Expresses how already crawled content may be used by a product	Google-Extended	Should covered Google-crawled content be available for the documented Gemini uses?

The examples above reflect provider documentation at the last-verified date. They are not a universal taxonomy imposed by a web standard.

OpenAI: OAI-SearchBot, GPTBot, and ChatGPT-User

OpenAI documents separate controls for ChatGPT search discovery and potential model-training use. For a balanced policy, a site can allow OAI-SearchBot while disallowing GPTBot.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

ChatGPT-User represents a separate user-triggered retrieval path. It should not be assumed to behave like background indexing.

See GPTBot vs OAI-SearchBot for a focused implementation guide.

Anthropic: Claude-SearchBot, ClaudeBot, and Claude-User

Anthropic documents three bots with separate purposes:

Claude-SearchBot for search-result quality;
ClaudeBot for content that may contribute to model training;
Claude-User for user-initiated retrieval.

A rule for ClaudeBot does not automatically become a rule for Claude-SearchBot. Each robots.txt product token needs to be considered deliberately.

Perplexity: PerplexityBot and Perplexity-User

Perplexity documents PerplexityBot as its search-discovery crawler and states that it is not used to crawl content for foundation-model training. It also documents Perplexity-User for user actions.

Perplexity publishes current IP information for these agents. That can support request verification, but the IP data is time-sensitive and should be read from the provider’s current endpoint rather than copied permanently into a WAF rule without maintenance.

Google: Googlebot and Google-Extended

Googlebot crawls for Google Search, including Google’s Search features. Google-Extended is different: Google documents it as a standalone robots.txt product token with no separate HTTP user-agent string.

Google says that Google-Extended does not affect a site’s inclusion or ranking in Google Search. Blocking it should therefore not be reported as a Google Search failure.

A balanced multi-provider example

The following pattern expresses a preference to allow named search crawlers while restricting named training or data-use controls:

# Search discovery
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Model-training collection
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Google data-use control
User-agent: Google-Extended
Disallow: /

This is an example, not a universal recommendation. A site should review every token against its own legal, commercial, security, and visibility goals.

Common mistakes

Blocking only a provider’s most familiar token

A rule for GPTBot does not define the OAI-SearchBot policy. A rule for ClaudeBot does not define the Claude-SearchBot policy.

Treating robots.txt as security

RFC 9309 describes robots.txt as a protocol that compliant crawlers are requested to honour. It is not authentication. Sensitive content must be protected with appropriate access controls.

Assuming “allowed” means “included”

Crawler access is only one precondition. A provider may still decide not to index, retrieve, cite, rank, or recommend the page.

Assuming a user-agent string proves identity

User-agent headers can be spoofed. Use provider-supported request-verification methods when authenticity matters.

How OpenForBots evaluates the policy

OpenForBots reads the site’s published robots.txt file and evaluates each registry token separately. It reports:

the exact matching group and rule;
the documented purpose of the token;
whether the observed policy matches the owner’s selected goal;
what the result does not prove;
the specific crawler reference and implementation guide.

A training-crawler block is not automatically negative under the balanced policy goal.

Next step

Review the crawler-policy audit inputs to see how the selected policy goal changes the interpretation of the same robots.txt rules.