AI Search Crawlers vs Training Crawlers

AI search crawlers and model-training crawlers may serve different provider purposes and should not be treated as one “AI bot” setting. Providers can also document user-triggered retrieval agents, ordinary search crawlers, and product-level data-use controls. Review each token against your business policy and the current provider documentation.

The five categories

Search discovery

A search-discovery crawler finds public pages that may support search or answer features. Allowing it can remove one technical policy barrier, but it does not guarantee indexing, selection, quotation, or citation.

Examples documented by providers include OAI-SearchBot, Claude-SearchBot, and PerplexityBot.

Model development or training

A training-related crawler collects public content that may be used for model development or training under the provider’s stated terms and controls.

Examples include GPTBot and ClaudeBot. A site may intentionally restrict these while allowing separate search-discovery tokens.

User-triggered retrieval

A user-triggered agent fetches a page because a user requested an action, opened a link, or asked a product to retrieve current information.

Examples include ChatGPT-User, Claude-User, and Perplexity-User. Providers may document different robots behaviour for these agents, so do not infer one universal rule.

Ordinary search crawling

Traditional search crawlers such as Googlebot remain relevant because AI answer features can rely on ordinary search indexes and systems. Blocking a dedicated training token does not necessarily control ordinary search discovery.

Data-use controls

Some product tokens express how content discovered through another crawler may be used. They may not identify a separate HTTP crawler.

Google-Extended is the important example: Google documents it as a product token controlling specified Gemini training and grounding uses. It has no separate HTTP user-agent string and does not determine Google Search inclusion or ranking.

Current provider-purpose matrix

Provider	Search discovery	Training or model development	User-triggered retrieval	Other control
OpenAI	OAI-SearchBot	GPTBot	ChatGPT-User	Review provider controls and terms separately
Anthropic	Claude-SearchBot	ClaudeBot	Claude-User	Current documentation says the named bots honour robots.txt
Perplexity	PerplexityBot	No foundation-model training purpose documented for PerplexityBot	Perplexity-User	Provider documents different robots behaviour for user-requested fetches
Google	Googlebot supports ordinary Search crawling	Not represented by Google-Extended as a separate crawler	Product-specific fetchers may exist	Google-Extended is a product token, not a separate HTTP user agent

This table is a dated summary, not a permanent registry. Follow each linked provider source before deploying a policy.

Why “allow AI” is too imprecise

A single broad instruction can hide several different decisions:

• Should public pages be discoverable for AI-assisted search?
• May a provider use public content for model development?
• Should a user-requested fetch be allowed?
• Should ordinary search indexing remain available?
• Does the site want to limit a documented product use without blocking Search?
• Are some paths public while others require real access control?

A policy tool should ask these questions instead of rewarding the maximum number of Allow rules.

A balanced policy example

The following expresses one possible intent: allow named search-discovery crawlers while restricting named training crawlers.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /

This is not a universal recommendation. A publisher may choose maximum public access, review-only handling, or stricter restrictions based on content rights, commercial strategy, legal advice, and operational needs.

Read How to Allow AI Search While Blocking Model Training for implementation detail.

OpenAI: three distinct paths

OpenAI currently documents:

• OAI-SearchBot for search discovery connected to ChatGPT search;
• GPTBot for content that may be used in model development or training;
• ChatGPT-User for supported user-triggered retrieval.

A rule for one token does not automatically represent the other two. See GPTBot vs OAI-SearchBot.

Anthropic: separate search, training, and user agents

Anthropic currently documents:

• Claude-SearchBot for search-related discovery;
• ClaudeBot for content that may contribute to model training;
• Claude-User for user-initiated retrieval.

Anthropic’s current guidance says the named bots honour robots.txt. Store the verification date because product descriptions and network details may change.

Perplexity: background discovery and user-requested retrieval

Perplexity currently documents:

• PerplexityBot for surfacing and linking websites in search results, and states it is not used to crawl content for foundation-model training;
• Perplexity-User for user-requested retrieval, with different documented robots behaviour.

Do not use Perplexity-User as evidence that a site has granted broad background crawling permission.

Google: Search crawling and Google-Extended are different controls

Google documents Googlebot for ordinary Search crawling. Google-Extended is a separate product token that publishers can use to manage specified Gemini training and grounding uses of content discovered by Google.

Important boundaries:

• Google-Extended has no separate HTTP user-agent string;
• it should not be authenticated or monitored as though it independently crawls pages;
• Google states that it does not affect Google Search inclusion or ranking;
• it is not a complete proxy for eligibility in every AI feature.

Policy is not delivery

A robots rule is only one layer. The page may still return:

• a bot-management challenge;
• HTTP 403 or 429;
• a redirect loop;
• an empty application shell;
• a consent wall;
• a timeout;
• a different response on a subdomain.

Use How to Check AI Crawler Access to inspect both policy and public delivery.

Policy is not security

robots.txt is public and voluntary. It is not suitable for protecting:

• customer data;
• paid or licensed content;
• private documentation;
• administrative pages;
• unpublished material;
• credentials or secrets.

Use authentication, authorization, network controls, and appropriate application security.

Decision checklist

For each provider and path, record:

1. the exact token;
2. documented purpose;
3. primary source and verification date;
4. desired business policy;
5. effective robots rule;
6. public response behaviour;
7. important limitations;
8. owner and review date;
9. post-change re-check result.

What a good audit should report

A good audit should say:

• what was documented;
• what was observed;
• how the rule was parsed;
• whether a simulated response differed;
• whether the result matches a declared policy;
• what cannot be concluded.

It should not label all training restrictions as failures or claim that maximum crawler access guarantees visibility.

Run the OpenForBots audit to review documented crawler purposes and deployed policy separately.