OpenForBotsAI discovery readiness
Run website auditRun audit

How to Allow AI Search While Blocking Model Training

A provider-specific robots.txt pattern for allowing named AI search crawlers while restricting named model-training crawlers, with separate treatment for user-triggered fetchers and data-use controls.

You can allow named AI search crawlers while restricting named model-training crawlers because major providers document separate tokens for different purposes. A common balanced pattern allows OAI-SearchBot, Claude-SearchBot, and PerplexityBot, while disallowing GPTBot and ClaudeBot.

That pattern does not mean every user-triggered agent can be controlled the same way. Current provider documentation differs materially for ChatGPT-User, Claude-User, and Perplexity-User.

Start with the policy goal

A balanced policy usually means:

  • public pages may be discovered by named search crawlers;
  • provider-documented training crawlers are restricted when that matches the owner’s policy;
  • user-triggered retrieval is reviewed provider by provider;
  • ordinary search crawling remains governed by the site’s existing search policy;
  • private content stays behind real access controls;
  • no allow rule is treated as a promise of indexing or citation.

This is a business and governance choice, not a universal best practice.

A cautious multi-provider example

# OpenAI Search
User-agent: OAI-SearchBot
Allow: /

# OpenAI training crawl
User-agent: GPTBot
Disallow: /

# Anthropic search
User-agent: Claude-SearchBot
Allow: /

# Anthropic training crawl
User-agent: ClaudeBot
Disallow: /

# Perplexity search
User-agent: PerplexityBot
Allow: /

# Conventional Google Search
User-agent: Googlebot
Allow: /

# Restrict the documented Google-Extended data uses
User-agent: Google-Extended
Disallow: /

Review the current provider documentation before deploying any copied example. Tokens, descriptions, network ranges, and control semantics can change.

User-triggered retrieval is provider-specific

ChatGPT-User

OpenAI documents ChatGPT-User for certain actions initiated by ChatGPT or Custom GPT users. OpenAI says that because these actions are user-initiated, robots.txt rules may not apply.

Therefore, a block such as:

User-agent: ChatGPT-User
Disallow: /

can be recorded as the site’s declared robots policy, but it should not be presented as guaranteed enforcement for every ChatGPT-User request. Use OAI-SearchBot for ChatGPT Search opt-outs and automatic crawling policy.

Claude-User

Anthropic explicitly says its named bots honor industry-standard robots.txt directives, including Claude-User. A site can therefore set a separate Claude user-triggered retrieval policy, for example:

User-agent: Claude-User
Allow: /

Anthropic says disabling Claude-User prevents its system from retrieving the site’s content in response to a user query, which may reduce visibility for user-directed web search.

Perplexity-User

Perplexity documents Perplexity-User for user-requested retrieval and says it generally ignores robots.txt because the user requested the fetch. Do not present a matching robots rule as a reliable Perplexity-User enforcement control.

The practical lesson is simple: classify by provider and purpose, then apply that provider’s documented control semantics.

Provider distinctions

OpenAI

  • OAI-SearchBot: automatic Search discovery for ChatGPT Search.
  • GPTBot: crawl that may supply training data for OpenAI’s generative AI foundation models.
  • ChatGPT-User: user-triggered retrieval; robots.txt may not apply.

Anthropic

  • Claude-SearchBot: search-related discovery.
  • ClaudeBot: content that may contribute to model training.
  • Claude-User: user-directed retrieval.

Anthropic says all three named bots honor robots.txt and also documents support for the non-standard Crawl-delay extension.

Perplexity

  • PerplexityBot: search discovery and linked results; Perplexity says it is not used to crawl content for foundation-model training.
  • Perplexity-User: user-requested retrieval; Perplexity says it generally ignores robots.txt.

Perplexity publishes current IP-range endpoints for both paths. If you use network controls, verify against the live provider data rather than a copied range.

Google

  • Googlebot: ordinary Search crawling.
  • Google-Extended: a product token controlling documented Gemini training and grounding uses; it has no separate HTTP user-agent string.

Google states that Google-Extended does not affect Google Search inclusion or ranking.

Preserve ordinary search access intentionally

A broad wildcard block can undermine the policy you intended:

User-agent: *
Disallow: /

If conventional search visibility matters, review how wildcard and token-specific groups apply. Do not assume that a few AI-specific Allow lines fix every crawler or path interaction.

Apply the policy to every relevant origin

A robots.txt file applies to the origin where it is served. Review the apex domain, www, documentation subdomains, shops, regional hosts, and any other public origin that matters.

Keep private content private

Never rely on robots.txt to protect customer records, account pages, private documentation, licensed content that requires access control, unpublished material, or secrets. RFC 9309 explicitly treats robots.txt as crawler guidance rather than authorization.

Common implementation errors

Copying a stale token list

Provider documentation changes. Store a verification date and keep links to primary sources.

Treating all user-triggered agents alike

This is now demonstrably wrong: OpenAI, Anthropic, and Perplexity currently document different robots semantics for their user-triggered agents.

Confusing Google-Extended with a crawler

Google-Extended has no separate HTTP user-agent. It is a data-use control, not a standalone crawler identity.

Treating a training block as a visibility failure

Blocking GPTBot or ClaudeBot may be the site’s intended policy. It should not automatically reduce an opaque readiness score.

Ignoring WAF or CDN behavior

A crawler can be allowed in robots.txt and still receive a challenge, 403, 429, timeout, or other unusable response.

Verification checklist

  1. Confirm each token and purpose against the current provider source.
  2. Record whether the provider actually documents robots.txt as an applicable control for that token.
  3. Fetch /robots.txt from the correct origin.
  4. Resolve the rule for / and important subpaths.
  5. Keep policy evidence separate from real response delivery.
  6. Confirm ordinary search crawlers retain the intended access.
  7. Treat user-triggered agents according to their provider-specific semantics.
  8. Re-run the audit after deployment.

What this configuration does not guarantee

It does not guarantee indexing, citation, ranking, recommendation, increased traffic, successful user-triggered retrieval, or exclusion of content already obtained through other sources.

Next step

Run the crawler-policy audit to inspect the deployed automatic-crawler rules, then review user-triggered agents using the provider-specific caveats above.

Related context

Useful context for this page