Guides / GPTBot vs OAI-SearchBot

GPTBot vs OAI-SearchBot

The difference between OpenAI's model-training crawler and ChatGPT search crawler, with separate robots.txt examples and verification steps.

GPTBot and OAI-SearchBot are separate OpenAI crawler tokens with different purposes. OpenAI documents GPTBot for content that may be used in model development or training, while OAI-SearchBot is relevant to discovery for ChatGPT search. Blocking GPTBot does not automatically block OAI-SearchBot, and allowing either token does not guarantee that a page will be indexed or cited.

Quick comparison

Token	Documented purpose	Direct search-visibility relevance	Typical balanced policy
`OAI-SearchBot`	Discover content for ChatGPT search summaries and snippets	Direct	Allow public pages intended for discovery
`GPTBot`	Crawl content that may contribute to model development or training	Not direct	Allow or block according to the site’s training policy
`ChatGPT-User`	Retrieve content in supported user-triggered workflows	Indirect	Decide separately from search and training

The provider’s documentation is the source of truth. OpenForBots stores these as separate registry records so a report cannot collapse them into one “OpenAI bot” verdict.

Recommended configuration for a balanced policy

This pattern allows OpenAI’s search crawler while expressing a model-training opt-out:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

The rule expresses a preference. It does not guarantee ChatGPT search inclusion and it does not remove content already obtained through other lawful or authorised sources.

Maximum-access configuration

A site that intentionally wants to allow both documented crawler purposes can use:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

Do not choose this automatically because a readiness tool assigns more points. Training access is a policy decision involving the site owner’s commercial, legal, and content strategy.

Restrict both crawlers

A site that does not want either crawler to access any path can use:

User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Disallow: /

This may reduce the site’s eligibility for ChatGPT search summaries and snippets because OpenAI advises publishers not to block OAI-SearchBot when they want that visibility.

Restrict only part of a site

A site may want public marketing and documentation pages discoverable while excluding an internal or licensed-content area:

User-agent: OAI-SearchBot
Allow: /
Disallow: /member-library/

User-agent: GPTBot
Disallow: /

robots.txt is public and is not a security boundary. Private or paid content must still require authentication or another effective access control.

How robots.txt matching works

Under the Robots Exclusion Protocol, a crawler identifies the group that matches its product token and applies the most specific matching path rule. If a token-specific group exists, do not assume that a wildcard group will override it in the way you intended.

Keep rules easy to review:

use the documented token exactly;
avoid duplicate groups unless you understand how they are combined;
keep important allow and disallow paths explicit;
test the deployed file at /robots.txt rather than only reviewing a local source file;
repeat the policy on every relevant subdomain.

What about ChatGPT-User?

ChatGPT-User represents a separate user-triggered retrieval category. A site’s policy may allow search discovery while blocking a user-triggered fetch, or make the opposite choice.

Do not assume that a rule for OAI-SearchBot or GPTBot automatically controls ChatGPT-User.

Common mistakes

Blocking GPTBot and expecting ChatGPT search to disappear

GPTBot is not the ChatGPT search crawler. OpenAI documents OAI-SearchBot separately.

Allowing OAI-SearchBot and claiming guaranteed citations

Access is only a technical precondition. OpenAI may still decide not to retrieve, index, select, quote, or cite the page for a specific query.

Using robots.txt to protect confidential information

Compliant crawlers may honour the file, but robots.txt is publicly readable and does not prevent access by ordinary users or non-compliant clients.

Adding provider tokens without reviewing wildcard rules

A broad User-agent: * disallow can still matter when there is no matching token-specific group or when a file is structured incorrectly.

Testing only the user-agent header

A request can claim any user-agent string. When request authenticity matters, use provider-supported verification methods rather than trusting the header alone.

How to verify the deployed policy

Open https://your-domain.example/robots.txt in a fresh browser session.
Confirm it returns a successful plain-text response from the expected hostname.
Locate the exact OAI-SearchBot and GPTBot groups.
Resolve the rule for / and for any important restricted path.
Check CDN and WAF settings for challenges that may block automated access independently of robots.txt.
Re-run the OpenForBots policy check after deployment.
Record the reason for each allow or disallow decision.

What the result does not prove

A correct robots.txt configuration does not prove:

that OpenAI fetched the page;
that the page is present in a search index;
that a model was or was not trained on earlier copies;
that ChatGPT will cite the page;
that referral traffic will increase;
that a request bearing the token is authentic.

Next step

Preview the OpenAI crawler-policy check to see how OpenForBots reports each token separately and explains what the observed rule does not prove.