GPTBot vs OAI-SearchBot

The current OpenAI-documented difference between its model-training crawler, ChatGPT Search crawler, and user-triggered ChatGPT-User agent.

GPTBot and OAI-SearchBot are separate OpenAI crawler tokens with different documented purposes. OpenAI uses OAI-SearchBot for ChatGPT Search and GPTBot to crawl content that may be used to train its generative AI foundation models. Each setting is independent.

OpenAI also documents ChatGPT-User, but it is different again: it is used for certain user actions rather than automatic crawling, and OpenAI says robots.txt rules may not apply to those user-initiated actions.

Quick comparison

Token	Documented purpose	robots.txt role	Search relevance
`OAI-SearchBot`	Surface websites in ChatGPT Search results	Provider-documented Search control	Direct
`GPTBot`	Crawl content that may be used for foundation-model training	Provider-documented training control	Not direct
`ChatGPT-User`	Certain user-triggered retrieval in ChatGPT and Custom GPTs	OpenAI says robots.txt rules may not apply	Indirect

Do not collapse these into one “OpenAI bot” setting.

OAI-SearchBot controls ChatGPT Search crawling

OpenAI currently says sites that opt out of OAI-SearchBot will not be shown in ChatGPT Search answers, although they may still appear as navigational links. OpenAI recommends allowing the crawler if you want the site to appear in Search results.

A simple Search-allow rule is:

User-agent: OAI-SearchBot
Allow: /

That removes a documented robots.txt restriction. It does not guarantee that a particular page will be selected, ranked, quoted, or cited for a query.

OpenAI notes that robots.txt changes can take roughly 24 hours to be reflected in its Search systems, so verify the deployed file and allow for that delay before treating a provider-side result as contradictory.

GPTBot is a separate training decision

OpenAI documents GPTBot as a crawler for content that may be used in training its generative AI foundation models. Disallowing it indicates that the site’s content should not be used through that training crawl.

A site can therefore express a balanced policy such as:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

This says “allow the Search crawler, restrict the training crawler.” It is a governance choice, not an SEO score target.

Blocking GPTBot does not automatically block ChatGPT Search because Search uses a separate token.

What about ChatGPT-User?

ChatGPT-User is used for certain actions initiated by a ChatGPT or Custom GPT user. OpenAI explicitly distinguishes it from automatic web crawling and says that because these actions are user-initiated, robots.txt rules may not apply.

That changes how a site owner should interpret a ChatGPT-User group:

OpenForBots can show that the group exists and parse the matching directive as declared site policy;
it should not claim that the directive is guaranteed to block or allow a real user-triggered ChatGPT fetch;
ChatGPT-User is not the Search opt-out control;
use OAI-SearchBot to manage ChatGPT Search automatic crawling;
protect non-public content with authentication and authorization rather than robots.txt.

This provider-specific nuance is why user-triggered fetchers should not be treated as one universal category.

Restrict only part of a site

A site may want marketing and documentation pages discoverable while excluding an area from Search crawling:

User-agent: OAI-SearchBot
Allow: /
Disallow: /member-library/

User-agent: GPTBot
Disallow: /

Remember that robots.txt is public and voluntary. A member or licensed-content area still requires real access control.

How robots.txt matching works

Under the Robots Exclusion Protocol, a compliant automatic crawler identifies the group matching its product token and applies the most specific matching path rule.

Keep the file easy to review:

use the documented token exactly;
avoid duplicate groups unless their combined behavior is intentional;
keep important path rules explicit;
test the deployed /robots.txt, not only a source file in the repository;
repeat the intended policy on every relevant origin and subdomain.

Common mistakes

Blocking GPTBot and expecting ChatGPT Search to disappear

GPTBot is not the ChatGPT Search crawler. OAI-SearchBot is the relevant robots token for Search.

Allowing OAI-SearchBot and claiming guaranteed citations

The robots rule affects a documented access condition. It does not determine which source OpenAI selects or cites for a particular answer.

Treating ChatGPT-User like OAI-SearchBot

OpenAI says robots.txt rules may not apply to ChatGPT-User because its requests are user-initiated. A parsed rule is therefore not the same thing as a provider-enforced verdict.

Using robots.txt to protect confidential information

Robots rules are not security controls. Private content requires authentication, authorization, and appropriate server-side controls.

Trusting only the user-agent string

A request can spoof a user-agent. When request authenticity matters, use provider-published IP information or other provider-supported verification methods in addition to the string.

How to verify the deployed policy

Open the deployed /robots.txt on the correct hostname.
Confirm the exact OAI-SearchBot and GPTBot groups against current OpenAI documentation.
Resolve the rule for / and any important restricted path.
Check CDN and WAF behavior separately from robots.txt.
Treat any ChatGPT-User rule as declared policy evidence with OpenAI’s “may not apply” caveat.
Re-run OpenForBots after deployment.
Record the business reason for each allow or disallow decision.

What the result does not prove

A correct robots.txt configuration does not prove:

that OpenAI fetched the page;
that the page is present in an internal index;
that a model was or was not trained on earlier copies;
that ChatGPT will select or cite the page for a query;
that a ChatGPT-User request will follow the same robots semantics as an automatic crawler;
that referral traffic will increase.

Next step

Run the OpenAI crawler-policy check to inspect OAI-SearchBot and GPTBot separately, and keep any ChatGPT-User result within the provider caveat above.

Related context