OpenAI
GPTBot
Documented Provider-maintained crawler reference.
GPTBot is OpenAI's robots.txt token for web content that may be used to train its generative AI foundation models. Disallowing GPTBot indicates that site content should not be used for that training crawl; it is not, by itself, a request to remove the site from ChatGPT search.
Reference facts
- Provider
- OpenAI
- Provider identifier
- Neutral OpenForBots monogram
- robots.txt token
GPTBot- Purpose
- Model training
- Crawls pages
- Yes
- Obeys robots.txt
- Yes — documented by provider
- Search-visibility relevance
- Not direct
- Status
- active
- Information checked
What it controls
- Whether GPTBot may request matching URLs under the site's robots.txt policy.
- A provider-documented signal about use of crawled site content for training OpenAI's generative AI foundation models.
What it does not control
- ChatGPT search discovery; OAI-SearchBot is the separate Search crawler token.
- User-triggered retrieval through ChatGPT-User.
- Whether a page is already present in an existing model or third-party search index.
Recommended policy notes
- Treat GPTBot as a model-training policy decision, not as the switch for ChatGPT search visibility.
- Configure OAI-SearchBot separately when search discovery matters.
Example policies
Block model-training crawling while allowing ChatGPT search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /This is a policy preference, not a security boundary and not a guarantee of search inclusion.
Allow GPTBot
User-agent: GPTBot
Allow: /Common misconceptions
- Blocking GPTBot automatically blocks ChatGPT search.
- GPTBot activity proves that a page is being cited in ChatGPT answers.
How OpenForBots evaluates it
- Parses the robots.txt group that applies to GPTBot and reports the exact matching rule.
- Treats a GPTBot block as intentional under the balanced policy goal rather than as an automatic failure.
- Keeps the GPTBot result separate from OAI-SearchBot and ChatGPT-User findings.
Limitations
- robots.txt rules apply prospectively to compliant crawler requests and do not remove information already obtained elsewhere.
- OpenForBots cannot determine whether a provider used a particular page in training.
- A user-agent string can be spoofed; robots.txt is not proof that a request came from OpenAI.
Related context