AI crawler directory / GPTBot
GPTBot
Documented Provider-maintained crawler reference.
Content ID: OFB-C005
GPTBot is OpenAI's robots.txt token for web content that may be used in model development or training. Blocking GPTBot is a training opt-out; it is not, by itself, a request to remove a site from ChatGPT search.
Reference facts
- Provider
- OpenAI
- robots.txt token
GPTBot- Purpose
- Model training
- Crawls pages
- Yes
- Obeys robots.txt
- Yes — documented by provider
- Search-visibility relevance
- Not direct
- Status
- active
- Last verified
- Revision
- 1.0
What it controls
- Whether GPTBot may request matching URLs under the site's robots.txt policy.
- A provider-documented signal about potential use of future site content in model-development datasets.
What it does not control
- ChatGPT search discovery; OAI-SearchBot is the separate search crawler token.
- User-triggered retrieval through ChatGPT-User.
- Whether a page is already present in an existing model or third-party search index.
Recommended policy notes
- Treat GPTBot as a model-training policy decision, not as the switch for ChatGPT search visibility.
- Configure OAI-SearchBot separately when search discovery matters.
Example policies
Block model-training crawling while allowing ChatGPT search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /This is a policy preference, not a security boundary and not a guarantee of search inclusion.
Allow GPTBot
User-agent: GPTBot
Allow: /Common misconceptions
- Blocking GPTBot automatically blocks ChatGPT search.
- GPTBot activity proves that a page is being cited in ChatGPT answers.
How OpenForBots evaluates it
- Parses the robots.txt group that applies to GPTBot and reports the exact matching rule.
- Treats a GPTBot block as intentional under the balanced policy goal rather than as an automatic failure.
- Keeps the GPTBot result separate from OAI-SearchBot and ChatGPT-User findings.
Limitations
- robots.txt rules apply prospectively to compliant crawler requests and do not remove information already obtained elsewhere.
- OpenForBots cannot determine whether a provider used a particular page in training.
- A user-agent string can be spoofed; robots.txt is not proof that a request came from OpenAI.
Related guides and concepts
- AI Search Crawlers vs Training Crawlers
- GPTBot vs OAI-SearchBot
- Allow AI Search While Blocking Model Training