OpenAI
OAI-SearchBot
Documented Provider-maintained crawler reference.
OAI-SearchBot is OpenAI's robots.txt token for the crawler used to surface websites in ChatGPT search results. OpenAI recommends allowing it for Search visibility and says sites opted out will not be shown in ChatGPT search answers, though they may still appear as navigational links. Allowing access still does not guarantee selection or citation for a particular query.
Reference facts
- Provider
- OpenAI
- Provider identifier
- Neutral OpenForBots monogram
- robots.txt token
OAI-SearchBot- Purpose
- Search and discovery
- Crawls pages
- Yes
- Obeys robots.txt
- Yes — documented by provider
- Search-visibility relevance
- Direct
- Status
- active
- Information checked
What it controls
- Whether OAI-SearchBot may request matching URLs under the site's robots.txt rules.
- A provider-documented access condition for appearing in ChatGPT search results.
What it does not control
- Potential use of content for model training; GPTBot is the separate token for that policy.
- Whether a user-triggered fetch succeeds in every ChatGPT workflow.
- Selection, ranking, citation, recommendation, or referral traffic for a particular query.
Recommended policy notes
- Allow OAI-SearchBot when the site's policy goal includes appearing in ChatGPT search results.
- Configure it separately from GPTBot because Search access and potential model-training use are independent settings.
Example policies
Allow ChatGPT search while controlling training separately
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /This expresses separate preferences. It does not guarantee inclusion in a particular ChatGPT answer.
Allow OAI-SearchBot
User-agent: OAI-SearchBot
Allow: /Common misconceptions
- Blocking GPTBot also blocks OAI-SearchBot.
- Allowing OAI-SearchBot guarantees that a page will be indexed, cited, ranked, or recommended.
How OpenForBots evaluates it
- Fetches the root robots.txt file and resolves the group that applies to OAI-SearchBot.
- Reports the exact matching allow or disallow rule and labels the result as observed.
- Evaluates the observed rule against the policy goal selected by the site owner.
Limitations
- robots.txt expresses crawler preferences; it is not authentication or an access-control system.
- OpenForBots can test the published rule but cannot prove that OpenAI has indexed or will cite a page for a particular query.
- CDN, WAF, authentication, DNS, or server errors can still prevent retrieval even when robots.txt allows it.
Related context