AI crawler directory / OAI-SearchBot
OAI-SearchBot
Documented Provider-maintained crawler reference.
Content ID: OFB-C004
OAI-SearchBot is OpenAI's robots.txt token for the crawler used to discover content for ChatGPT search. OpenAI says sites should allow it for content to be eligible for summaries and snippets, but access does not guarantee that a page will appear or be cited.
Reference facts
- Provider
- OpenAI
- robots.txt token
OAI-SearchBot- Purpose
- Search and discovery
- Crawls pages
- Yes
- Obeys robots.txt
- Yes — documented by provider
- Search-visibility relevance
- Direct
- Status
- active
- Last verified
- Revision
- 1.0
What it controls
- Whether OAI-SearchBot may request matching URLs under the site's robots.txt rules.
- A technical access condition relevant to ChatGPT search discovery and snippets.
What it does not control
- Potential use of content for model training; GPTBot is the separate token for that policy.
- Whether a user-triggered fetch succeeds in every ChatGPT workflow.
- Indexing, ranking, citation, recommendation, or referral traffic.
Recommended policy notes
- Allow OAI-SearchBot when the site's policy goal includes discoverability in ChatGPT search.
- Configure it separately from GPTBot because search access and potential model-training use are different decisions.
Example policies
Allow ChatGPT search while controlling training separately
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /This expresses separate preferences. It does not guarantee inclusion in a ChatGPT answer.
Allow OAI-SearchBot
User-agent: OAI-SearchBot
Allow: /Common misconceptions
- Blocking GPTBot also blocks OAI-SearchBot.
- Allowing OAI-SearchBot guarantees that a page will be indexed, cited, ranked, or recommended.
How OpenForBots evaluates it
- Fetches the root robots.txt file and resolves the group that applies to OAI-SearchBot.
- Reports the exact matching allow or disallow rule and labels the result as observed.
- Evaluates the observed rule against the policy goal selected by the site owner.
Limitations
- robots.txt expresses crawler preferences; it is not authentication or an access-control system.
- OpenForBots can test the published rule but cannot prove that OpenAI has indexed or will cite a page.
- CDN, WAF, authentication, DNS, or server errors can still prevent retrieval even when robots.txt allows it.
Related guides and concepts
- AI Search Crawlers vs Training Crawlers
- GPTBot vs OAI-SearchBot
- Allow AI Search While Blocking Model Training