Crawler registry

Verified crawler identities, purposes, and policy controls

Provider-sourced records that keep search discovery, model training, user-triggered retrieval, general search, and data-use controls separate.

12 recordsLatest registry verification

Independent reference: Provider identifiers help users scan the registry and do not imply sponsorship or endorsement. A neutral OpenForBots monogram is used where third-party logo use requires permission or an official asset review.

Find a crawler reference

12 crawler records shown

ReferenceProviderPurposeSearch relevanceStatusLast verified
AppleApplebotAppleSearch and discoverydirectactive
AppleApplebot-ExtendedAppleData-use controlnot directactive
OpenAIChatGPT-UserOFB-C006OpenAIUser-triggered retrievalindirectactive
AnthropicClaude-SearchBotOFB-C007AnthropicSearch and discoverydirectactive
AnthropicClaude-UserAnthropicUser-triggered retrievalindirectactive
AnthropicClaudeBotAnthropicModel trainingnot directactive
GoogleGoogle-ExtendedOFB-C009GoogleData-use controlnot directactive
GoogleGooglebotGoogleGeneral web searchdirectactive
OpenAIGPTBotOFB-C005OpenAIModel trainingnot directactive
OpenAIOAI-SearchBotOFB-C004OpenAISearch and discoverydirectactive
PerplexityPerplexity-UserPerplexityUser-triggered retrievalindirectactive
PerplexityPerplexityBotOFB-C008PerplexitySearch and discoverydirectactive
Provider
Apple
Token
Applebot
Purpose
Search and discovery
Search relevance
direct
Last verified
Provider
OpenAI
Token
ChatGPT-User
Purpose
User-triggered retrieval
Search relevance
indirect
Last verified
Provider
Google
Token
Googlebot
Purpose
General web search
Search relevance
direct
Last verified
Provider
OpenAI
Token
GPTBot
Purpose
Model training
Search relevance
not direct
Last verified
Provider
OpenAI
Token
OAI-SearchBot
Purpose
Search and discovery
Search relevance
direct
Last verified