Crawler registry
Verified crawler identities, purposes, and policy controls
Provider-sourced records that keep search discovery, model training, user-triggered retrieval, general search, and data-use controls separate.
| Reference | Provider | Purpose | Search relevance | Status | Last verified |
|---|---|---|---|---|---|
| Applebot | Apple | Search and discovery | direct | active | |
| Applebot-Extended | Apple | Data-use control | not direct | active | |
| ChatGPT-User OFB-C006 | OpenAI | User-triggered retrieval | indirect | active | |
| Claude-SearchBot OFB-C007 | Anthropic | Search and discovery | direct | active | |
| Claude-User | Anthropic | User-triggered retrieval | indirect | active | |
| ClaudeBot | Anthropic | Model training | not direct | active | |
| Google-Extended OFB-C009 | Data-use control | not direct | active | ||
| Googlebot | General web search | direct | active | ||
| GPTBot OFB-C005 | OpenAI | Model training | not direct | active | |
| OAI-SearchBot OFB-C004 | OpenAI | Search and discovery | direct | active | |
| Perplexity-User | Perplexity | User-triggered retrieval | indirect | active | |
| PerplexityBot OFB-C008 | Perplexity | Search and discovery | direct | active |