AI crawler directory / Claude-SearchBot
Claude-SearchBot
Documented Provider-maintained crawler reference.
Content ID: OFB-C007
Claude-SearchBot is Anthropic's crawler for improving the relevance and accuracy of Claude search results. Anthropic documents it separately from ClaudeBot and Claude-User, so search, training, and user-triggered retrieval can be controlled independently.
Reference facts
- Provider
- Anthropic
- robots.txt token
Claude-SearchBot- Purpose
- Search and discovery
- Crawls pages
- Yes
- Obeys robots.txt
- Yes — documented by provider
- Search-visibility relevance
- Direct
- Status
- active
- Last verified
- Revision
- 1.0
What it controls
- Whether Claude-SearchBot may crawl matching public URLs for Anthropic's search systems.
- A technical access condition that may affect the site's availability to Claude search.
What it does not control
- ClaudeBot model-training collection.
- Claude-User requests made at a user's direction.
- Citation, ranking, recommendation, or inclusion in a particular Claude answer.
Recommended policy notes
- Allow Claude-SearchBot when the site's policy goal includes visibility in Claude web-search experiences.
- Configure it separately from ClaudeBot, which Anthropic documents for potential model-training collection.
Example policies
Allow Claude search while blocking training collection
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /Anthropic documents these as separate robots.txt tokens.
Allow Claude-SearchBot
User-agent: Claude-SearchBot
Allow: /Common misconceptions
- A rule for ClaudeBot automatically applies to Claude-SearchBot.
- Allowing Claude-SearchBot guarantees that Claude will cite or recommend the site.
How OpenForBots evaluates it
- Parses the robots.txt group that applies to Claude-SearchBot and records the exact matching rule.
- Keeps the result separate from ClaudeBot and Claude-User.
- Evaluates the rule against the selected policy goal without treating a training opt-out as a search failure.
Limitations
- OpenForBots can inspect the published directive but cannot verify Anthropic's indexing state.
- Anthropic does not currently publish stable bot IP ranges, so a user-agent string alone is not authentication.
- WAF, CDN, network, or server behaviour may block access independently of robots.txt.
Related guides and concepts
- AI Search Crawlers vs Training Crawlers
- Allow AI Search While Blocking Model Training
- How the OpenForBots Audit Works