Anthropic
Claude-SearchBot
Documented Provider-maintained crawler reference.
Claude-SearchBot is Anthropic's crawler for improving the relevance and accuracy of Claude search results. Anthropic documents it separately from ClaudeBot and Claude-User, says its bots honor robots.txt, and publishes a crawler source-IP list for verification support.
Reference facts
- Provider
- Anthropic
- Provider identifier
- Reviewed provider reference mark
- robots.txt token
Claude-SearchBot- Purpose
- Search and discovery
- Crawls pages
- Yes
- Obeys robots.txt
- Yes — documented by provider
- Search-visibility relevance
- Direct
- Status
- active
- Information checked
What it controls
- Whether Claude-SearchBot may crawl matching public URLs for Anthropic's search systems.
- A technical access condition that may affect the site's availability to Claude search.
What it does not control
- ClaudeBot model-training collection.
- Claude-User requests made at a user's direction.
- Citation, ranking, recommendation, or inclusion in a particular Claude answer.
Recommended policy notes
- Allow Claude-SearchBot when the site's policy goal includes visibility in Claude web-search experiences.
- Configure it separately from ClaudeBot, which Anthropic documents for potential model-training collection.
Example policies
Allow Claude search while blocking training collection
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /Anthropic documents these as separate robots.txt tokens.
Allow Claude-SearchBot
User-agent: Claude-SearchBot
Allow: /Common misconceptions
- A rule for ClaudeBot automatically applies to Claude-SearchBot.
- Allowing Claude-SearchBot guarantees that Claude will cite or recommend the site.
How OpenForBots evaluates it
- Parses the robots.txt group that applies to Claude-SearchBot and records the exact matching rule.
- Keeps the result separate from ClaudeBot and Claude-User.
- Links the current Anthropic crawler IP list as supporting verification evidence without treating a user-agent string alone as authentication.
Limitations
- OpenForBots can inspect the published directive but cannot verify Anthropic's indexing state.
- Anthropic publishes a crawler source-IP list, but ranges can change and should be checked live when request authenticity matters.
- WAF, CDN, network, or server behaviour may block access independently of robots.txt.
Related context