AI crawler directory / ClaudeBot
ClaudeBot
Documented Provider-maintained crawler reference.
Reference facts
- Provider
- Anthropic
- robots.txt token
ClaudeBot- Purpose
- Model training
- Crawls pages
- Yes
- Obeys robots.txt
- Yes — documented by provider
- Search-visibility relevance
- Not direct
- Status
- active
- Last verified
- Revision
- 1.0
Recommended policy notes
- ClaudeBot is documented for model training and is separate from Claude-SearchBot and Claude-User.
- Blocking ClaudeBot can be an intentional training opt-out.
Common misconceptions
- That ClaudeBot controls every Claude product surface.