AI crawler directory / Claude-User
Claude-User
Documented Provider-maintained crawler reference.
Reference facts
- Provider
- Anthropic
- robots.txt token
Claude-User- Purpose
- User-triggered retrieval
- Crawls pages
- Yes
- Obeys robots.txt
- Yes — documented by provider
- Search-visibility relevance
- Indirect
- Status
- active
- Last verified
- Revision
- 1.0
Recommended policy notes
- Claude-User represents retrieval initiated by a Claude user.
- Treat it separately from ClaudeBot and Claude-SearchBot.
Common misconceptions
- That Claude-User is a search-index crawler.