Crawler registry

Verified crawler identities, purposes, and policy controls

Provider-sourced records that keep search discovery, model training, user-triggered retrieval, general search, and data-use controls separate.

12 recordsLatest registry verification

Independent reference: Provider identifiers help users scan the registry and do not imply sponsorship or endorsement. A neutral OpenForBots monogram is used where third-party logo use requires permission or an official asset review.

Crawler registry vs crawler registration

This page is a registry: a maintained, provider-sourced reference of documented AI crawler identities, purposes, and policy controls. It is not a registration service. OpenForBots documents known crawler identities and purposes; it does not register, authenticate, or independently certify third-party crawlers.

When a provider offers its own registration or verification process — for example, publishing IP ranges, supporting reverse-DNS confirmation, or a cryptographic signing mechanism — that process happens directly with the provider or a platform they've registered with, not with OpenForBots. See how crawler registration and verification actually work for the current documented mechanisms and their limits.

Find a crawler reference

12 crawler records shown

ReferenceProviderPurposeSearch relevanceStatusLast verified
AppleApplebotAppleSearch and discoverydirectactive
AppleApplebot-ExtendedAppleData-use controlnot directactive
OpenAIChatGPT-UserOFB-C006OpenAIUser-triggered retrievalindirectactive
AnthropicClaude-SearchBotOFB-C007AnthropicSearch and discoverydirectactive
AnthropicClaude-UserAnthropicUser-triggered retrievalindirectactive
AnthropicClaudeBotAnthropicModel trainingnot directactive
GoogleGoogle-ExtendedOFB-C009GoogleData-use controlnot directactive
GoogleGooglebotGoogleGeneral web searchdirectactive
OpenAIGPTBotOFB-C005OpenAIModel trainingnot directactive
OpenAIOAI-SearchBotOFB-C004OpenAISearch and discoverydirectactive
PerplexityPerplexity-UserPerplexityUser-triggered retrievalindirectactive
PerplexityPerplexityBotOFB-C008PerplexitySearch and discoverydirectactive
Provider
Apple
Token
Applebot
Purpose
Search and discovery
Search relevance
direct
Last verified
Provider
OpenAI
Token
ChatGPT-User
Purpose
User-triggered retrieval
Search relevance
indirect
Last verified
Provider
Google
Token
Googlebot
Purpose
General web search
Search relevance
direct
Last verified
Provider
OpenAI
Token
GPTBot
Purpose
Model training
Search relevance
not direct
Last verified
Provider
OpenAI
Token
OAI-SearchBot
Purpose
Search and discovery
Search relevance
direct
Last verified