AI crawler directory / Googlebot
Googlebot
Documented Provider-maintained crawler reference.
Reference facts
- Provider
- robots.txt token
Googlebot- Purpose
- General web search
- Crawls pages
- Yes
- Obeys robots.txt
- Yes — documented by provider
- Search-visibility relevance
- Direct
- Status
- active
- Last verified
- Revision
- 1.0
Recommended policy notes
- Googlebot is Google's primary crawler for Search and is distinct from Google-Extended.
- A Googlebot block affects ordinary Google Search crawling.
Common misconceptions
- That Google-Extended replaces Googlebot's crawl.