Crawler policy
Checks recognised crawler tokens separately by search, training, retrieval, and data-use purpose.
Evidence-first AI discovery audit
See whether AI systems can access and understand your website
Check crawler access, public HTML, and business clarity—while keeping AI search and model-training controls separate.
- Evidence-labelled findings
- Provider-sourced crawler data
- No ranking guarantees
Crawler access summary
Example audit result
OAI-SearchBot Search discovery
Open GPTBot Model training
Blocked Claude-SearchBot Search discovery
Open Google-Extended Data-use control
Review Statuses reflect crawler purpose. Blocking training is not automatically a search-access problem.
What gets evaluated
A practical audit, not a generic AI score
OpenForBots focuses on retrievability, policy accuracy, business clarity, and the evidence behind each conclusion.
Public retrieval
Examines the resources and initial HTML a deterministic fetch can retrieve without browser automation.
Business clarity
Tests whether the company, category, audience, offer, and important supporting facts are plainly extractable.
Evidence quality
Separates direct observations and provider documentation from heuristics, inference, and manual verification.
Trust through explicit limits
Every finding explains how it was established
The same issue can carry very different confidence depending on whether OpenForBots fetched it directly, simulated a request, or inferred a likely impact.
No access result is presented as a guarantee of indexing, citation, ranking, recommendation, or traffic.
Observed
Observed
Directly retrieved or parsed during a deterministic check.
Documented
Documented
Supported by a current provider-maintained source.
Simulated
Simulated
Produced by a controlled request that does not prove real crawler behaviour.
Heuristic
Heuristic
A useful rule of thumb with visible limits and no false precision.
Inferred
Inferred
A reasoned conclusion rather than a directly observed fact.
Manual check
Manual check
Something the automated audit cannot determine responsibly.
Crawler purpose matters
Search access and model training are separate decisions
Blocking a training crawler may be an intentional policy. OpenForBots does not automatically turn that choice into a search-visibility failure.
Read the crawler distinctionSearch discovery
Controls that can affect whether a provider may retrieve pages for search or answer experiences.
Model training
Controls related to using public content to develop future models, treated as a separate policy goal.
User-triggered retrieval
Requests initiated by a user action, which may follow different product and robots rules.
Data-use controls
Tokens such as Google-Extended that govern data use rather than operating as standalone page crawlers.
Provider-sourced reference data
Latest verified crawler references
Each registry entry records the crawler token, purpose, policy notes, common misconceptions, official source, and verification date.
Published methodology
See exactly what the audit can and cannot establish
Review the evidence taxonomy, policy goals, deterministic test boundaries, scoring principles, and current preview limitations.