Guides / How to Allow AI Search While Blocking Model Training

How to Allow AI Search While Blocking Model Training

A provider-specific robots.txt pattern for allowing AI search discovery while restricting named model-training crawlers and data-use controls.

You can allow named AI search crawlers while blocking named model-training crawlers because major providers document separate robots.txt tokens for different purposes. A balanced policy commonly allows OAI-SearchBot, Claude-SearchBot, and PerplexityBot, while disallowing GPTBot and ClaudeBot. Google-Extended is a separate data-use control and does not determine Google Search inclusion.

Start with the policy goal

Do not copy a robots.txt block before deciding what you want.

A balanced policy means:

public pages may be discovered for AI-assisted search;
provider-documented training crawlers are restricted;
user-triggered retrieval is reviewed separately;
ordinary search crawlers remain governed by the site’s existing SEO policy;
sensitive information stays behind real authentication;
no access setting is treated as a promise of indexing or citation.

This is a business and governance choice, not a universal best practice.

A cautious multi-provider example

# OpenAI search discovery
User-agent: OAI-SearchBot
Allow: /

# OpenAI potential model-training collection
User-agent: GPTBot
Disallow: /

# Anthropic search discovery
User-agent: Claude-SearchBot
Allow: /

# Anthropic potential model-training collection
User-agent: ClaudeBot
Disallow: /

# Perplexity search discovery
User-agent: PerplexityBot
Allow: /

# Google Search remains controlled through Googlebot.
User-agent: Googlebot
Allow: /

# Restrict the documented Google-Extended data uses.
User-agent: Google-Extended
Disallow: /

Review the provider documentation before deploying this example. Tokens, product descriptions, and verification methods can change.

Decide user-triggered retrieval separately

OpenAI, Anthropic, and Perplexity document user-triggered agents separately from background discovery crawlers.

A site that wants to allow public-page retrieval in supported user workflows might add:

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

Perplexity documents Perplexity-User as a user-requested fetcher and states that it generally ignores robots.txt because the user requested the retrieval. Treat that behaviour as provider-specific rather than assuming one common rule across every product.

User-triggered retrieval still does not provide access to content protected by authentication.

Provider distinctions

OpenAI

OAI-SearchBot: search discovery for ChatGPT search.
GPTBot: potential model-development or training use.
ChatGPT-User: user-triggered retrieval.

See GPTBot vs OAI-SearchBot.

Anthropic

Claude-SearchBot: search-result relevance and accuracy.
ClaudeBot: content that may contribute to model training.
Claude-User: user-initiated retrieval.

Anthropic says its bots honour robots.txt and currently does not publish stable IP ranges for them.

Perplexity

PerplexityBot: search discovery and linked results; Perplexity states that it is not used to crawl content for foundation-model training.
Perplexity-User: user-requested retrieval; the provider documents different robots behaviour for this path.

Perplexity publishes current IP-range endpoints. Use the live provider data if configuring a WAF.

Google

Googlebot: crawling for Google Search, including Google’s Search features.
Google-Extended: a product token controlling documented Gemini training and grounding uses. It has no separate HTTP user-agent string.

Google says Google-Extended does not affect Search inclusion or ranking. Do not use it as a proxy for AI Overviews eligibility.

Preserve existing search-engine access intentionally

A broad wildcard block can undermine the policy you intended:

User-agent: *
Disallow: /

If the site needs normal search visibility, review how wildcard and token-specific groups apply. Do not assume that adding a few AI-specific Allow lines fixes every crawler or path interaction.

The Robots Exclusion Protocol uses product-token group matching and the most specific matching path rule. Keep the file simple enough to review manually.

Apply the policy to every relevant hostname

A robots.txt file applies to the specific scheme, hostname, and port where it is served. Review:

the apex domain;
the www hostname;
documentation, support, shop, or app subdomains;
regional or language hostnames;
staging environments that may have become public.

Do not assume the root-domain file governs every subdomain.

Keep private content private

Never rely on robots.txt for:

customer records;
unpublished documents;
account pages;
licensed content that requires access control;
administrative URLs;
secrets or API credentials.

RFC 9309 states that robots.txt rules are not access authorisation. Use authentication, authorisation, network controls, and appropriate response headers.

Common implementation errors

Copying a stale token list

Provider products change. Store a last-verified date and link to the primary documentation rather than presenting a copied list as permanently complete.

Confusing a data-use token with a crawler

Google-Extended has no separate HTTP user-agent string. It should not be monitored as though it independently crawls pages.

Treating a training block as a visibility error

Blocking GPTBot or ClaudeBot can be the intended balanced policy. A readiness score should not punish that choice automatically.

Ignoring WAF or CDN behaviour

A crawler may be allowed in robots.txt but receive a JavaScript challenge, rate-limit response, or access-denied page.

Publishing contradictory rules

Multiple matching groups may be combined by compliant parsers. Duplicate groups and inherited wildcard rules can produce a result different from a quick visual reading.

Verification checklist

Open each deployed /robots.txt URL and confirm a successful plain-text response.
Check the exact token spelling against current provider documentation.
Resolve the rule for / and important restricted paths.
Confirm that ordinary search crawlers still have the intended access.
Test for CDN or WAF challenges using safe, authorised requests.
Check server logs only as supporting evidence; user-agent strings can be spoofed.
Record the policy owner, rationale, and review date.
Re-run the policy audit after every material change.

What this configuration does not guarantee

It does not guarantee:

indexing by an AI search provider;
citation in an answer;
ranking or recommendation;
increased traffic;
exclusion of content already obtained through other sources;
compliance by malicious or non-conforming bots;
successful retrieval through every product workflow.

Next step

Preview the balanced crawler-policy audit to see how OpenForBots evaluates the same directive differently under balanced, maximum-access, and review-only goals.