OpenForBotsAI discovery readiness
Run website auditRun audit

Free robots.txt builder

Build a starting robots.txt from a declared policy goal

Choose a declared policy goal and this page builds robots.txt groups for the documented AI crawler tokens that goal has an explicit preference for, which you can review, edit, and publish.

This is a content-authoring tool, not a live check of your site. It does not read your existing robots.txt and does not prove that any provider will honor the generated rules once published.

Runs only in your browser Nothing sent to a server Review before publishing

Methodology reviewed

Live preview

robots.txt preview

This updates as you edit the fields above. Nothing is sent to a server.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

This is a generated starting point, not a verified fix and not a file already published on your site. Review every line, then publish it yourself.

More free tools

Try the other free AI discovery tools

Each tool looks at a different layer of AI discovery readiness. Use them together, or open the complete audit for all of it at once.

Check robots.txt rules for documented AI crawlers

Enter a public hostname to test whether its deployed robots.txt policy allows, blocks, or has no matching restriction for documented AI crawler tokens at the root path.

Check AI crawler rules
Inspect a deployed llms.txt file and its linked resources

Enter a public hostname to inspect its root-level llms.txt response, title, introduction, sections, links, malformed Markdown, duplicate destinations, and insecure URLs.

Inspect llms.txt
Check how clearly your website identifies the business and its entities

Enter a public hostname to compare the visible business name and opening copy with the Organization, WebSite, Person, or SoftwareApplication entities declared in the homepage JSON-LD.

Check structure and identity
Check Recipe structured data on a food publisher page

Enter a public hostname to inspect Recipe entities declared in its homepage JSON-LD and see which of Google’s documented required and recommended Recipe properties were observed.

Check Recipe schema
Build a starting llms.txt file

Fill in a title, short context, and the resource links you want to list, and this page builds a valid llms.txt file for you to review, edit, and publish.

Build llms.txt
Build a starting Organization JSON-LD block

Fill in your business name, canonical URL, logo, and official profile links, and this page builds a valid Organization JSON-LD block for you to review, edit, and publish.

Build JSON-LD
AI citation forensics

Paste a prompt and the source an AI answer cited. See which public evidence differs from your site, what is worth testing, and what no external audit can prove.

Compare the evidence

How this builder works

From a few fields to a file you can publish

Everything runs locally in your browser — there is no server round-trip and nothing is stored.

  1. 1

    Choose a declared policy goal — the same six goals used in the complete audit’s policy-goal comparison.

  2. 2

    Review the crawler groups the tool generates for that goal from the documented crawler registry.

  3. 3

    Add or remove custom allow/disallow rows for specific crawler tokens if you need to.

  4. 4

    Review the live preview, then copy or download the generated file.

Example generated output

A training opt-out group for GPTBot

Your selected policy goal and any custom rows replace this example as you choose them.

Generated example
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Understand the output

What this builder creates—and what it cannot tell you

What it builds

  • One User-agent / Allow or Disallow group per documented crawler token the chosen goal has an explicit preference for.
  • An optional default User-agent: * group.
  • Any custom rows you add for a specific crawler token.

What remains your responsibility

  • The generated file is a starting point; review every group before publishing it.
  • This tool does not read or preserve an existing robots.txt — use the robots.txt checker to see what is currently deployed.
  • A declared policy goal reflects your stated preference; it is not proof that a provider will honor the resulting rules.
  • Only documented crawler tokens in the current registry are included; undocumented or newly announced tokens are not covered.

Learn and take action

Related guides and tools

Already have one? Validate it

Check a deployed robots.txt policy against documented AI crawler tokens.

Read more AI crawler registry

Review the documented crawler references used to generate groups.

Read more How the audit works

Review the declared policy goals and how they compare against observed rules.

Read more

Common questions

Questions about this builder

Does this read my existing robots.txt first?

No. This tool generates a file from scratch based on your selected goal and does not fetch or merge your currently deployed robots.txt. Use the robots.txt checker to see what is deployed today.

Will providers definitely honor the generated rules?

No. robots.txt expresses a preference that compliant, well-behaved crawlers are expected to follow. It is not an access-control or security mechanism, and this tool cannot guarantee provider behavior.

What if a crawler I care about is missing?

Only documented tokens in OpenForBots’ current crawler registry are included. Add a custom row for any additional token you need, or check the AI crawler registry for what is currently documented.