Free robots.txt builder

Build a starting robots.txt from a declared policy goal

Choose a declared policy goal and this page builds robots.txt groups for the documented AI crawler tokens that goal has an explicit preference for, which you can review, edit, and publish.

This is a content-authoring tool, not a live check of your site. It does not read your existing robots.txt and does not prove that any provider will honor the generated rules once published.

Runs only in your browser Nothing sent to a server Review before publishing

Methodology reviewed 2026-08-08

Live preview

robots.txt preview

This updates as you edit the fields above. Nothing is sent to a server.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

This is a generated starting point, not a verified fix and not a file already published on your site. Review every line, then publish it yourself.

More free tools

Try the other free AI discovery tools

Each tool looks at a different layer of AI discovery readiness. Use them together, or open the complete audit for all of it at once.

Check robots.txt rules for documented AI crawlers

Enter a public hostname to test whether its deployed robots.txt policy allows, blocks, or has no matching restriction for documented AI crawler tokens at the root path.

Check AI crawler rules

llms.txt checker and validator

Check the live /llms.txt file on a public website, or paste a draft to validate it before deployment. Both modes use the same rules for title, introduction, sections, links, malformed Markdown, duplicate destinations, and insecure URLs.

Inspect llms.txt

Check how clearly your website identifies the business and its entities

Compare visible homepage identity with structured entities, graph identifiers, sameAs identity references, and trust signals.

Check structure and identity

Check Recipe structured data on a food publisher page

Check the actual public recipe page for Recipe JSON-LD, Google-required fields, and selected recommended properties.

Check Recipe schema

Build a starting llms.txt file

Fill in a title, short context, and the resource links you want to list, and this page builds a valid llms.txt file for you to review, edit, and publish.

Build llms.txt

Build a starting Organization JSON-LD block

Build Organization JSON-LD with a reusable #organization @id and optional identity-reference URLs.

Build JSON-LD

AI citation forensics

Paste a prompt and the source an AI answer cited. See which public evidence differs from your site, what is worth testing, and what no external audit can prove.

Compare the evidence

Business facts consistency checker

Compare the business name, phone, email, address, and official profiles stated on your homepage, About page, Contact page, and JSON-LD, and see where any value differs.

Check business facts

Website launch and migration check

Compare the new version of a site or page with the current live one under the same audit rules, and see regressions in crawler access, content, and identity before you launch.

Compare before and after

How this builder works

From a few fields to a file you can publish

Everything runs locally in your browser — there is no server round-trip and nothing is stored.

1
Choose a declared policy goal — the same six goals used in the complete audit’s policy-goal comparison.
2
Review the crawler groups the tool generates for that goal from the documented crawler registry.
3
Add or remove custom allow/disallow rows for specific crawler tokens if you need to.
4
Review the live preview, then copy or download the generated file.

Example generated output

A training opt-out group for GPTBot

Your selected policy goal and any custom rows replace this example as you choose them.

Generated example

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Understand the output

What this builder creates—and what it cannot tell you

What it builds

One User-agent / Allow or Disallow group per documented crawler token the chosen goal has an explicit preference for.
An optional default User-agent: * group.
Any custom rows you add for a specific crawler token.

What remains your responsibility

The generated file is a starting point; review every group before publishing it.
This tool does not read or preserve an existing robots.txt — use the robots.txt checker to see what is currently deployed.
A declared policy goal reflects your stated preference; it is not proof that a provider will honor the resulting rules.
Only documented crawler tokens in the current registry are included; undocumented or newly announced tokens are not covered.

Learn and take action

Related guides and tools

Already have one? Validate it

Check a deployed robots.txt policy against documented AI crawler tokens.

Questions about this builder

Does this read my existing robots.txt first?

No. This tool generates a file from scratch based on your selected goal and does not fetch or merge your currently deployed robots.txt. Use the robots.txt checker to see what is deployed today.

Will providers definitely honor the generated rules?

No. robots.txt expresses a preference that compliant, well-behaved crawlers are expected to follow. It is not an access-control or security mechanism, and this tool cannot guarantee provider behavior.

What if a crawler I care about is missing?

Only documented tokens in OpenForBots’ current crawler registry are included. Add a custom row for any additional token you need, or check the AI crawler registry for what is currently documented.