Free robots.txt builder
Build a starting robots.txt from a declared policy goal
Choose a declared policy goal and this page builds robots.txt groups for the documented AI crawler tokens that goal has an explicit preference for, which you can review, edit, and publish.
This is a content-authoring tool, not a live check of your site. It does not read your existing robots.txt and does not prove that any provider will honor the generated rules once published.
Live preview
robots.txt preview
This updates as you edit the fields above. Nothing is sent to a server.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /This is a generated starting point, not a verified fix and not a file already published on your site. Review every line, then publish it yourself.
More free tools
Try the other free AI discovery tools
Each tool looks at a different layer of AI discovery readiness. Use them together, or open the complete audit for all of it at once.
Enter a public hostname to test whether its deployed robots.txt policy allows, blocks, or has no matching restriction for documented AI crawler tokens at the root path.
Enter a public hostname to inspect its root-level llms.txt response, title, introduction, sections, links, malformed Markdown, duplicate destinations, and insecure URLs.
Enter a public hostname to compare the visible business name and opening copy with the Organization, WebSite, Person, or SoftwareApplication entities declared in the homepage JSON-LD.
Enter a public hostname to inspect Recipe entities declared in its homepage JSON-LD and see which of Google’s documented required and recommended Recipe properties were observed.
Fill in a title, short context, and the resource links you want to list, and this page builds a valid llms.txt file for you to review, edit, and publish.
Fill in your business name, canonical URL, logo, and official profile links, and this page builds a valid Organization JSON-LD block for you to review, edit, and publish.
How this builder works
From a few fields to a file you can publish
Everything runs locally in your browser — there is no server round-trip and nothing is stored.
- 1
Choose a declared policy goal — the same six goals used in the complete audit’s policy-goal comparison.
- 2
Review the crawler groups the tool generates for that goal from the documented crawler registry.
- 3
Add or remove custom allow/disallow rows for specific crawler tokens if you need to.
- 4
Review the live preview, then copy or download the generated file.
Example generated output
A training opt-out group for GPTBot
Your selected policy goal and any custom rows replace this example as you choose them.
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: /
Understand the output
What this builder creates—and what it cannot tell you
What it builds
- One
User-agent/AlloworDisallowgroup per documented crawler token the chosen goal has an explicit preference for. - An optional default
User-agent: *group. - Any custom rows you add for a specific crawler token.
What remains your responsibility
- The generated file is a starting point; review every group before publishing it.
- This tool does not read or preserve an existing robots.txt — use the robots.txt checker to see what is currently deployed.
- A declared policy goal reflects your stated preference; it is not proof that a provider will honor the resulting rules.
- Only documented crawler tokens in the current registry are included; undocumented or newly announced tokens are not covered.
Learn and take action
Related guides and tools
Check a deployed robots.txt policy against documented AI crawler tokens.
Read more AI crawler registryReview the documented crawler references used to generate groups.
Read more How the audit worksReview the declared policy goals and how they compare against observed rules.
Read moreCommon questions
Questions about this builder
Does this read my existing robots.txt first?
No. This tool generates a file from scratch based on your selected goal and does not fetch or merge your currently deployed robots.txt. Use the robots.txt checker to see what is deployed today.
Will providers definitely honor the generated rules?
No. robots.txt expresses a preference that compliant, well-behaved crawlers are expected to follow. It is not an access-control or security mechanism, and this tool cannot guarantee provider behavior.
What if a crawler I care about is missing?
Only documented tokens in OpenForBots’ current crawler registry are included. Add a custom row for any additional token you need, or check the AI crawler registry for what is currently documented.