Free AI crawler policy checker

Check robots.txt rules for documented AI crawlers

Enter a public hostname to inspect how its deployed robots.txt policy matches documented crawler tokens at the root path. For user-triggered agents whose provider says robots.txt may not apply or is generally ignored, OpenForBots shows the matching directive as site-declared policy evidence rather than a guaranteed provider-enforcement verdict.

The result separates search discovery, model training, user-triggered retrieval, and other documented purposes. Direct Search relevance is kept separate from indirect or non-Search tokens, and provider-specific robots.txt applicability is preserved rather than inferred.

Public website check Inspectable evidence Clear limitations

Methodology reviewed 2026-10-04

More free checks

Try the other free AI discovery checks

Each check looks at a different layer of AI discovery readiness. Run them together for a fuller picture, or open the complete audit for all of it at once.

llms.txt checker and validator

Check the live /llms.txt file on a public website, or paste a draft to validate it before deployment. Both modes use the same rules for title, introduction, sections, links, malformed Markdown, duplicate destinations, and insecure URLs.

Inspect llms.txt

Check how clearly your website identifies the business and its entities

Compare visible homepage identity with structured entities, graph identifiers, sameAs identity references, and trust signals.

Check structure and identity

Check Recipe structured data on a food publisher page

Check the actual public recipe page for Recipe JSON-LD, Google-required fields, and selected recommended properties.

Check Recipe schema

Build a starting llms.txt file

Fill in a title, short context, and the resource links you want to list, and this page builds a valid llms.txt file for you to review, edit, and publish.

Build llms.txt

Build a starting robots.txt from a declared policy goal

Choose a declared policy goal and this page builds robots.txt groups for the documented AI crawler tokens that goal has an explicit preference for, which you can review, edit, and publish.

Build robots.txt

Build a starting Organization JSON-LD block

Build Organization JSON-LD with a reusable #organization @id and optional identity-reference URLs.

Build JSON-LD

AI citation forensics

Paste a prompt and the source an AI answer cited. See which public evidence differs from your site, what is worth testing, and what no external audit can prove.

Compare the evidence

Business facts consistency checker

Compare the business name, phone, email, address, and official profiles stated on your homepage, About page, Contact page, and JSON-LD, and see where any value differs.

Check business facts

Website launch and migration check

Compare the new version of a site or page with the current live one under the same audit rules, and see regressions in crawler access, content, and identity before you launch.

Compare before and after

What this check looks for

See the public evidence behind the result

OpenForBots examines the relevant public page or resource, shows what it found, and explains what to review next.

1
Retrieves robots.txt from the public standard location over HTTPS.
2
Resolves the tested root path for each documented crawler token.
3
Shows the matched allow or disallow rule and source line when available.
4
Keeps crawler purpose, direct Search relevance, and provider-documented robots.txt applicability separate.
5
Treats non-applicable or uncertain user-triggered robots controls as declared policy evidence rather than provider-enforcement proof.
6
Links each crawler result to its current provider documentation.

Example evidence result

OAI-SearchBot is blocked at the tested root path

The live result uses the deployed page or resource and exact check time. This example shows the evidence structure, not a claim about a real domain.

Observed evidence

Observed source: https://example.com/robots.txt
Matched rule: Disallow: /
Documented purpose: search discovery
What the result means: the deployed policy disallows this token at /

Recommended next action

Confirm that the restriction is intentional, change only the relevant rule when authorized, then run a fresh check against the deployed policy.

Understand the result

What the check confirms—and what it cannot tell you

What it can confirm

What robots.txt response OpenForBots observed at check time.
Which directive syntactically matches the documented token at the tested root path.
Whether the provider documents robots.txt as a reliable control for that token when current source data supports the conclusion.

What remains unknown

An allowed policy does not prove that the provider visited or indexed the page.
A blocked training crawler does not imply that a separate Search crawler is blocked.
OpenAI says robots.txt rules may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores robots.txt; those parser matches are not enforcement verdicts.
robots.txt does not override authentication, CDN challenges, WAF rules, or response failures.
The focused result tests the root path; important subpaths may resolve differently.

Learn and take action

Related guides and methodology

Don't have a robots.txt yet? Build one

Generate a starting robots.txt from a declared policy goal and the documented crawler registry.

What changed on September 15, 2026, how Cloudflare can add rules to your robots.txt, and how to check what crawlers are told.

Policy, public delivery, initial HTML, and fresh verification explained as separate checks.

Understand why provider tokens with different purposes need separate policy decisions.

Questions about this check

Does allowing an AI crawler guarantee citations?

No. The checker evaluates the observed robots.txt policy. It cannot prove provider visits, indexing, retrieval, citation, ranking, recommendation, model use, traffic, or revenue.

Does blocking GPTBot also block ChatGPT search?

Not necessarily. OpenAI documents separate tokens for different purposes. The checker evaluates GPTBot and OAI-SearchBot independently rather than treating them as one control.

Why does my robots.txt show rules I did not write?

If your site uses Cloudflare, it can add a section between "BEGIN Cloudflare Managed content" and "END Cloudflare Managed Content" comments to the robots.txt it serves. The checker marks those rules as added by Cloudflare. They are controlled by Cloudflare’s AI bot settings, not by the file you deploy.

Does this test every page on the website?

This focused result resolves the root path. Use the evidence and methodology to review important subpaths separately when their rules or delivery conditions may differ.

Does a blocked ChatGPT-User or Perplexity-User rule prove those user-requested fetches are blocked?

No. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt because the fetch is user-requested. OpenForBots preserves the matching directive as site-declared evidence without converting it into guaranteed provider enforcement.