Free AI crawler policy checker

Check robots.txt rules for documented AI crawlers

Enter a public hostname to test whether its deployed robots.txt policy allows, blocks, or has no matching restriction for documented AI crawler tokens at the root path.

The result separates search-discovery, model-training, user-triggered, and other documented crawler purposes. It reports policy evidence only; it does not claim that a provider visited, indexed, cited, or used the site.

Public website check Inspectable evidence Clear limitations

Methodology reviewed 2026-08-05

More free checks

Try the other free AI discovery checks

Each check looks at a different layer of AI discovery readiness. Run them together for a fuller picture, or open the complete audit for all of it at once.

Inspect a deployed llms.txt file and its linked resources

Enter a public hostname to inspect its root-level llms.txt response, title, introduction, sections, links, malformed Markdown, duplicate destinations, and insecure URLs.

Inspect llms.txt

Check how clearly your website identifies the business and its entities

Enter a public hostname to compare the visible business name and opening copy with the Organization, WebSite, Person, or SoftwareApplication entities declared in the homepage JSON-LD.

Check structure and identity

Check Recipe structured data on a food publisher page

Enter a public hostname to inspect Recipe entities declared in its homepage JSON-LD and see which of Google’s documented required and recommended Recipe properties were observed.

Check Recipe schema

Build a starting llms.txt file

Fill in a title, short context, and the resource links you want to list, and this page builds a valid llms.txt file for you to review, edit, and publish.

Build llms.txt

Build a starting robots.txt from a declared policy goal

Choose a declared policy goal and this page builds robots.txt groups for the documented AI crawler tokens that goal has an explicit preference for, which you can review, edit, and publish.

Build robots.txt

Build a starting Organization JSON-LD block

Fill in your business name, canonical URL, logo, and official profile links, and this page builds a valid Organization JSON-LD block for you to review, edit, and publish.

Build JSON-LD

What this check looks for

See the public evidence behind the result

OpenForBots examines the relevant public page or resource, shows what it found, and explains what to review next.

1
Retrieves robots.txt from the public standard location over HTTPS.
2
Evaluates the tested root path for each documented crawler token.
3
Shows the matched allow or disallow rule and source line when available.
4
Keeps crawler purpose and search relevance separate from the policy verdict.
5
Links each crawler result to its current provider documentation.

Example evidence result

OAI-SearchBot is blocked at the tested root path

The live result uses the deployed page or resource and exact check time. This example shows the evidence structure, not a claim about a real domain.

Observed evidence

Observed source: https://example.com/robots.txt
Matched rule: Disallow: /
Documented purpose: search discovery
What the result means: the deployed policy disallows this token at /

Recommended next action

Confirm that the restriction is intentional, change only the relevant rule when authorized, then run a fresh check against the deployed policy.

Understand the result

What the check confirms—and what it cannot tell you

What it can confirm

What robots.txt response OpenForBots observed at check time.
How the documented token resolves for the tested root path.
Which exact rule produced an explicit allow or block verdict.

What remains unknown

An allowed policy does not prove that the provider visited or indexed the page.
A blocked training crawler does not necessarily block a separate search crawler.
robots.txt does not override authentication, CDN challenges, WAF rules, or response failures.
The focused result tests the root path; important subpaths may resolve differently.

Learn and take action

Related guides and methodology

Don't have a robots.txt yet? Build one

Generate a starting robots.txt from a declared policy goal and the documented crawler registry.

Policy, public delivery, initial HTML, and fresh verification explained as separate checks.

Understand why provider tokens with different purposes need separate policy decisions.

Questions about this check

Does allowing an AI crawler guarantee citations?

No. The checker evaluates the observed robots.txt policy. It cannot prove provider visits, indexing, retrieval, citation, ranking, recommendation, model use, traffic, or revenue.

Does blocking GPTBot also block ChatGPT search?

Not necessarily. OpenAI documents separate tokens for different purposes. The checker evaluates GPTBot and OAI-SearchBot independently rather than treating them as one control.

Does this test every page on the website?

This focused result resolves the root path. Use the evidence and methodology to review important subpaths separately when their rules or delivery conditions may differ.