How the OpenForBots AI Discovery Audit Works

What the audit checks, how evidence is labelled, how crawler policy is interpreted, and what the results cannot prove.

OpenForBots examines a small set of public website resources and connects every finding to the evidence that produced it. It checks crawler policy, the public homepage response, initial HTML, business clarity, structured data, llms.txt, and public identity signals. It does not claim to reproduce an AI provider’s private index, ranking system, source-selection process, or model behaviour.

The audit is intended to answer practical questions such as:

• Can the public website and standard policy files be reached?
• What do the deployed robots.txt rules say for documented crawler tokens?
• Does the initial response clearly identify the organisation, offer, audience, and purpose?
• Are structured data and public identity signals consistent with visible content?
• What should be reviewed or changed first?
• What should a fresh check confirm after that change?

What the audit requests

The audit accepts a public hostname, such as example.com. It does not accept credentials, private network addresses, local development hosts, arbitrary ports, or authenticated content.

For a normal check, OpenForBots may request:

• the public homepage;
• /robots.txt;
• /llms.txt;
• the secure redirect destination when the submitted hostname redirects;
• another public resource only when a displayed check clearly requires it.

Requests have time, response-size, redirect, and content-type limits. These limits protect the service and keep the check focused on information a normal public visitor or automated client could request.

What happens during a check

1. The hostname is validated

OpenForBots normalises the hostname and confirms that it points to a public destination. Redirect destinations are checked again before they are requested.

2. Public responses are collected

The audit records response availability, status, redirect information, relevant headers, and the safely limited content needed for the checks. Completed reports and raw page HTML are not intentionally stored after the request finishes.

3. Crawler policy is interpreted

The robots.txt response is evaluated for documented crawler tokens. Search discovery, model development, user-requested retrieval, ordinary search, and data-use controls are kept separate because providers often publish different controls for different purposes.

A block for GPTBot, for example, is not presented as proof that OAI-SearchBot is blocked. The result shows the token, tested path, matching rule, and the provider source used to describe that token’s purpose.

4. The initial HTML is inspected

OpenForBots reads the HTML returned in the first public response. It does not run an unrestricted browser session or click through interactive workflows.

The check looks for signals including:

• page title, main heading, and opening explanation;
• canonical and robots directives;
• meaningful visible text in the initial response;
• organisation, product, category, audience, and outcome signals;
• crawlable links;
• selected JSON-LD and other structured data;
• public identity and trust references.

A page that relies heavily on client-side rendering may receive a caution. That caution means the initial response contains less useful information; it does not prove that a particular provider cannot render the page.

5. Optional llms.txt information is reviewed

When /llms.txt is available, OpenForBots checks response availability, basic document structure, sections, links, and obvious maintenance issues. A missing file is treated as an optional opportunity, not as proof that the website is blocked or unable to appear in AI-supported experiences.

6. Findings are organised into practical next steps

The report keeps access, content, structure, identity, and verification findings separate. It highlights strong foundations, important concerns, and the first useful action without combining everything into a single visibility score.

Each recommended action should explain:

• what was observed;
• why it may matter;
• what the evidence does not prove;
• the smallest useful next step;
• what a later public check should confirm.

How evidence labels work

OpenForBots uses evidence labels so a reader can tell how strongly a statement is supported.

Observed

The relevant public response or value was collected during the check. Examples include an HTTP status, a matching robots.txt rule, or text present in the initial HTML.

Documented

The statement comes from a provider-maintained source or a primary standard. Provider documentation can change, so crawler references include a source and review date.

Simulated

OpenForBots sent a controlled request using a documented token or request pattern. This can reveal a response difference, but it does not authenticate the request as coming from the real provider.

Heuristic

A repeatable rule found a pattern that requires interpretation, such as very little meaningful initial HTML or an unclear opening explanation. A heuristic is a useful warning, not a proven provider outcome.

Inferred

The conclusion follows from several observations but was not directly measured. The report should show the supporting evidence and make the inference clear.

Manual review

The question cannot be answered reliably from the public check alone. Examples include whether a provider has indexed the newest page, why a particular answer selected one source, or whether a complex interactive workflow is fully usable by an external agent.

How to interpret crawler results

Crawler settings reflect the website owner’s policy. The same rule can be appropriate for one business and unsuitable for another.

The audit report includes an optional declared policy goal comparison on the crawler-policy result. It is off by default: nothing on the report changes unless a goal is selected, and the underlying robots.txt observation is identical either way. Selecting a goal only adds an extra column of interpretation on top of the same evidence.

Six goals are available:

For each documented crawler token, comparing the observed robots.txt rule against the selected goal produces one of five results:

• Aligned — the observed rule matches what the goal asks for.
• Conflicting — the observed rule contradicts the goal (for example, the goal wants a training crawler blocked, but nothing in the deployed file actually blocks it).
• Intentionally restricted — the crawler is blocked, but the goal expressed no preference for its purpose. This is treated as a deliberate site-owner choice, not a problem.
• Undocumented purpose — the provider has not clearly documented this crawler’s purpose, so alignment is not claimed either way.
• Not determined — no goal was selected, or the crawler’s purpose falls outside the supported categories.

The underlying observation never changes with the selected goal. Only which of these five labels is shown changes — and a blocked training crawler is never treated as a discovery failure just because a goal was selected.

Delivery diagnostics for selected paths

The homepage check on its own cannot show whether other important pages — a pricing page, a key blog post — are delivered normally. Delivery diagnostics is an optional addition: enter up to four same-site paths before running the audit, and OpenForBots checks the homepage plus those paths for redirects, denials, rate limits, errors, or unsupported responses. Leaving the field blank skips this check entirely; nothing else about the audit changes.

Each path is requested twice, under two request profiles:

• Default request — the same request OpenForBots makes for the homepage check, evidence-labelled observed.
• Simulated automated client — the same request with generic, non-browser-shaped headers, evidence-labelled simulated.

Both profiles are real requests made by OpenForBots. Neither impersonates a specific AI provider’s crawler — that is a deliberate boundary, not a limitation of what was technically possible. The simulated profile exists only to reveal whether a broad bot-mitigation rule treats any non-browser-shaped request differently, which is a legitimate delivery signal on its own.

Each checked path receives one of these outcomes: available, redirected off-site, not found, denied, rate limited, server error, timed out, oversized, or an unsupported content type. A same-site redirect is followed (up to two hops); a redirect to a different hostname is recorded but never followed. When the two request profiles disagree on a path, the report shows the difference as a review signal — for example, “the default request was available, but the simulated automated-client request was denied” — without asserting a cause. A difference between two OpenForBots-originated requests is not proof that a specific provider is blocked.

Suggested fixes and remediation guidance

Where the evidence supports it, the report includes a suggested robots.txt snippet or short guidance note alongside a finding, so acting on a result doesn’t require re-deriving the fix by hand.

Two kinds of remediation appear, both generated deterministically from evidence already in the report — no AI model is involved, and identical evidence always produces identical output:

• robots.txt patch snippets — shown next to a conflicting crawler in the declared-policy-goal comparison. If the crawler has no dedicated group in the deployed file, the report offers an append-only Allow or Disallow group for that token, with copy and download actions. If the crawler already has its own dedicated group, no snippet is generated — appending a second group for the same token would be ambiguous under robots.txt group-matching rules, so the report instead explains that the existing group needs a manual edit.
• Delivery guidance — shown next to a path with a non-available delivery-diagnostics outcome (denied, rate limited, server error, and so on). This is a short, provider-agnostic pointer at the category of configuration to check (bot-management, WAF, rate limiting, hosting) — never a guess at a specific vendor or product.

A suggested snippet is a proposed addition, not a change OpenForBots has made to the site. Copying or downloading it does not alter robots.txt; the change still has to be deployed by the site owner, after which “Re-run audit” confirms whether it took effect. Remediation is only ever offered for the same five-way alignment result described above — a crawler whose purpose is undocumented or not determined never receives a snippet, since suggesting a fix for an ambiguous finding would overstate what the evidence actually shows.

Security and privacy safeguards

The audit is limited to public website information. It does not intentionally request private or authenticated content.

Safeguards include:

• public-hostname validation;
• blocking private, local, and reserved network destinations;
• rechecking redirects;
• limits on request duration and response size;
• content-type checks;
• safe handling of the evidence shown in the report;
• no intentional storage of completed reports or raw page HTML after the request finishes.

robots.txt is a crawler-policy file, not a security control. Sensitive information should never rely on robots.txt for protection.

What the audit cannot determine

A public technical check cannot establish:

• whether a provider has indexed the latest page;
• whether a specific question will produce a citation;
• how a provider ranks or selects sources;
• whether a particular document was used for model development;
• whether a crawler-looking request in server logs is authentic without provider-supported verification;
• the full experience after arbitrary browser interaction;
• private analytics, Search Console, backlink, revenue, or customer data;
• whether a website change caused a later traffic or citation change.

These questions require different evidence. OpenForBots states that boundary rather than replacing it with a speculative score.

How to reproduce or challenge a finding

A useful finding should identify the public URL, check time, relevant evidence, provider source where applicable, and the condition a fresh check should pass.

To challenge a finding:

1. confirm that the deployed public response has not changed since the check;
2. identify the exact observation or interpretation that appears wrong;
3. provide a current primary source when disputing a crawler purpose or standard;
4. send the reproducible details through the methodology feedback route.

Corrections should improve both the affected explanation and the checks that protect it from returning.

Next step

Run website audit and use the evidence, limitations, and verification condition to decide whether a change is appropriate for your website.