How the OpenForBots AI Discovery Audit Works

What the audit checks, how evidence is labelled, how crawler policy and structured-data requirements are interpreted, how optional saved history works, and what the results cannot prove.

OpenForBots examines a bounded set of public website resources and connects every finding to the evidence that produced it. The normal full audit checks crawler policy, the public homepage response, initial HTML, business clarity, structured data, llms.txt, public identity signals, and optional same-site delivery paths. A clearly labelled focused tool may instead inspect a specific public same-site page when that page is the evidence target, such as a Recipe structured-data check. OpenForBots does not claim to reproduce an AI provider’s private index, ranking system, source-selection process, or model behaviour.

The audit is intended to answer practical questions such as:

• can the relevant public resources be retrieved;
• what crawler policy is actually deployed;
• whether that provider documents robots.txt as an applicable control for the tested token;
• whether the initial response contains useful content;
• whether business and entity identity are clear;
• whether structured declarations are internally coherent;
• which structured-data conclusions come from provider documentation and which are OpenForBots heuristics;
• what should be changed first;
• what a later fresh check must confirm.

What OpenForBots requests

For a normal check, OpenForBots may request:

• the submitted public hostname;
• /robots.txt;
• /llms.txt;
• the secure redirect destination when the submitted hostname redirects;
• selected same-site public paths when the user explicitly supplies them for delivery diagnostics;
• a specific same-site public page when a displayed focused check clearly requires that page as its evidence target;
• another public resource only when a displayed check clearly requires it.

Requests have time, response-size, redirect, and content-type limits. These limits protect the service and keep the check focused on information a normal public visitor or automated client could request.

What happens during a check

1. The hostname is validated

OpenForBots normalises the hostname and confirms that it points to a public destination. Redirect destinations are checked again before they are requested.

2. Public responses are collected

The audit records response availability, status, redirect information, relevant headers, and the safely limited content needed for the checks. Normal completed homepage/site audit snapshots are persisted as described in Privacy. Raw fetched page HTML is not intentionally stored as a separate long-term page archive.

An anonymous, unclaimed audit snapshot is retained for up to 30 days. A user may optionally sign in and explicitly save the audit to personal history; saved history is retained until the user deletes the audit, the saved site, or the account.

A page-targeted focused diagnostic is not added to the normal saved-site audit-history store in the current release. This prevents a specific page result from later being compared with a homepage snapshot as though both represented the same evidence target.

3. Crawler policy is interpreted

The robots.txt response is parsed for documented crawler tokens, but a syntactic rule match is not automatically treated as proof that the provider enforces that rule for every token.

OpenForBots first keeps the provider’s documented purpose and control semantics separate:

• search discovery;
• model development or training;
• user-requested retrieval;
• ordinary search;
• data-use controls that may not identify a separate HTTP crawler.

For automatic crawlers where the provider documents robots.txt as an applicable control, the result can state whether the tested path matches an allow, disallow, or no matching restriction. A block for GPTBot, for example, is not presented as proof that OAI-SearchBot is blocked.

User-triggered agents require provider-specific treatment. Current provider documentation differs: Anthropic says its named bots, including Claude-User, honor robots.txt; OpenAI says robots.txt rules may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores robots.txt because the fetch is user-requested. For a token in the latter two cases, OpenForBots can still show the matching directive as site-declared policy evidence, but policy-goal alignment remains not determined rather than claiming provider enforcement.

The report keeps direct Search relevance distinct from indirect or non-Search purposes. A blocked training crawler is therefore not counted as a blocked Search crawler merely because it has a registry entry.

The result shows the token, tested path, matching rule, provider source, purpose, and relevant limitation. Provider documentation can change, so these records carry review dates and current primary-source links.

Some CDNs add their own rules to the robots.txt they serve, so the file crawlers read can differ from the file a site deploys. When the served file contains Cloudflare’s BEGIN Cloudflare Managed content and END Cloudflare Managed Content markers, each result also records whether its deciding rule sits inside that section. Conflicts decided there point to Cloudflare’s AI bot settings rather than to a robots.txt edit. The audit reads the served file only; it cannot see blocking a CDN applies at its network edge.

4. The initial HTML is inspected

OpenForBots reads the HTML returned in the first public response. For the normal full audit this is the homepage response. For a page-targeted focused diagnostic, it is the specific public page the user submitted. OpenForBots does not run an unrestricted browser session or click through interactive workflows.

The check looks for signals including:

• page title, main heading, and opening explanation;
• the canonical link;
• noindex and none rules in robots meta tags (robots, googlebot, googlebot-news, bingbot) and in the X-Robots-Tag response header;
• meaningful visible text in the initial response;
• organisation, product, category, audience, and outcome signals;
• crawlable links;
• selected JSON-LD and other structured data;
• public identity and trust references.

A noindex rule is reported as needing attention because search engines that support it, such as Google, do not show the page in their results. The finding shows the rule as observed; it does not show whether the page is indexed today. Crawlers blocked by robots.txt cannot read the rule, and rules added by JavaScript after the page loads are not evaluated.

A page that relies heavily on client-side rendering may receive a caution. That caution means the initial response contains less useful information; it does not prove that a particular provider cannot render the page.

5. Optional llms.txt information is reviewed

OpenForBots currently checks the root /llms.txt file. When it is available, the audit reviews response availability, basic document structure, sections, links, and obvious maintenance issues.

The llms.txt v2 proposal also permits subpath-scoped files such as /docs/llms.txt and describes standard discovery links. The current OpenForBots checker does not yet discover or validate those scoped files or discovery relations. Therefore, a missing root file means only that /llms.txt was not found; it is not proof that no scoped v2 file exists elsewhere, and it is not a universal visibility failure.

6. Findings are organised into practical next steps

The report keeps access, content, structure, identity, and verification findings separate. It highlights strong foundations, important concerns, and the first useful action without combining everything into a single visibility score.

Each recommended action should explain:

• what was observed;
• why it may matter;
• what the evidence does not prove;
• the smallest useful next step;
• what a later public check should confirm.

Optional saved history and same-site comparison

The basic public audit does not require an account. After a completed homepage/site audit, a user can optionally sign in and explicitly save that audit to a personal history. Saved snapshots preserve the report that was produced at check time, including its methodology and rule versions, rather than silently reinterpreting old evidence under current rules.

When two saved audits for the same site are available, OpenForBots can compare them on demand. The comparison classifies supported changes as:

• Website evidence changed — the relevant public evidence differs under compatible evaluation rules.
• OpenForBots evaluation changed — a rule, parser, methodology, or capability version changed, so the difference must not be presented as a website regression.
• Cause unclear — the available evidence does not support a reliable attribution.

Unavailable, partial, and incompatible evidence remains visible instead of being converted into a pass or failure. The history comparison is manual and on demand; it is not scheduled monitoring, an alerting service, or proof that an external AI provider changed its treatment of the site. Page-targeted focused diagnostics are not added to this homepage/site history model in the current release.

How evidence labels work

OpenForBots uses evidence labels so a reader can tell how strongly a statement is supported.

Observed

The relevant public response or value was collected during the check. Examples include an HTTP status, a matching robots.txt rule, or text present in the initial HTML.

Documented

The statement comes from a provider-maintained source or a primary standard. Provider documentation can change, so provider-backed structured-data and crawler conclusions include current source review in the methodology and rule version history.

Simulated

OpenForBots sent a controlled request using a documented token or request pattern. This can reveal a response difference, but it does not authenticate the request as coming from the real provider.

Heuristic

The result comes from a bounded OpenForBots rule or threshold rather than a provider-published requirement. Heuristic findings are labelled and should be reviewed by a person before a production change is made. A stable structured-data @id, for example, can be a useful graph-consistency convention without being presented as a Schema.org or Google requirement.

Inferred

The conclusion combines observed or documented facts but is not itself directly stated by the source. Inferences should remain narrower than the evidence behind them.

Manual check

The product does not have enough deterministic evidence to claim an automated answer. The report states what a reviewer should inspect manually.

How crawler-purpose confidence works

OpenForBots does not infer a crawler’s purpose solely from its name. Each registry record carries a purpose-confidence state.

• Documented means the provider describes the purpose in a maintained source.
• Inferred means available sources support a cautious interpretation but do not state it directly enough for policy alignment.
• Unknown means OpenForBots cannot currently classify the purpose responsibly.

Policy-goal alignment requires documented purpose confidence. A crawler with inferred or unknown purpose remains outside deterministic alignment conclusions.

A separate robots-applicability condition is also required. Even a documented purpose does not justify an aligned/conflicting robots conclusion if the provider says the relevant user-triggered agent may ignore or may not be governed by robots.txt.

How policy-goal alignment works

A site owner can compare observed crawler policy against a declared goal such as maximum public visibility, balanced Search visibility with training restrictions, or a training opt-out.

Alignment is deterministic only when:

1. the crawler purpose is documented;
2. the purpose is part of the supported taxonomy;
3. the provider documents robots.txt as an applicable control for that token;
4. the target state for that purpose is defined;
5. the deployed robots.txt verdict can be compared with that target.

If one of those conditions is not met, the result stays undocumented or not determined rather than manufacturing a pass/fail conclusion.

The current policy-alignment rules version is published in the Trust Center and completed-report methodology metadata so a later comparison can distinguish a site change from an OpenForBots evaluation change.

What structured-data checks mean

OpenForBots parses JSON-LD from the retrieved initial HTML and checks selected deterministic properties and relationships. It can identify malformed blocks, supported entity types, graph identifiers, names, logos, sameAs identity references, and selected relationships.

The structured-data model deliberately separates three things:

• provider-required properties — only a property explicitly documented as required by the relevant provider check can produce a provider-required failure;
• provider-recommended properties — documented recommendations can produce review opportunities but are not converted into required failures;
• OpenForBots focused-shape heuristics — useful consistency checks remain heuristic and are labelled as such rather than presented as provider eligibility rules.

For example, current Google Organization guidance documents no required Organization properties, so a missing Organization URL is not treated as a Google-required failure. For Recipe rich results, current Google guidance requires name and image; when per-serving nutrition calories are declared, recipeYield is also required. OpenForBots surfaces a selected practical subset of Google’s recommended Recipe fields but does not claim to reproduce the complete Rich Results Test.

A stable @id is treated as an OpenForBots graph-consistency heuristic. sameAs follows the Schema.org identity relationship: the URL should unambiguously identify the same entity; it is not restricted to a profile the site owner controls and does not itself prove ownership or authority.

A valid JSON-LD block means only that the observed structured declaration is syntactically or structurally usable under the implemented checks. It does not prove external knowledge-graph recognition, rich results, indexing, ranking, retrieval, or citation.

What business-clarity checks mean

Business-clarity findings use bounded wording heuristics on the initial public HTML. They look for whether the page visibly identifies the organisation or product, category, audience, offer, and intended outcome.

These checks can surface weak or missing public signals. They do not measure what a private model “understands” internally and do not claim semantic certainty.

Why OpenForBots does not use an aggregate score

A single score can hide important differences between evidence types. A website can intentionally block a training crawler while remaining open to search discovery, or have strong structured data while the initial content is unavailable.

OpenForBots therefore reports capability states, findings, evidence, action priority, and verification conditions instead of collapsing everything into one percentage.

What the audit cannot prove

The public audit cannot prove:

• that a named provider actually visited the site;
• that a provider indexed or stored a page;
• that a page is present in a private retrieval system;
• that a provider used content for training;
• that a page will be cited, ranked, recommended, or selected for a prompt;
• that a technical change caused an external answer-engine outcome;
• future traffic, leads, revenue, or conversion impact.

Where provider outcome evidence matters, use separately dated observations or provider-owned tools rather than converting a public technical condition into a stronger claim.

Request and security boundaries

OpenForBots only evaluates public targets and applies request controls intended to reduce SSRF and abuse risk.

Safeguards include:

• hostname normalization;
• DNS and destination validation;
• rejection of private, loopback, and other non-public targets;
• redirect re-validation;
• limits on request duration and response size;
• content-type checks;
• safe handling of the evidence shown in the report;
• bounded report persistence and user-controlled saved-history deletion as described in Privacy.

robots.txt is a crawler-policy file, not a security control. Sensitive information should never rely on robots.txt for protection.

How to reproduce a result

For an audit finding, record:

1. the hostname and tested page or path;
2. the check timestamp;
3. the report and rule versions;
4. the relevant provider or standards source and review date when the finding depends on documented semantics;
5. the observed public response or declaration;
6. whether robots.txt is documented as applicable to the tested token when crawler policy is involved;
7. whether a structured-data conclusion is provider-required, provider-recommended, or heuristic when structured data is involved;
8. the evidence label and limitation;
9. the expected condition after remediation.

Run a fresh check after deployment rather than assuming a code change reached production.

Corrections and change control

Provider references and audit rules can change. Material changes to interpretation, rule versions, or evidence handling are recorded in the methodology changelog.

If you find an outdated provider source, unsupported conclusion, or reproducible discrepancy, use the contact and correction routes. Security-sensitive findings should use the private security-reporting route.