Methodology / How the OpenForBots AI Discovery Audit Works

How the OpenForBots AI Discovery Audit Works

The audit scope, evidence labels, policy goals, deterministic checks, scoring principles, security boundaries, and known limitations.

The OpenForBots AI Discovery Audit is designed as a deterministic, evidence-labelled review of a limited public website surface. It checks crawler policy, retrieves selected public resources, inspects initial HTML, evaluates whether basic business information is extractable, and produces implementation-oriented findings. It does not reproduce a provider’s private index, execute paid model prompts, or guarantee citation, ranking, recommendation, or traffic.

Current status: this page publishes the Release 1 methodology before the live network scanner is enabled. The audit page currently previews inputs and policy interpretation. Results must not be represented as live observations until the secure fetch, rule engine, storage, and report flow are implemented and validated.

Methodology version

Version: 0.1-preview
Last verified: 2026-08-01

Every future report will record the methodology version active at scan time. A material rule, weight, evidence, or applicability change requires a version change and a public changelog entry.

Audit scope

Release 1 evaluates one public domain and a deliberately small set of resources. The normal scan path does not perform unrestricted site crawling and does not run a headless browser.

Planned public requests include, where applicable:

the normalised root URL;
/robots.txt;
/llms.txt when present;
a bounded sitemap resource when required by an applicable rule;
redirect destinations within the secure request policy.

The scanner must remain within strict request, time, size, content-type, and redirect limits.

User policy goals

Crawler findings depend on the site owner’s intent. The audit therefore requires one policy goal.

Allow AI search, restrict training

The balanced default permits recognised search/discovery and relevant user-triggered access while treating documented model-training opt-outs as intentional rather than negative.

Maximise AI access

This goal recommends allowing recognised search, retrieval, and training-related tokens unless another security, privacy, legal, or business requirement overrides that preference.

Analyse without policy recommendation

This goal reports observed rules and documented purposes without marking an intentional allow or block choice as a failure.

The same robots.txt file can therefore produce different recommendations under different goals while the underlying observation remains unchanged.

Evidence taxonomy

Every finding carries exactly one evidence label.

Observed

OpenForBots directly retrieved or parsed the relevant public response during the scan. Examples include an HTTP status, a matching robots.txt rule, or text present in initial HTML.

Observed does not mean that every downstream interpretation is certain. The finding still states its limits.

Documented

The claim comes from an official provider, standard, or primary source. Examples include a provider’s declared crawler purpose or RFC 9309 behaviour.

Provider documentation can change. The crawler registry records a last-verified date and source URL.

Simulated

OpenForBots sent a controlled request or evaluated a request pattern that approximates crawler behaviour but did not authenticate the requester as the provider’s production crawler.

A user-agent simulation must never be described as proof that the real crawler receives the same response.

Heuristic

A deterministic rule identifies a useful pattern that is not a definitive fact. Examples include very little meaningful text in initial HTML or difficulty extracting a clear product category.

Heuristic findings explain the rule and avoid false precision.

Inferred

The conclusion follows from multiple facts but was not directly measured. The report identifies the supporting observations and makes the inference visible.

Manual check

The question is not reliably answerable by the automated Release 1 scan. Examples include whether a provider currently cites the site for a specific prompt or whether a browser-rendered workflow is fully understood.

Audit stages

1. Validate the domain

The input is normalised to an approved HTTP or HTTPS hostname. Release 1 rejects arbitrary paths, credentials, unsupported ports, IP literals, local addresses, and private or reserved network targets.

2. Resolve and validate the target

DNS results are checked before any request. Every redirect destination is resolved and validated again. Redirects are followed manually within a strict cap.

This is required to reduce server-side request forgery risk.

3. Retrieve bounded public resources

Requests use timeouts, response-size limits, content-type checks, and safe decompression limits. The scanner does not store raw fetched HTML after extraction.

4. Evaluate crawler policy

The robots.txt parser resolves product-token groups and path rules using the documented protocol. Each recognised token is mapped to the source-controlled crawler registry.

The audit keeps these purposes separate:

search discovery;
model training;
user-triggered retrieval;
data-use control;
general search;
mixed or uncertain purpose.

A GPTBot block is not reported as a ChatGPT search block. A Google-Extended block is not reported as removal from Google Search.

5. Inspect the initial HTML

The parser extracts a bounded set of information from the returned HTML without executing JavaScript:

title and metadata;
canonical and robots directives;
headings and meaningful text;
organisation and product names;
category, audience, and value-proposition candidates;
crawlable links;
selected structured data;
visible source, date, and evidence signals.

A client-rendering dependency is labelled as a heuristic. The audit does not claim that a specific provider cannot render the page.

6. Evaluate optional resources

When /llms.txt exists, OpenForBots may inspect its response, basic structure, links, and obvious consistency with the public site.

The file’s absence is not a critical failure and does not reduce the core access score.

7. Apply applicable rules

Rules declare their audience, evidence type, policy-goal behaviour, severity, score category, canonical explanation, and rescan condition.

An inapplicable rule does not reduce the score. A local business, SaaS product, publisher, and API platform do not require identical signals.

8. Build the report

The report presents category scores, top fixes, evidence counts, methodology version, timestamp, final resolved URL, and individual findings.

Each finding includes:

title;
status and severity;
evidence label and confidence;
affected URL or resource;
exact or safely truncated evidence;
why the finding matters;
what it does not prove;
recommended fix;
copyable implementation where appropriate;
official or methodology source;
rescan pass condition.

Scoring principles

The score is a summary of applicable deterministic rules, not a prediction of traffic or citations.

Release 1 uses four core categories:

crawler access and policy;
retrieval and initial HTML;
business understanding and structure;
evidence and source readiness.

Agent actionability may appear as a supplemental section when applicable, but missing agent protocols are not universal failures.

Rules follow these constraints:

inapplicable checks carry no penalty;
training opt-outs are interpreted against the selected policy goal;
missing llms.txt is not a critical failure;
generic schema presence is not rewarded without relevance and visible consistency;
heuristics have bounded influence;
no score is labelled as a citation probability;
methodology changes that affect scores update fixtures and version history.

Security and privacy boundaries

The scanner is an internet-facing fetch service and must be treated as a security-sensitive system.

Required controls include:

Turnstile and rate limiting;
public-hostname validation;
private and reserved IPv4 and IPv6 blocking;
DNS-result checking before each request;
redirect revalidation;
timeout, size, decompression, and content-type caps;
safe XML parsing without external entities;
evidence sanitisation;
unguessable report and manage tokens;
no raw HTML storage;
retention limits;
no cross-site tracking identifiers.

Robots.txt is crawler guidance, not security. OpenForBots will not fetch authenticated or private content.

Report privacy

Private reports are the default. They are unlisted, marked noindex, and expire under the Release 1 retention policy.

A public report requires explicit opt-in after the user reviews it, followed by a separate confirmation. Public reports display the domain, timestamp, methodology version, and limitations and can be revoked with the manage token.

Reports must not be published automatically as a link-building tactic.

Known limitations

Release 1 cannot determine automatically:

whether a provider has indexed the latest page;
whether a specific prompt produces a citation;
how a provider weights or ranks a source;
whether model training used a particular document;
whether a user-agent string in logs is authentic without provider-supported verification;
the full content visible after arbitrary browser interaction;
private Search Console, analytics, backlink, brand, or revenue data;
causal impact on traffic after a change.

The report labels these boundaries rather than replacing them with speculative scores.

Reproducing or challenging a finding

A finding should be reproducible from:

the scanned public URL and timestamp;
the methodology and rule version;
the safely recorded evidence;
the official crawler source and last-verified date;
the stated pass condition.

A user should be able to implement the fix, rescan, and see whether the deterministic finding changes.

If a finding is factually wrong, the correction process should identify the affected rule, fixture, crawler record, content page, and reports. Critical incorrect claims remain unresolved launch blockers until corrected.

Next step

Preview the Release 1 audit inputs and review how the selected policy goal changes recommendations without changing the observed evidence.