Methodology / Changelog
Version history
Methodology changelog
This page records material changes that can affect how evidence is classified, findings are presented, actions are ordered, or public trust records are interpreted. Editorial corrections that do not change meaning may be applied without a separate entry.
Current report contract: ofb-audit-r2.1. History reconciled through .
The robots.txt checker now shows rules Cloudflare adds
When Cloudflare adds its managed section to the robots.txt your site serves, the robots.txt AI crawler checker and the audit now say which crawler results come from that section, so you know to change Cloudflare's settings rather than your own file.
Platform-managed robots.txt sections identified
- Rule family
- Crawler policy
- Current version
ofb-platform-robots-r1-2026-10-19
New
- The robots.txt AI crawler checker shows the Cloudflare Managed content section when it finds one: its lines, the crawler groups inside it, and any content signal. Crawlers blocked or named by that section are marked "Added by Cloudflare".
- You can compare the checked rules with a goal, such as allowing search while opting out of training, and see which conflicts come from Cloudflare's section.
- A guide to Cloudflare's September 15, 2026 AI crawler defaults and how to check what your site now tells AI crawlers.
Fixed
- For a conflict decided by Cloudflare's section, the suggested fix in the audit report no longer tells you to edit a robots.txt group that is not in your own file. It points to Cloudflare's AI bot settings instead.
Good to know
- The check reads the robots.txt Cloudflare serves. It cannot see blocking that Cloudflare applies at its network edge.
- Crawler verdicts are unchanged; only where each rule came from is new.
Audits now report noindex rules
The audit, the launch check, and site comparison now read noindex rules from robots meta tags and the X-Robots-Tag response header, so a noindex left over from staging is reported before it keeps a page out of search results.
Indexing rules added
- Rule family
- Audit criteria
- Version
ofb-criteria-r1-2026-08-09ofb-criteria-r2-2026-10-18
New
- A "Page asks search engines not to index it" finding when the inspected page has a noindex or none rule in a robots, googlebot, googlebot-news, or bingbot meta tag, or in the X-Robots-Tag header. The finding quotes the exact tag or header.
- The launch and migration check lists a noindex on the new version that the live site does not send as a regression.
Good to know
- The finding shows the rule the page sends; it does not show whether the page is indexed today.
- Rules added by JavaScript after the page loads are not evaluated.
- In a re-check against a saved audit from before this change, the first noindex result is shown as new, not as a change to your site.
Blocked search crawlers are reported in audits again
Since 4 October 2026, the full audit could miss a robots.txt rule that blocks a search crawler such as Googlebot or OAI-SearchBot. Audits now report these blocks again.
Crawler applicability restored
- Rule family
- Crawler policy alignment
- Current version
ofb-policy-alignment-r1.1-2026-10-04
Fixed
- A robots.txt rule that blocks a documented search crawler appears again as a "Search-relevant crawler is blocked" finding, with the matching rule as evidence.
- Provider notes for user-triggered agents, such as Perplexity-User generally ignoring robots.txt, appear in crawler results again.
Good to know
- The per-crawler allowed and blocked verdicts were correct throughout; only the report findings and summaries built from them were affected.
- Run a fresh audit to see the corrected result. Saved reports from the affected period are not changed.
Recipe checks now inspect the page you actually want to test
Recipe and Organization checks now stay closer to the submitted page, current provider requirements, and Schema.org identity semantics.
Structured-data interpretation updated
- Rule family
- Structured data and entity identity
- Current version
ofb-jsonld-r1.2-2026-10-04
New
- Recipe Schema Checker can now inspect the actual public recipe page URL instead of falling back to the site homepage.
Improved
- Organization findings now distinguish provider-required properties from OpenForBots recommendations.
- JSON-LD Generator now creates the stable #organization identifier used by the adjacent identity guidance.
- sameAs guidance now follows Schema.org identity-reference semantics instead of treating a profile link as ownership proof.
Good to know
- Page-targeted Recipe diagnostics are not added to normal saved-site audit history.
- Valid structured data does not guarantee rich results, ranking, retrieval, or AI citation.
Crawler access checks now reflect provider-specific behavior
Crawler findings now separate documented provider behavior, site-declared robots.txt policy, and what OpenForBots cannot verify.
Crawler-policy interpretation updated
- Rule family
- Crawler policy alignment
- Current version
ofb-policy-alignment-r1.1-2026-10-04
Improved
- OpenAI, Anthropic, and Perplexity user-triggered crawler behavior is interpreted according to each provider's published robots.txt guidance.
- Search-relevant findings no longer inherit unrelated training or indirect-crawler states.
- Anthropic crawler IP references are current, and llms.txt guidance now covers v2 scoped files and discovery links.
Fixed
- Policy-goal alignment no longer presents an uncertain or non-applicable robots.txt match as provider-enforced behavior.
Good to know
- The current llms.txt checker still evaluates the root file only.
- A robots.txt match does not prove enforcement when a provider documents different behavior for that crawler.
Saved audit history and account controls are now documented accurately
Public documentation now describes the shipped retention, saved-history, comparison, export, and deletion boundaries consistently.
Interpretation and disclosure updated
- Rule family
- Persistence and history comparison
Improved
- Public documentation now distinguishes anonymous 30-day retention from audit history a signed-in user explicitly saves.
- Account export and deletion controls are described consistently across Privacy, Terms, the Trust Center, and the self-audit.
- Same-site history comparison now explains the difference between website changes and methodology-driven changes.
Good to know
- The basic public audit remains available without an account.
Public change history reconciled
Consolidated the product and methodology change records into one dated source, backfilled verified material releases, and kept the methodology page as the narrower view of changes that affect evidence, reports, verification, or public trust interpretation.
Owner-first report overview and finding presentation
Reorganized audit and comparison reports around what needs attention, what to change first, what is working, what could not be checked, and how to verify a change. Technical evidence remains available in labelled disclosures, while the mobile report header keeps the result and next action close at hand.
Published real audit and fresh-verification proof
Added a source-backed InvoiceCraftly audit example showing observed findings, implemented changes, fresh re-check outcomes, and the limits of what a public technical audit can prove.
Citation Forensics evidence comparison
Added a tool for comparing public evidence after a user records an AI-answer citation outcome. Results keep the observed outcome, verified differences, plausible tests, and unknown provider logic separate, and do not claim to prove why a source was cited.
Methodology-aware saved-report comparison
Saved-report comparisons now distinguish website evidence changes from evaluation-rule changes, preserve unavailable and incompatible states, and show supported Organization identity changes without treating harmless ordering differences as changes.
Crawler records separated from site registration
Clarified that the crawler registry documents provider-published crawler identities and purposes, while site submission, verification, indexing, and inclusion are separate provider-controlled processes.
Two-site evidence comparison and gap plan
Added an ephemeral two-site comparison keyed to stable audit criteria. It reports bounded lead, parity, shared-gap, and not-comparable states, then orders existing evidence-backed actions without creating a score or claiming provider outcomes.
Finding-level fresh re-check outcomes
Fresh verification now shows whether each compatible finding was resolved, remains open, regressed, or could not be compared, instead of relying only on a report-level before-and-after summary.
Audit retention disclosures and request protection
Enabled best-effort anonymous audit snapshot retention for up to 30 days, documented the handling boundary, and added Turnstile protection to public audit and focused-check requests without making an account mandatory.
Evidence-first finding contract
Added stable finding identities, explicit evidence and confidence fields, preserved unavailable states, evidence-backed recommendations, and fresh-verification conditions across report producers and portable output.
Recipe checks and deterministic file builders
Added a Recipe structured-data audit pack plus browser-local robots.txt and llms.txt builders. Builder output is a deterministic draft for review, not evidence that a live site changed.
Citation Lab observation records
Added dated, manual observations of answer-engine mentions and citations with prompt, source, accuracy, and limitation fields. An observation describes one checked result and does not predict future discovery or citation.
Stateless fresh re-check and local comparison
Added a fresh audit action and browser-local before-and-after comparison. A finding changes state only from newly retrieved evidence; marking work complete in the interface does not count as verification.
Declared policy goals, delivery diagnostics, and remediation
Added crawler-purpose policy goals, critical-path public delivery diagnostics, and deterministic remediation suggestions tied to observed evidence and stated verification conditions.
Portable PDF and JSON reports
Added browser-local PDF export and bounded JSON export/import while preserving report versions, evidence, limitations, action ordering, and the original checked time.
Focused public checkers and shared report rendering
Added focused robots.txt, llms.txt, and structure-and-identity checks, and consolidated audit results onto the shared report renderer so the same evidence states and actions are presented consistently.
Public trust records, accessibility, and evidence-presentation corrections
Corrected preview-era methodology versions, stale freshness and scoring wording, dark-section contrast, and locale-dependent report timestamps. Added shared release and self-audit records, the Trust Center, generated llms.txt and security.txt, visible privacy assurances, and regression checks.
Report scanability and trust-readiness contract
Added progressive evidence disclosure, active in-page navigation, discoverability metadata, public feedback routes, and explicit version and freshness presentation without changing audit evidence or roadmap ordering.
Partial-audit roadmap semantics
Partial reports now prioritize retrying unavailable checks and no longer present maintenance as the first action when required evidence could not be retrieved.
Evidence-based combined report and deterministic roadmap
Introduced one report contract for observations, findings, capability status, prioritized actions, dependencies, limitations, and later verification conditions.