OpenForBotsAI discovery readiness
Run website auditRun audit

Product changelog

What's new in OpenForBots

Follow dated product, report, tool, account, trust, and audit-methodology releases, with the practical impact, limitations, and shipped surfaces kept together.

Latest release
Current report contract
ofb-audit-r2.1
Improvement
ToolsAudit methodMethodology changed

The robots.txt checker now shows rules Cloudflare adds

When Cloudflare adds its managed section to the robots.txt your site serves, the robots.txt AI crawler checker and the audit now say which crawler results come from that section, so you know to change Cloudflare's settings rather than your own file.

Platform-managed robots.txt sections identified

Rule family
Crawler policy
Current version
ofb-platform-robots-r1-2026-10-19

New

  • The robots.txt AI crawler checker shows the Cloudflare Managed content section when it finds one: its lines, the crawler groups inside it, and any content signal. Crawlers blocked or named by that section are marked "Added by Cloudflare".
  • You can compare the checked rules with a goal, such as allowing search while opting out of training, and see which conflicts come from Cloudflare's section.
  • A guide to Cloudflare's September 15, 2026 AI crawler defaults and how to check what your site now tells AI crawlers.

Fixed

  • For a conflict decided by Cloudflare's section, the suggested fix in the audit report no longer tells you to edit a robots.txt group that is not in your own file. It points to Cloudflare's AI bot settings instead.

Good to know

  • The check reads the robots.txt Cloudflare serves. It cannot see blocking that Cloudflare applies at its network edge.
  • Crawler verdicts are unchanged; only where each rule came from is new.
Improvement
Tools

Comparisons now catch a problem that appears on only one side

The launch check and site comparison now treat a check that reports only problems as passed on the side where it found none, so JSON-LD removed from a new version is listed as a regression instead of as not comparable.

Improved

  • Structured data, foundational HTML, page content, business clarity, identity, and robots.txt presence checks report a finding only when they see a problem. A version with no such finding now counts as passing that check, with the rule's pass condition shown as the evidence.
  • JSON-LD removed in a redesign or migration now appears under regressions and in the fix-before-launch list.

Good to know

  • A check that could not run on one side, such as a page that could not be retrieved, or one that runs only on request, is still listed as not comparable.
  • Audit findings and report results are unchanged; only how two audits are compared changed.
Audit method
Audit methodMethodology changed

Audits now report noindex rules

The audit, the launch check, and site comparison now read noindex rules from robots meta tags and the X-Robots-Tag response header, so a noindex left over from staging is reported before it keeps a page out of search results.

Indexing rules added

Rule family
Audit criteria
Version
ofb-criteria-r1-2026-08-09ofb-criteria-r2-2026-10-18

New

  • A "Page asks search engines not to index it" finding when the inspected page has a noindex or none rule in a robots, googlebot, googlebot-news, or bingbot meta tag, or in the X-Robots-Tag header. The finding quotes the exact tag or header.
  • The launch and migration check lists a noindex on the new version that the live site does not send as a regression.

Good to know

  • The finding shows the rule the page sends; it does not show whether the page is indexed today.
  • Rules added by JavaScript after the page loads are not evaluated.
  • In a re-check against a saved audit from before this change, the first noindex result is shown as new, not as a change to your site.
New tool
Tools

Compare a new version with your live site before launch

The new launch and migration check audits the current live site and the new version with the same rules and lists the checks that got worse first.

New

  • Compare a staging host, new domain, or migrated page with the current live version.
  • Regressions, improvements, and gaps in both versions are grouped, with evidence from each side and an action for the new version.

Improved

  • Site comparison now also accepts a full https:// page URL for either side, not only a domain.

Good to know

  • Pages behind a login or IP allowlist cannot be checked, and noindex directives are not checked yet.
  • Neither site is saved.
Fix
Audit methodMethodology changed

Blocked search crawlers are reported in audits again

Since 4 October 2026, the full audit could miss a robots.txt rule that blocks a search crawler such as Googlebot or OAI-SearchBot. Audits now report these blocks again.

Crawler applicability restored

Rule family
Crawler policy alignment
Current version
ofb-policy-alignment-r1.1-2026-10-04

Fixed

  • A robots.txt rule that blocks a documented search crawler appears again as a "Search-relevant crawler is blocked" finding, with the matching rule as evidence.
  • Provider notes for user-triggered agents, such as Perplexity-User generally ignoring robots.txt, appear in crawler results again.

Good to know

  • The per-crawler allowed and blocked verdicts were correct throughout; only the report findings and summaries built from them were affected.
  • Run a fresh audit to see the corrected result. Saved reports from the affected period are not changed.
New tool
Tools

Check that your business facts agree across pages

A new free tool compares the business name, phone, email, postal address, and official profiles stated on your homepage, About page, Contact page, and JSON-LD.

New

  • Every value is listed with the page and source it came from, such as a tel: link or the JSON-LD telephone property.
  • Conflicts, facts that appear in only one source, and facts that were not found are reported separately.

Good to know

  • The check compares the website with itself. It does not verify facts against business registries, maps listings, or the profiles.
  • Only the homepage and the About and Contact pages it links to are checked, and nothing is saved.
Integrations
Tools

Use OpenForBots from Claude Code and other MCP clients

A read-only Model Context Protocol server lets AI assistants and coding agents call OpenForBots checks directly, without an account.

New

  • Look up documented AI crawlers by robots.txt token, provider, or user-agent string.
  • Validate llms.txt drafts with the same rules as the llms.txt checker, without requesting any website.

Good to know

  • Website checks for robots.txt, llms.txt, and the full audit appear in your client’s tool list while they are available on the server.
  • Websites, llms.txt content, and lookup text sent to the server are not saved.
Tool update
Tools

Validate an llms.txt draft before you publish it

The llms.txt checker can now validate pasted content in your browser, in addition to checking the live file on a website.

New

  • llms.txt Checker now has a paste mode for drafts and scoped files such as /docs/llms.txt. It uses the same structure rules as the live check.

Good to know

  • Pasted content is validated in your browser. It is not uploaded and is not included in analytics events.
  • A valid draft does not show that a file is deployed or served correctly; check the live website after publishing.
Reference data
Site experience

Download the AI crawler registry as JSON or CSV

The crawler registry is now available as machine-readable JSON and CSV, generated from the same records as the registry pages.

New

  • Each record includes the provider, robots.txt token, documented purpose, primary source, verification date, and canonical record URL.

Good to know

  • Values the provider does not document are left empty rather than marked false.
  • The dataset is a maintained reference, not a complete list of web crawlers. OpenForBots does not authenticate or certify crawlers.
Structured data release
Audit methodToolsTrust and safetyMethodology changed

Recipe checks now inspect the page you actually want to test

Recipe and Organization checks now stay closer to the submitted page, current provider requirements, and Schema.org identity semantics.

Structured-data interpretation updated

Rule family
Structured data and entity identity
Current version
ofb-jsonld-r1.2-2026-10-04

New

  • Recipe Schema Checker can now inspect the actual public recipe page URL instead of falling back to the site homepage.

Improved

  • Organization findings now distinguish provider-required properties from OpenForBots recommendations.
  • JSON-LD Generator now creates the stable #organization identifier used by the adjacent identity guidance.
  • sameAs guidance now follows Schema.org identity-reference semantics instead of treating a profile link as ownership proof.

Good to know

  • Page-targeted Recipe diagnostics are not added to normal saved-site audit history.
  • Valid structured data does not guarantee rich results, ranking, retrieval, or AI citation.
Crawler policy release
Audit methodToolsTrust and safetyMethodology changed

Crawler access checks now reflect provider-specific behavior

Crawler findings now separate documented provider behavior, site-declared robots.txt policy, and what OpenForBots cannot verify.

Crawler-policy interpretation updated

Rule family
Crawler policy alignment
Current version
ofb-policy-alignment-r1.1-2026-10-04

Improved

  • OpenAI, Anthropic, and Perplexity user-triggered crawler behavior is interpreted according to each provider's published robots.txt guidance.
  • Search-relevant findings no longer inherit unrelated training or indirect-crawler states.
  • Anthropic crawler IP references are current, and llms.txt guidance now covers v2 scoped files and discovery links.

Fixed

  • Policy-goal alignment no longer presents an uncertain or non-applicable robots.txt match as provider-enforced behavior.

Good to know

  • The current llms.txt checker still evaluates the root file only.
  • A robots.txt match does not prove enforcement when a provider documents different behavior for that crawler.
Account and trust release
Trust and safetyAccountsAudit methodMethodology changed

Saved audit history and account controls are now documented accurately

Public documentation now describes the shipped retention, saved-history, comparison, export, and deletion boundaries consistently.

Interpretation and disclosure updated

Rule family
Persistence and history comparison

Improved

  • Public documentation now distinguishes anonymous 30-day retention from audit history a signed-in user explicitly saves.
  • Account export and deletion controls are described consistently across Privacy, Terms, the Trust Center, and the self-audit.
  • Same-site history comparison now explains the difference between website changes and methodology-driven changes.

Good to know

  • The basic public audit remains available without an account.
Trust and safetySite experienceMethodology changed

Public change history reconciled

Consolidated the product and methodology change records into one dated source, backfilled verified material releases, and kept the methodology page as the narrower view of changes that affect evidence, reports, verification, or public trust interpretation.

Audit methodReportsMethodology changed

Owner-first report overview and finding presentation

Reorganized audit and comparison reports around what needs attention, what to change first, what is working, what could not be checked, and how to verify a change. Technical evidence remains available in labelled disclosures, while the mobile report header keeps the result and next action close at hand.

ReportsTrust and safetyMethodology changed

Published real audit and fresh-verification proof

Added a source-backed InvoiceCraftly audit example showing observed findings, implemented changes, fresh re-check outcomes, and the limits of what a public technical audit can prove.

ToolsAudit methodMethodology changed

Citation Forensics evidence comparison

Added a tool for comparing public evidence after a user records an AI-answer citation outcome. Results keep the observed outcome, verified differences, plausible tests, and unknown provider logic separate, and do not claim to prove why a source was cited.

AccountsAudit methodMethodology changed

Methodology-aware saved-report comparison

Saved-report comparisons now distinguish website evidence changes from evaluation-rule changes, preserve unavailable and incompatible states, and show supported Organization identity changes without treating harmless ordering differences as changes.

Trust and safetyAudit methodMethodology changed

Crawler records separated from site registration

Clarified that the crawler registry documents provider-published crawler identities and purposes, while site submission, verification, indexing, and inclusion are separate provider-controlled processes.

ReportsAudit methodMethodology changed

Two-site evidence comparison and gap plan

Added an ephemeral two-site comparison keyed to stable audit criteria. It reports bounded lead, parity, shared-gap, and not-comparable states, then orders existing evidence-backed actions without creating a score or claiming provider outcomes.

AccountsTrust and safety

Saved-audit export and deletion controls

Added user-controlled deletion for saved audits and sites, account data export, and full account deletion while retaining workspace authorization checks.

ReportsAudit methodMethodology changed

Finding-level fresh re-check outcomes

Fresh verification now shows whether each compatible finding was resolved, remains open, regressed, or could not be compared, instead of relying only on a report-level before-and-after summary.

AccountsReports

Saved sites and audit history

Added optional GitHub and Google sign-in, audit claiming, a personal saved-site dashboard, and access-controlled report history. The basic public audit remains available without an account.

ReportsSite experience

Bounded remediation request handoff

Added a contextual route for requesting implementation help from supported audit findings, with a bounded intake and explicit handling disclosures.

Trust and safetyAccountsMethodology changed

Audit retention disclosures and request protection

Enabled best-effort anonymous audit snapshot retention for up to 30 days, documented the handling boundary, and added Turnstile protection to public audit and focused-check requests without making an account mandatory.

Audit methodReportsMethodology changed

Evidence-first finding contract

Added stable finding identities, explicit evidence and confidence fields, preserved unavailable states, evidence-backed recommendations, and fresh-verification conditions across report producers and portable output.

ToolsAudit methodMethodology changed

Recipe checks and deterministic file builders

Added a Recipe structured-data audit pack plus browser-local robots.txt and llms.txt builders. Builder output is a deterministic draft for review, not evidence that a live site changed.

Trust and safetyAudit methodMethodology changed

Citation Lab observation records

Added dated, manual observations of answer-engine mentions and citations with prompt, source, accuracy, and limitation fields. An observation describes one checked result and does not predict future discovery or citation.

ReportsAudit methodMethodology changed

Stateless fresh re-check and local comparison

Added a fresh audit action and browser-local before-and-after comparison. A finding changes state only from newly retrieved evidence; marking work complete in the interface does not count as verification.

Audit methodReportsMethodology changed

Declared policy goals, delivery diagnostics, and remediation

Added crawler-purpose policy goals, critical-path public delivery diagnostics, and deterministic remediation suggestions tied to observed evidence and stated verification conditions.

ReportsMethodology changed

Portable PDF and JSON reports

Added browser-local PDF export and bounded JSON export/import while preserving report versions, evidence, limitations, action ordering, and the original checked time.

Site experienceTrust and safety

Contextual discovery and feedback routes

Added deliberate related-resource and next-action links across important pages, plus dedicated contact, correction, feedback, and partnership routes.

ToolsReportsAudit methodMethodology changed

Focused public checkers and shared report rendering

Added focused robots.txt, llms.txt, and structure-and-identity checks, and consolidated audit results onto the shared report renderer so the same evidence states and actions are presented consistently.

Trust and safetyReportsMethodology changed

Public trust records, accessibility, and evidence-presentation corrections

Corrected preview-era methodology versions, stale freshness and scoring wording, dark-section contrast, and locale-dependent report timestamps. Added shared release and self-audit records, the Trust Center, generated llms.txt and security.txt, visible privacy assurances, and regression checks.

ReportsTrust and safetyMethodology changed

Report scanability and trust-readiness contract

Added progressive evidence disclosure, active in-page navigation, discoverability metadata, public feedback routes, and explicit version and freshness presentation without changing audit evidence or roadmap ordering.

Audit methodReportsMethodology changed

Partial-audit roadmap semantics

Partial reports now prioritize retrying unavailable checks and no longer present maintenance as the first action when required evidence could not be retrieved.

Audit methodReportsMethodology changed

Evidence-based combined report and deterministic roadmap

Introduced one report contract for observations, findings, capability status, prioritized actions, dependencies, limitations, and later verification conditions.