# OpenForBots

Last reviewed: 2026-08-10
Product release: Release 0.9

> OpenForBots is an evidence-first website audit for AI discovery readiness. It checks robots.txt crawler policy, public retrieval, initial HTML, business clarity, structured data, entity identity, and re-check conditions using evidence-labelled findings. It does not produce an aggregate readiness score or guarantee indexing, citations, rankings, recommendations, traffic, or model use.

## Product
- [Run the public website audit](https://openforbots.com/audit/): Check public crawler policy, homepage delivery, initial HTML, business clarity, structured data, llms.txt, and identity signals, with evidence and practical next steps
- [Free focused diagnostic tools](https://openforbots.com/tools/): Focused public checks for AI crawler rules, llms.txt, and homepage structure and identity
- [Robots.txt AI crawler checker](https://openforbots.com/tools/robots-txt-ai-crawler-checker/): Evaluate documented AI crawler tokens at the tested root path and inspect the exact matched robots.txt rules and limitations
- [llms.txt checker](https://openforbots.com/tools/llms-txt-checker/): Inspect root-level response availability, Markdown structure, linked resources, file-quality findings, and convention limits
- [llms.txt generator](https://openforbots.com/tools/llms-txt-generator/): Build a starting llms.txt from a title, context summary, and resource links, with live preview and download
- [robots.txt generator](https://openforbots.com/tools/robots-txt-generator/): Build a starting robots.txt from a policy goal and documented AI crawler tokens, with live preview and download
- [AI structure and identity checker](https://openforbots.com/tools/ai-structure-and-identity-checker/): Compare visible business naming with supported JSON-LD entities, stable identifiers, official profile relationships, and trust-page signals
- [Recipe schema checker](https://openforbots.com/tools/recipe-schema-audit/): Check Recipe JSON-LD on a food publisher page against Google's documented required and recommended properties
- [AI readiness audits for agencies](https://openforbots.com/for-agencies/ai-readiness-audits/): Use the public audit to qualify technical issues, explain evidence to clients, guide remediation, and run a fresh check after changes

## Core concepts
- [What Is AI Discovery Readiness?](https://openforbots.com/concepts/ai-discovery-readiness/): Operational definition, components, boundaries, and practical checklist
- [What Is AI Discovery?](https://openforbots.com/concepts/what-is-ai-discovery/): How systems find, retrieve, interpret, cite, and use website information
- [What Is Answer Engine Optimization?](https://openforbots.com/concepts/what-is-answer-engine-optimization/): Website practices that improve source clarity and usefulness for answer-oriented discovery without promising citations
- [AI Search vs Training Crawlers](https://openforbots.com/concepts/ai-search-vs-training-crawlers/): Why crawler purpose matters before changing access policy
- [What Is an AI Citation?](https://openforbots.com/concepts/what-is-an-ai-citation/): Citation, retrieval, answer influence, and business outcome explained separately
- [What Is Retrieval-Augmented Generation?](https://openforbots.com/concepts/what-is-retrieval-augmented-generation/): Why external retrieval matters and what website owners can influence

## Practical guides
- [Why AI Tools May Not Cite Your Website](https://openforbots.com/guides/why-ai-tools-do-not-cite-your-website/): Diagnostic sequence across access, retrieval, answer usefulness, identity, source selection, and result variability
- [How to Check AI Crawler Access](https://openforbots.com/guides/how-to-check-ai-crawler-access/): Crawler purpose, robots policy, path scope, public delivery, initial HTML, remediation, and fresh re-checking
- [AI Search Crawlers vs Training Crawlers](https://openforbots.com/guides/search-crawlers-vs-training-crawlers/): Search discovery, training, user-triggered retrieval, ordinary search, and data-use controls
- [GPTBot vs OAI-SearchBot](https://openforbots.com/guides/gptbot-vs-oai-searchbot/): Configure OpenAI search and model-development preferences independently
- [Allow AI Search While Blocking Model Training](https://openforbots.com/guides/allow-ai-search-block-ai-training/): Cautious multi-provider robots.txt policy pattern and verification checklist
- [Is llms.txt Required for AI Visibility?](https://openforbots.com/guides/does-llms-txt-improve-ai-visibility/): Proposal, practical uses, adoption evidence, and non-guarantees
- [AI Discovery Readiness Audit Checklist](https://openforbots.com/guides/ai-discovery-readiness-audit-checklist/): Reusable checklist across access, content, structure, identity, remediation, and verification
- [AI Visibility Tracking vs Technical Readiness](https://openforbots.com/guides/ai-visibility-tracking-vs-technical-readiness/): Outcome observation and website diagnostics compared without false causality
- [Can AI Crawlers Read React Websites?](https://openforbots.com/guides/can-ai-crawlers-read-react-websites/): Initial HTML, SSR, static generation, hydration, and client-rendering limits

## Crawler references
- [OAI-SearchBot](https://openforbots.com/ai-crawlers/oai-searchbot/): OpenAI search-discovery crawler
- [GPTBot](https://openforbots.com/ai-crawlers/gptbot/): OpenAI model-development crawler
- [ChatGPT-User](https://openforbots.com/ai-crawlers/chatgpt-user/): OpenAI user-triggered retrieval
- [Claude-SearchBot](https://openforbots.com/ai-crawlers/claude-searchbot/): Anthropic search-discovery crawler
- [ClaudeBot](https://openforbots.com/ai-crawlers/claudebot/): Anthropic training-related crawler
- [PerplexityBot](https://openforbots.com/ai-crawlers/perplexitybot/): Perplexity search-discovery crawler
- [Google-Extended](https://openforbots.com/ai-crawlers/google-extended/): Google data-use control, not a separate HTTP crawler identity
- [Complete crawler registry](https://openforbots.com/ai-crawlers/): Provider-sourced crawler records with documentation links, purpose notes, and verification dates

## Evidence and trust
- [OpenForBots Trust Center](https://openforbots.com/trust/): Methodology, self-audit, corrections, security, privacy, terms, and current product boundaries
- [OpenForBots self-audit](https://openforbots.com/trust/self-audit/): Findings OpenForBots identified on its own site, the changes made, fresh verification conditions, and known limitations
- [OpenForBots Citation Lab](https://openforbots.com/lab/): Versioned prompt library and manual observations of whether AI answer engines mention or cite OpenForBots, including negative and untested results
- [Published case studies](https://openforbots.com/case-studies/): Public examples where audit evidence, changes, fresh verification, and limitations can be reviewed together
- [Machine-readable self-audit record](https://openforbots.com/trust/self-audit.json): Operator-maintained JSON record; not signed, tamper-evident, or an independent certification

## Methodology and operations
- [How the OpenForBots audit works](https://openforbots.com/methodology/how-the-audit-works/): Scope, evidence labels, crawler-policy interpretation, automated and heuristic checks, security, privacy, and limitations
- [Published methodology](https://openforbots.com/methodology/): What the audit checks, how findings are interpreted, how actions are prioritized, and what the audit cannot prove
- [Methodology changelog](https://openforbots.com/methodology/changelog/): Material changes affecting evidence, interpretation, and action ordering
- [Contact and corrections](https://openforbots.com/contact/): Audit support, crawler-data corrections, methodology feedback, and security routing
- [Security reporting](https://openforbots.com/security/): Human-readable responsible disclosure policy
- [Machine-readable security policy](https://openforbots.com/.well-known/security.txt): Security contact, policy, canonical location, preferred language, and expiry

## Current boundaries
OpenForBots audits public website conditions that can be checked from outside the site. It does not inspect a provider's private index, ranking logic, retrieval system, prompts, training data, or source-selection process. A successful technical check does not guarantee that a page will be indexed, retrieved, cited, ranked, recommended, used for model training, or sent traffic.