Decision summary
Understand what completed, what matters, and the first useful action.
AI crawler & discovery readiness audit
See what AI-related crawlers can access — and what to fix first
OpenForBots checks your website's public crawler rules, content, structured data, and identity signals. You get evidence-backed findings, a clear next action, and a way to re-check after you make a change.
Inspect how deployed rules treat recognised crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot without collapsing their different purposes into one verdict.
- Find access or public-delivery problems
- See missing or unclear machine-readable signals
- Fix the most useful issue first, then verify it
See the output before you run it
From website check to a clear next decision
Preview the same evidence-first report structure used by the audit, including a disclosed versioned sample rather than an opaque marketing mockup.
Versioned sample reportopenforbots.com
Audit completeWhat should happen next?
Clarify the homepage positioning
Make the product or organization, intended audience and practical outcome explicit in the title, H1 and opening copy.
FindFixVerify
- Priority actions
- 1
- Evidence model
- Inspectable
- Overall score
- None
- Verification
- Fresh re-check
Fix path
Follow ordered improvements with dependencies and practical remediation where supported.
Fresh verification
Re-run the public evidence check and compare whether the observed condition actually changed.
Audit → fix → verify
The report is designed to move you forward
OpenForBots keeps the evidence available, but the first question is always what you should do next.
- 01
Check
Inspect public crawler rules, delivery, content, structured data, and identity signals.
- 02
Decide
See which observed issue matters first instead of decoding a composite score.
- 03
Fix
Use the recommended change or generated remediation where the evidence supports one.
- 04
Verify
Run a fresh check and confirm the public evidence changed before calling the issue resolved.
Five dimensions
What OpenForBots checks
Five clear dimensions keep different kinds of website evidence understandable without collapsing them into one score.
Access
What does the deployed crawler policy allow, and can the requested public resources be retrieved?
- Crawler policy
- Public retrieval
- Response conditions
Content
Does the initial response clearly explain the offer, audience, and practical outcome?
- Initial HTML
- Business clarity
- Page framing
Structure
Are public declarations organised for reliable machine interpretation?
- JSON-LD
- Entity relationships
- Optional llms.txt
Identity
Are brand, organisation, authorship, and trust signals coherent and reviewable?
- Public entities
- Stable identifiers
- Trust relationships
Verification
Does each action state what a later public audit must confirm?
- Recheck conditions
- Dependencies
- Known limits
Free focused tools
Check or generate one signal without running the full audit
Use a focused checker when you already know the question, or jump to a generator when the missing signal is clear.
AccessAI crawler robots.txt checker
See how deployed robots.txt rules treat recognised AI crawler tokens.
Discovery filellms.txt checkerInspect whether an llms.txt file is available and what it exposes.
StructureJSON-LD generatorCreate machine-readable organisation or website identity data.
Remediationrobots.txt generatorBuild clearer crawler directives after you decide the access policy you want.
Explore by goal
Choose the path that matches what you need to do
Use a focused diagnostic, build the underlying mental model, or follow the end-to-end readiness checklist.
Check AI crawler rules
Inspect deployed robots.txt policy for documented AI crawler tokens before changing access rules.
Understand AI discovery
Start with the concepts behind discovery, retrieval, citations, readiness, and their practical limits.
Work through the checklist
Review access, content, structure, identity, remediation, and verification as one workflow.
Need the technical detail?
The methodology is public and reviewable
Every conclusion remains connected to the evidence and limits behind it. OpenForBots keeps severity separate from evidence confidence. Public checks cannot prove provider indexing, citation, ranking, recommendation, or model use.
- Report version
- ofb-audit-r2.1
- Information checked
- Report storage
- Anonymous audit snapshots retained for up to 30 days
- Methodology
- Published and reviewable
- Corrections
- Public feedback route
We audit ourselves — and publish the result
The Citation Lab tracks whether AI answer engines mention OpenForBots
Technical readiness does not guarantee that an AI answer engine will discover, mention, or cite a source. The Citation Lab keeps a versioned prompt library and reports every completed observation, including negative and untested results.
- Prompts tracked
- 24
- Completed observations
- 0 of 24
- Method
- Manual, one prompt and engine at a time
- Guarantee
- None — observed outcomes only
Common questions
Know the boundaries before you act
Is the OpenForBots audit live?
Yes. OpenForBots retrieves public website signals including robots.txt, the homepage response, initial HTML, JSON-LD, llms.txt, business clarity, and identity or trust signals. Completed anonymous audit snapshots are retained for up to 30 days under the published privacy policy.
Does blocking GPTBot block ChatGPT Search?
Not by itself. OpenAI documents GPTBot and OAI-SearchBot as separate controls with different purposes. OpenForBots evaluates recognised crawler tokens individually instead of treating every OpenAI-related rule as the same decision.
Is llms.txt required for AI search visibility?
No universal requirement has been established. OpenForBots inspects the file when it exists, but its absence is not treated as a critical failure or a guaranteed visibility problem.
Can OpenForBots guarantee AI citations or rankings?
No. OpenForBots evaluates observable public conditions. It cannot see a provider’s complete private index, ranking logic, retrieval process, or recommendation system.
Does the audit execute JavaScript?
The audit focuses on the initial public response and does not run an unrestricted browser session. A heavy client-rendering dependency may be reported as a possible concern, not as proof that a particular provider cannot render the page.
What does business clarity mean?
It checks whether the public page clearly exposes the organisation or product, category, intended audience, primary outcome, title, heading, and opening explanation. It does not claim to observe what a private model understands internally.