How to Check AI Crawler Access

To check AI crawler access, identify the exact crawler token and purpose, resolve the deployed robots.txt rule for the exact path, and then inspect what the public URL actually returns. Keep those observations separate: policy does not prove delivery, and a successful diagnostic request does not prove that a provider indexed or used the page.

The evidence chain

A useful access review follows this sequence:

provider documentation
→ exact crawler token and purpose
→ deployed robots.txt policy
→ path-specific rule resolution
→ public response delivery
→ initial content evidence
→ bounded conclusion
→ remediation
→ fresh re-check

Skipping a step creates common mistakes. For example, reading only robots.txt misses a CDN challenge, while sending only a custom user-agent request misses the site’s declared policy.

1. Identify the crawler and its purpose

Do not start with “AI bots” as one category. Provider documentation may distinguish:

• search discovery crawlers;
• model-development or training crawlers;
• user-triggered retrieval agents;
• ordinary search crawlers;
• data-use controls that are not separate HTTP crawler identities.

Choose the token connected to the job you are investigating. Blocking a training crawler can be an intentional policy and should not automatically be reported as a search-visibility defect.

Use the OpenForBots crawler registry to find the current token, provider source, purpose, verification method, and known limitations.

2. Fetch the deployed robots.txt file

The standard location is the root of the relevant origin:

https://example.com/robots.txt

Check the actual scheme, hostname, and port. These are separate scopes. The following can publish different policies:

• https://example.com/robots.txt
• https://www.example.com/robots.txt
• https://docs.example.com/robots.txt
• http://example.com/robots.txt

Record:

• final URL after redirects;
• HTTP status;
• content type;
• response body used for interpretation;
• retrieval time;
• any timeout, size, or security limit.

A missing file and an unavailable file are not the same finding. Preserve the difference.

3. Resolve the rule for the exact token and path

A useful result is not merely “the token appears in the file.” Resolve the effective rule for a concrete path such as:

/
/docs/getting-started/
/pricing/
/research/report-2026/

Review:

• exact product-token matching;
• wildcard groups;
• duplicate groups;
• Allow and Disallow precedence;
• longest matching path;
• percent encoding and path case where relevant;
• comments that may describe intent but do not change parsing.

RFC 9309 defines the Robots Exclusion Protocol and makes clear that the file is not access authorization. Confidential pages require authentication and authorization regardless of crawler rules.

4. Inspect public response delivery

A crawler can be allowed by policy and still receive an unusable response.

Check the selected public page for:

• successful HTTP status;
• redirect chain and final destination;
• response content type;
• CDN, bot-management, or JavaScript challenge;
• rate-limit or access-denied response;
• timeout;
• unusually large response;
• canonical URL;
• meaningful initial HTML.

A controlled diagnostic request can compare a normal request with a crawler-like user-agent. Label the result as simulated. A user-agent string is easy to spoof and does not authenticate the caller.

The homepage alone will not surface a delivery problem on the pages that actually matter, such as a pricing page or a key blog post. The OpenForBots audit accepts up to four extra same-site paths and checks each one under a default request and a simulated automated-client request, reporting redirects, denials, rate limits, and profile differences without asserting a cause. See delivery diagnostics for selected paths. A problem outcome — denied, rate limited, a server error — comes with a short, provider-agnostic guidance note pointing at the category of configuration to check (bot-management, WAF, rate limiting), so a flagged path isn’t a dead end. See suggested fixes and remediation guidance.

5. Inspect the initial HTML

Successful delivery is not the end of the test. Determine whether the initial response contains useful public information:

• a descriptive title;
• one clear H1;
• visible explanatory text;
• canonical links and navigation;
• the primary entity and offer;
• relevant structured data that matches visible content;
• links to the page’s sources or supporting detail.

A client-rendered application may become complete after JavaScript runs, but weak initial HTML remains a portability and diagnostic risk. Report what was observed rather than claiming that every provider behaves identically.

6. Separate findings from conclusions

A high-quality report keeps the raw observation and its interpretation distinct.

Layer	Example
Documented	OpenAI describes OAI-SearchBot as relevant to ChatGPT search discovery.
Observed	The deployed file contains User-agent: OAI-SearchBot and Disallow: /docs/.
Parsed	The most specific matching rule disallows /docs/getting-started/.
Simulated	A bounded request using the documented token received HTTP 403.
Conclusion	The tested path has both a policy restriction and a delivery restriction that merit review.
Limitation	This does not prove whether OpenAI attempted to crawl or index the page.

This structure is more trustworthy than one unexplained “blocked” badge.

7. Apply minimal remediation

Change only what the evidence supports.

Possible actions include:

• correct a misspelled token;
• remove an unintended broad disallow;
• add an explicit path rule matching the owner’s declared policy;
• allow the selected path through a CDN or WAF challenge;
• restore a successful public response;
• add meaningful server-rendered or statically rendered content;
• align canonical URLs and redirects;
• keep private content behind real access controls.

Do not open training access merely to improve a generic score. Search, training, and user-triggered retrieval are separate governance choices.

8. Perform a fresh re-check

A code change, ticket status, or user claim is not verified resolution.

Repeat the same checks after deployment:

1. retrieve the current robots.txt;
2. resolve the same token and path;
3. retrieve the same public URL;
4. record status, redirects, headers, and initial content;
5. compare the new evidence with the earlier record;
6. state whether the specific condition changed;
7. retain all external-outcome limitations.

Only fresh compatible evidence should move a technical condition from unresolved to resolved.

A compact manual checklist

• Confirm the exact crawler token in current provider documentation.
• Classify its purpose.
• Fetch robots.txt from the correct origin.
• Resolve the rule for the exact path.
• Inspect the real public response.
• Record redirects, status, headers, and content type.
• Inspect initial HTML and canonical destination.
• Label simulated requests as simulated.
• Record unknowns and unavailable checks.
• Apply the smallest evidence-supported fix.
• Re-run the same checks after deployment.

What the check cannot prove

Even a fully allowed and successfully delivered page does not prove:

• that a provider fetched it;
• that it entered a private index;
• that it influenced model training;
• that it will be selected for a prompt;
• that it will be visibly cited;
• that traffic or revenue will increase.

The correct conclusion may simply be: no material public access barrier was observed under the tested conditions.

Run the evidence-based check

Use the OpenForBots public audit to inspect crawler policy and public delivery with evidence labels and explicit limitations. Then continue with Search Crawlers vs Training Crawlers before choosing a policy.