How to Check AI Crawler Access

To check AI crawler access, start with the exact provider token and ask one question before parsing any rule: does the provider actually document robots.txt as an applicable control for this token? Only then should you resolve the deployed rule, inspect what the public URL returns, and draw a bounded conclusion.

That extra step matters because current provider documentation is not uniform for user-triggered retrieval agents.

The evidence chain

A useful access review follows this sequence:

provider documentation
→ exact token and purpose
→ does robots.txt apply to this token?
→ deployed robots.txt policy where applicable
→ path-specific rule resolution
→ public response delivery
→ initial content evidence
→ bounded conclusion
→ remediation
→ fresh re-check

Skipping a step creates common errors. Reading only robots.txt can miss a WAF challenge. Sending only a custom user-agent request misses declared policy. Parsing a user-triggered token without checking provider semantics can produce a confident-looking verdict that the provider itself does not support.

1. Identify the token and purpose

Do not start with “AI bots” as one category. Provider documentation may distinguish:

• search-discovery crawlers;
• model-development or training crawlers;
• user-triggered retrieval agents;
• ordinary search crawlers;
• data-use controls that are not separate HTTP crawler identities.

Use the OpenForBots crawler registry to identify the current token, purpose, primary source, verification date, and known limitations.

2. Confirm whether robots.txt applies

For automatic crawlers such as OAI-SearchBot, GPTBot, Claude-SearchBot, ClaudeBot, and PerplexityBot, the current provider documentation supplies a robots.txt control surface.

User-triggered agents require more care:

• ChatGPT-User: OpenAI says robots.txt rules may not apply because requests are initiated by a user.
• Claude-User: Anthropic says its named bots honor robots.txt and describes disabling Claude-User as preventing retrieval in response to a user query.
• Perplexity-User: Perplexity says the fetcher generally ignores robots.txt because the user requested the retrieval.

If a provider says robots.txt may not apply or is generally ignored, OpenForBots can still show the site’s matching directive as declared policy evidence, but that must not be converted into a guaranteed provider-level “allowed” or “blocked” verdict.

3. Fetch the deployed robots.txt

The standard location is the root of the relevant origin:

https://example.com/robots.txt

Scheme, hostname, and port define the scope. example.com, www.example.com, and docs.example.com can publish different files.

Record:

• final URL after redirects;
• HTTP status;
• content type;
• response body used for interpretation;
• retrieval time;
• any timeout, size, or security limit.

A missing file and an unavailable file are different observations. Preserve the difference.

4. Resolve the rule where robots.txt is applicable

A useful result is not merely “the token appears in the file.” Resolve the effective rule for a concrete path such as /, /docs/getting-started/, or /pricing/.

Review exact product-token matching, wildcard groups, duplicate groups, Allow/Disallow precedence, and the longest matching path.

For a token such as ChatGPT-User or Perplexity-User, a parser can still report which directive syntactically matches the token. The report must then carry the provider caveat rather than implying enforcement that the source does not support.

5. Inspect public response delivery

A crawler can be allowed by policy and still receive an unusable response. Check the selected public page for:

• successful HTTP status;
• redirect chain and final destination;
• response content type;
• CDN, bot-management, or JavaScript challenge;
• 403, 429, server error, or timeout;
• meaningful initial HTML.

A controlled request using a crawler-like user-agent is simulated evidence. A user-agent string is easy to spoof and does not authenticate the caller.

The OpenForBots audit can also check selected same-site paths under bounded request profiles. Those diagnostics report what OpenForBots observed; they do not prove how a provider’s private production crawler behaved.

6. Inspect the initial HTML

Successful delivery is not the end of the test. Determine whether the initial response exposes useful public information such as a descriptive title, clear H1, visible explanation, canonical links, navigation, primary entity, and relevant structured data that matches visible content.

A client-rendered application may become more complete after JavaScript runs, but weak initial HTML remains a real portability and diagnostic risk. Report the observed response rather than assuming every provider renders identically.

7. Separate observations from conclusions

Layer	Example
Documented	OpenAI describes OAI-SearchBot as the Search crawler and says robots.txt controls it.
Observed	The deployed file contains User-agent: OAI-SearchBot and Disallow: /docs/.
Parsed	The most specific matching rule disallows /docs/getting-started/.
Simulated	A bounded request using an automated-client profile received HTTP 403.
Conclusion	The tested path has both a policy restriction and a delivery restriction that merit review.
Limitation	This does not prove whether OpenAI attempted to crawl, index, select, or cite the page.

For ChatGPT-User or Perplexity-User, the conclusion must additionally preserve the provider’s documented robots caveat.

8. Apply minimal remediation

Change only what the evidence supports. Possible actions include correcting a token, removing an unintended broad disallow, adding a path rule where the provider documents robots.txt as applicable, fixing a WAF challenge, restoring a successful response, improving initial HTML, or protecting private content with real access controls.

Do not open training access merely to improve a generic score, and do not promise that a ChatGPT-User or Perplexity-User robots change will control a product path when the provider documentation says otherwise.

9. Perform a fresh re-check

A deployment or ticket closure is not verified resolution. Repeat the same checks after the change:

1. re-check the current provider documentation when the token is time-sensitive;
2. retrieve the current robots.txt;
3. resolve the same token and path where applicable;
4. retrieve the same public URL;
5. compare the new evidence with the earlier record;
6. state whether the specific public condition changed;
7. keep provider-outcome limitations intact.

Only fresh compatible evidence should move a technical condition from unresolved to resolved.

Compact checklist

• Confirm the exact token in current provider documentation.
• Classify its purpose.
• Confirm whether robots.txt applies to that token.
• Fetch robots.txt from the correct origin.
• Resolve the rule for the exact path where appropriate.
• Inspect the real public response.
• Label simulated requests as simulated.
• Inspect initial HTML and canonical destination.
• Preserve provider-specific caveats and unknowns.
• Apply the smallest evidence-supported fix.
• Re-run the same checks after deployment.

What the check cannot prove

Even a fully allowed and successfully delivered page does not prove that a provider fetched it, indexed it, used it for training, selected it for a prompt, cited it, or will send traffic or revenue.

The correct conclusion may simply be: no material public access barrier was observed under the tested conditions, for the controls that provider documents as applicable.

Run the evidence-based check

Use the OpenForBots public audit to inspect crawler policy and public delivery with evidence labels and explicit limitations. Then continue with Search Crawlers vs Training Crawlers before choosing a policy.