Learn the foundations. Apply what fits.
Start with how website discovery is changing, then use the practical guides to investigate a specific issue, make an informed change, and check the deployed result.
A research-backed guide to how web discovery evolved from ranked links into AI-generated answers, citations, recommendations, and agent actions—and what website owners should do now.
Understand the visibility shiftPractical guides
Filter by the five areas used in the OpenForBots report. Each guide shows who it is for, how long it takes to read, and when its information was checked.
The current OpenAI-documented difference between its model-training crawler, ChatGPT Search crawler, and user-triggered ChatGPT-User agent.
Read guide →A provider-specific robots.txt pattern for allowing named AI search crawlers while restricting named model-training crawlers, with separate treatment for user-triggered fetchers and data-use controls.
Read guide →How initial HTML, server rendering, static generation, hydration, and client-side rendering affect what automated systems can retrieve from React sites.
Read guide →What the llms.txt v2 proposal can provide, what adoption evidence is missing, and why it is not a universal access, ranking, or citation requirement.
Read guide →A disciplined diagnostic for missing AI citations that separates retrieval, answer usefulness, identity, source selection, and result variability.
Read guide →A reproducible process for checking crawler purpose, whether robots.txt actually applies, deployed policy, public delivery, initial HTML, and fresh post-fix evidence.
Read guide →A current provider-sourced comparison of search discovery, model training, user-triggered retrieval, ordinary search crawling, and data-use controls.
Read guide →A practical, evidence-first checklist covering crawler purpose, public delivery, initial HTML, content clarity, structured data, identity, remediation, and re-checking.
Read guide →How answer-outcome tracking and evidence-based website audits answer different questions, where they overlap, and how to combine them without false causality.
Read guide →What crawler registration and verification mean in practice, how providers document IP ranges, reverse DNS, and cryptographic bot authentication, and why robots.txt alone is policy, not proof of identity.
Read guide →What Cloudflare's September 15, 2026 AI crawler defaults change, how Cloudflare can add rules to the robots.txt it serves, and how to check what your site now tells AI crawlers.
Read guide →No guides match these filters. Clear one or more filters to continue.