Learn the foundations. Apply what fits.
Start with how website discovery is changing, then use the practical guides to investigate a specific issue, make an informed change, and check the deployed result.
A research-backed guide to how web discovery evolved from ranked links into AI-generated answers, citations, recommendations, and agent actions—and what website owners should do now.
Understand the visibility shiftPractical guides
Filter by the five areas used in the OpenForBots report. Each guide shows who it is for, how long it takes to read, and when its information was checked.
The provider-documented difference between OpenAI's model-development crawler and ChatGPT search crawler, with separate robots.txt examples and verification steps.
Read guide →A provider-specific robots.txt pattern for allowing AI search discovery while restricting named model-training crawlers and data-use controls.
Read guide →How initial HTML, server rendering, static generation, hydration, and client-side rendering affect what automated systems can retrieve from React sites.
Read guide →What the llms.txt proposal can provide, what adoption evidence is missing, and why it is not a universal access, ranking, or citation requirement.
Read guide →A disciplined diagnostic for missing AI citations that separates retrieval, answer usefulness, identity, source selection, and result variability.
Read guide →A reproducible process for checking crawler purpose, robots.txt policy, public delivery, initial HTML, and fresh post-fix evidence.
Read guide →A provider-sourced comparison of search discovery, model training, user-triggered retrieval, ordinary search crawling, and data-use controls.
Read guide →A practical, evidence-first checklist covering crawler purpose, public delivery, initial HTML, content clarity, structured data, identity, remediation, and re-checking.
Read guide →How answer-outcome tracking and evidence-based website audits answer different questions, where they overlap, and how to combine them without false causality.
Read guide →What crawler registration and verification mean in practice, how providers document IP ranges, reverse DNS, and cryptographic bot authentication, and why robots.txt alone is policy, not proof of identity.
Read guide →No guides match these filters. Clear one or more filters to continue.