In one sentence: four conditions stand between a URL and an AI answer, they are strictly ordered, and each one invalidates everything after it — so audit and fix them in order, not in parallel.
Gate 1 — Reachable
Can each named agent fetch the page? Not "does robots.txt permit it" — request the page once per user agent and compare status codes and served bytes.
Two things break here that a robots.txt reader will never see: CDN rules and WAFs that block by user agent, and origin rules that differ from what the edge advertises. A site can have a perfectly permissive robots.txt and return 403 to every named AI agent.
The second failure at this gate is a conceptual one: a failure to fetch is not the same as a page not existing. Any tool that reports "not present" when it means "could not fetch" will send you optimising content that was fine.
Gate 2 — Readable
Is the substance in the HTML, without JavaScript? Most AI crawlers do not execute JS. On modern front-end stacks this gate fails more often than any other, and it is invisible in a browser because your browser runs the JS.
Fetch as plain HTML and read what comes back. If the page body is an empty shell with a loading state, that is the model's entire view of your page.
Gate 3 — Extractable
Can a self-contained answer be lifted out? Take a heading matching a real question, read the first two sentences under it out of context, and ask whether they are a complete correct answer that names their own subject. Then check the structural traps: content behind accordions or tabs, facts split between a table and a footnote, hedges where a competitor states a number.
Gate 4 — Corroborated
Does the claim appear anywhere you do not own? This is the only gate that cannot be fixed from your own codebase, and it is the slowest. List the claims you want repeated about you, and for each one find a source that is not your domain. The gaps are the work.
Ordering, and why it matters so much
| Gate | Cost to test | Cost to fix | Invalidates |
|---|---|---|---|
| 1 Reachable | Minutes | Hours | Everything |
| 2 Readable | Minutes | Days to weeks | Gates 3 and 4 |
| 3 Extractable | Hours | Weeks | Gate 4's effect |
| 4 Corroborated | Hours | Months | — |
The cost asymmetry is the whole argument. Gates 1 and 2 are minutes to test and cannot be skipped; gate 4 is months of work that produces nothing while gate 1 is shut.
Re-test after every deploy that touches routing or the edge
These gates close silently. A CDN rule added for bot protection, a framework upgrade that moves rendering to the client, a routing change that turns canonical URLs into redirects — none of these produce an error anybody sees. Put the four checks in whatever runs after a deploy.