Test whether AI crawlers can actually read your site.
Check four access gates: robots.txt, real crawler identity, JavaScript shell content and meta directives.
What AI Crawlability helps the team understand.
Check four access gates: robots.txt, real crawler identity, JavaScript shell content and meta directives.
Crawler identity probe
Request the site as GPTBot and other crawler identities, with a browser control group.
JavaScript shell detection
Find pages where the initial HTML contains too little usable content.
Meta blocker check
Review page-level directives that can block indexing or snippets.
Keep every result connected to evidence and action.
The product explains the result, preserves its source context and makes the next review explicit.
The rules ai crawlability follows
A feature list is easy to copy; the rules behind the numbers are not. Each of these is enforced in the product, and most of them were added after getting it wrong once.
| Rule | Why it exists |
|---|---|
| Access is tested with the real user agent, not inferred from robots.txt. | A permissive robots.txt in front of a CDN rule or firewall that returns 403 to those agents is a block in practice, and the file will keep saying otherwise. |
| A 200 response with an empty body is reported as a failure, not a pass. | Most AI crawlers do not execute JavaScript. A page whose content appears only after client-side rendering is an empty shell to them, and nothing in your own analytics will ever tell you. |
| A browser control group runs alongside the agent probes. | Without it, a site that is simply down looks identical to a site that blocks AI agents. Removing the control group is how a checker starts producing false alarms. |
| "Could not fetch" is never reported as "not present". | A timeout on robots.txt tells you nothing about whether robots.txt exists. Collapsing the two states is how an audit tells a site owner to fix something that is already fine. |
Move from summary metrics into the underlying question.
Sample data demonstrates the evidence depth of a formal project.
Visibility evidence
Last 30 daysKeep exploring
Questions about AI Crawlability
Does this feature run from the public website?
No. Marketing pages explain the capability. Formal data collection begins only after a user enters the App and confirms the project scope.
Can snapshot results become part of this workflow?
Yes. A valid snapshot can be claimed as an initial sample, while onboarding still confirms markets, competitors and the formal monitored question set.
Are all metrics comparable across AI engines?
Core answer states can be normalized, but citations, search assistance and answer formats still need engine-specific evidence.
How often is the data refreshed?
Measurement runs on the schedule your plan allows, and monitoring quota is shared at the account level. Alerts require two consecutive runs to confirm, so model sampling noise does not become a false alarm.
Can the original AI answer be inspected?
Yes. Every result drills down to the answer that produced it, question by question and model by model, together with the citations found in that answer.
What happens when a model is unavailable?
That run is marked partial rather than being silently filled in. The remaining engines still complete, and the gap stays visible in the result.
Put AI Crawlability into a complete GEO workflow.
Create a project, confirm your market and start measuring the questions that matter.