THE USEFUL ANSWER
Audit AI-crawler traffic in four layers: verify the caller when a provider publishes a verification method, record the requested path and response, separate crawler purpose and access policy, then compare crawl observations with independent indexing, citation and referral evidence.
A successful 2xx response proves access to that response at that time. It does not prove the page entered an index, appeared in an answer or sent a visitor.
WHAT YOU'LL LEAVE WITH
A log-review sheet that preserves the difference between access, indexing, citation and referral instead of turning crawler counts into visibility claims.
- User-agent text alone is weak identity evidence when a provider offers IP or DNS verification.
- Path, status and response content explain access; they do not establish downstream use.
- Keep crawl, index, citation and referral measures in separate columns.
01A request is the first event, not the outcome
Cloudflare's AI Crawl Control documentation separates total requests, allowed requests, status codes, content formats and referrals. Its dashboard can group activity by crawler, operator, host and path. Those fields help explain what reached the edge and what the edge returned. They do not say that a model stored, indexed or cited the content. [1]
Swipe or scroll to compare
| Layer | Evidence to keep | Do not infer |
|---|---|---|
| Identity | Verified operator, IP or signed-agent evidence | That a claimed user agent is genuine |
| Access | Timestamp, path, status, bytes and content type | That the complete page was understood |
| Discovery | Separate index or retrieval observation | That every successful crawl entered an index |
| Answer | Saved prompt, engine, date and cited URL | That crawling caused a citation |
| Referral | First-party session or provider referral record | That crawler requests are human visits |
02Verify identity where a method exists
Google documents reverse and forward DNS verification for one-off crawler checks and published IP ranges for automated verification. The process confirms that a request claiming to be from Google resolves through the expected domain and back to the original IP. [2] Other operators may use published IP lists, signed-agent schemes or no equivalent public method. Record the verification method and keep unsupported identities unresolved.
03Read the response, not only the count
- Keep the canonical requested path and any redirect destination.
- Separate 2xx, 3xx, 4xx and 5xx responses.
- Record the served content type and whether the response was complete.
- Mark robots and access-policy changes with their effective time.
- Exclude monitoring, security scanning and user-triggered fetchers when their purpose differs.
- Preserve raw timestamps and timezone before producing daily totals.
A spike can reflect a new crawler, a redirect loop, repeated errors, a policy change or a small set of frequently fetched pages. Compare the distribution before changing access. Blocking a crawler can reduce requests while also removing a discovery path; allowing it can raise infrastructure cost without producing a citation or visitor.
04Use one decision rule
Change access only when the site owner can name the intended use, identify the operator with adequate confidence, and accept the cost and policy implications. Measure the next period with the same grouping. This is a proposed audit method; no crawler-log export, indexing experiment or referral study was performed for this article.
FOLLOW THE EVIDENCE
Sources & checks
- Cloudflare: Analyze AI traffic ↗
AI crawler request, status, content-format, operator, path and referral dimensions and their availability limits.
CHECKED 2026-11-10 - Google: Verify requests from Google crawlers and fetchers ↗
Reverse and forward DNS verification and published-IP methods for Google-controlled requests.
CHECKED 2026-11-10
Sources stay attached so you can check the argument. Meaningful revisions keep this address.