The likely problem
Most sites now receive three different kinds of automated traffic and treat all of it as one thing. Search crawlers that feed classic rankings. AI search crawlers that feed ChatGPT, Perplexity, and Google's AI answers. And a newer, much less predictable category: general-purpose agents that were not built to index you at all.
That third category is the one that causes surprises. In the past year, researchers documented agent swarms that created thousands of accounts on a public package registry, probed its build system, and attempted to extract user API keys. A separate swarm took over an obscure wiki to coordinate. Neither was a search crawler. Neither respected the conventions you would expect from one.
The diagnosis, in most cases, is not that you are under attack. It is that you cannot currently answer a simple question: what is actually crawling me right now, and which of it deserves access? This post is a way to answer that and act on it.
Symptoms that you need this audit
You probably need to run this if any of these are true:
- Your bot-traffic share has grown but you cannot name the top five non-search agents hitting you.
- You have seen signup spikes, form submissions, or 404 storms with no matching human referral source.
- Your
robots.txthas not changed since AI crawlers became a meaningful category. - Someone on the team proposed blocking "AI bots" wholesale, and nobody could say what that would cost you in AI search visibility.
- Your CDN or WAF is alerting on traffic patterns you cannot attribute.
If two or more apply, the audit below is worth the two hours.
What usually causes the confusion
Three structural problems make agent traffic hard to reason about.
User agents are self-declared and unverified. Any client can claim to be any crawler. A GPTBot string in your logs is a claim, not a fact. Verification requires checking it against something the claimant controls, such as reverse DNS or a published IP range.
The categories overlap in purpose. An AI search crawler and a general agent may both fetch the same URLs. The first wants to cite you. The second may want to extract data, complete a task, or test whether your forms work. Same request line, very different implication.
Your tooling was built for a two-party model. Traditional analytics assumes humans plus search engines. Agent traffic often lands in "direct" or "other," which means it never reaches the reports anyone looks at.
Fix 1: Build the traffic baseline before you classify anything
Do not start by blocking. Start by measuring, so you have a before-and-after you can trust.
Pull at least 30 days of raw access logs from your origin, CDN, or WAF. You want the fields that let you group and verify: timestamp, client IP, user agent, requested path, status code, and bytes served.
Group by user agent and count requests, unique IPs, and distinct paths. Sort by request volume and by distinct-path count. Those two orderings surface different things: volume finds the noisy crawlers, distinct paths finds the ones methodically walking your site.
Expected output: a ranked table of user agents with request counts and path diversity.
Quality check: your baseline window should include at least one normal week. If you only sample a launch week or an incident window, every conclusion afterward is distorted.
Recovery path: if you cannot get raw logs, your CDN's bot report is an acceptable substitute. Note that it is already classified by the vendor, so you inherit their definitions. Treat it as a starting point, not ground truth.
Fix 2: Classify each agent into one of four buckets
Take your ranked list and sort every entry into one of these four groups. The grouping matters more than the specific names, because it drives the access decision.
Bucket | What it looks like | Access decision |
|---|---|---|
Verified search crawler | Claims a major search engine, verifies against published IP ranges and reverse DNS | Allow. No rate limiting needed. |
Verified AI search crawler | Declared AI crawler that passes the vendor's own verification method | Allow, but confirm your key pages are reachable to it |
Unverified AI or agent traffic | Claims to be a known AI crawler but fails verification, or declares an agent framework | Rate limit. Allow identifiable pages, throttle the rest. |
Unattributed automated traffic | No consistent user agent, high path diversity, hits login or form endpoints | Investigate, then block or challenge |
Two rules keep this honest. First, verify before you trust: a declaration is not evidence. Second, never let "unattributed" silently mean "unwanted." Some legitimate monitors and uptime checkers land there and will page you at 3am if you block them.
Expected output: every high-volume agent assigned to a bucket, with the evidence for the assignment.
Quality check: for anything you place in the first two buckets, confirm you have positive verification, not just a matching string.
Recovery path: if a crawler's verification method is undocumented, leave it in the unverified bucket and rate limit rather than block. You can tighten later once you know the cost.
Fix 3: Verify what your server actually serves them
Classification tells you who is asking. It does not tell you what they receive. Those diverge more often than people expect, because JavaScript-rendered content and bot rules can serve a different page than the one you think you published.
Render the critical pages the way each crawler would and compare. Check your homepage, your top ten landing pages, and any page you want cited in AI answers. You are looking for three specific failures: content that exists only after client-side rendering, critical text hidden behind an interaction, and pages returning a challenge or error to non-browser clients.
To do this without guessing, replay the request as the specific crawler. Auspia's AI search crawler simulator lets you fetch a public URL as a named AI crawler and see the final rendered result, which turns "I think we are readable" into an observation.
Expected output: a short list of pages where the served version differs from the intended version.
Quality check: test as at least one verified search crawler and one AI crawler. If the two see different content, you have found a real problem.
Recovery path: if a page fails only for AI crawlers, check your bot rules before touching the template. Blocking rules added for a different purpose are a common cause.
Fix 4: Write access rules you can defend
Now you can set policy, because you know what you are allowing and denying.
State your intent in robots.txt for the crawlers you want to address by name, and set rate limits at the edge for everything else. Keep the two mechanisms separate: robots.txt expresses preference to cooperative crawlers, and rate limiting enforces behavior on the ones that ignore preference. Neither substitutes for the other.
A workable default for most content sites:
- Allow verified search and AI search crawlers on public content.
- Rate limit unverified AI and agent traffic to a level that cannot degrade origin performance.
- Challenge or block unattributed traffic that hits authentication, checkout, or form endpoints.
- Keep a named exception list for monitoring you rely on.
Expected output: an updated robots.txt plus at least one edge rule, both recorded in version control with a date and a reason.
Quality check: for every rule, you can state what breaks if it is wrong. A rule with no answer is a rule you cannot defend in a review.
Recovery path: ship rules in report-only or log mode first where your platform supports it. Watch what would have been blocked for a week before enforcing.
How to measure whether it worked
Pick two numbers and track them for 30 days: the share of automated traffic you can positively attribute, and the count of verified AI crawler requests reaching your key pages.
The first should rise. The second should not fall. If attribution improves while AI crawler access drops, you blocked something you wanted, and the fix is in your rules rather than your intent.
Do not use raw bot-request volume as a success metric. Lower bot traffic is not the goal. Correct bot traffic is.
When not to use this approach
Skip the full audit if you run a small site behind a managed platform that already handles bot classification, and you have no signs of abusive traffic. The cost of the baseline is not worth it.
Be cautious with aggressive blocking if you depend on AI search visibility. Blocking broadly to reduce noise is the fastest way to disappear from AI answers, and the loss is quiet. You will not get an error. You will just stop being cited.
FAQ
Is a user agent string enough to identify a crawler? No. It is a self-declared claim. Verify against published IP ranges or reverse DNS before you grant access.
Should I block AI crawlers to protect my content? Decide per crawler based on what you want from AI search. Blanket blocking reduces noise and citation surface at the same time.
What if I cannot tell whether traffic is an agent or a person? Look at path diversity and endpoint targeting. Sequential, methodical walking and repeated hits on forms or auth endpoints are strong automated signals.
How often should I rerun this? Quarterly, and after any bot-traffic incident. The crawler landscape changes faster than an annual review cycle.
Author: Julian Mercer, 14-Year Technical SEO Practitioner at Auspia. Julian writes about crawlability, rendering, schema, and the technical foundations that make content readable to both search engines and AI systems.
