How to detect AI crawlers and LLM referral traffic

AI systems touch your site in two ways, and standard analytics misses most of both. I built a detection SDK at ZeroClick that classified this traffic at the edge and fed an analytics pipeline. Several of the sharpest lessons came from things that broke and got fixed. This article is the version I wish I’d had on day one: where the traffic is observable, which signals to trust in what order, and the instrumentation rules that are cheap to follow up front and expensive to retrofit.

Crawls and referrals are opposite events

Two categories get lumped together as “AI traffic.” In the first, a machine fetches your page — to train on it, to index it, or to answer a question someone just asked. In the second, a human clicks a citation link inside an AI product and lands on your site. Same URL, opposite implications. A crawl means your content is being consumed without a visit. A referral means an AI product is sending you people. Everything downstream — content strategy, access policy, whether to celebrate or worry — depends on keeping the two apart.

One naming trap from experience: my SDK first labeled the machine case origin. Bad choice. In a library that lives at the edge and reads headers, “origin” collides with the HTTP Origin header and with CDN-origin terminology, and the ambiguity caused enough confusion that I renamed it to crawl. Pick unambiguous names for the taxonomy before it reaches a dashboard.

Where you can observe it

This is the part most teams get wrong. Client-side analytics tags require JavaScript execution, and crawlers do not run your JavaScript — so a JS tag can only ever see the referral half. At the other end, CDN caching means a crawler’s fetch may be served from cache and never reach your origin server, so origin request logs miss traffic too. The reliable observation point is the layer every request passes through: edge middleware or the CDN itself. My SDK started with a client-side script-tag install path. We abandoned it once the numbers made the blind spot obvious.

Running at the edge imposes constraints worth knowing before you build: worker-style runtimes want zero dependencies, Web Crypto (crypto.subtle) rather than Node’s crypto module, and code that tolerates being cancelled mid-request.

The signals, in precedence order

1. Campaign parameters on citation links.Some providers append a source parameter to the links they cite. Match these exactly — never by substring — so a hostile value like a lookalike domain can’t pass.
2. The Referer header.Present when a human clicks through from an AI product’s web interface. Substring matching on the header is spoofable — chatgpt.com as a substring also matches a hostile domain that merely contains it — so parse the URL and compare the host.
3. User-agent patterns. The only signal crawls give you. Self-declared and copyable, but the baseline every provider supports.

The ordering reflects a tradeoff I learned the fast way. The first version of my SDK relied on campaign parameters alone. It needed Referer support within a day. Parameter tagging is controlled by the AI provider, and a provider can change or drop it unilaterally — your attribution goes dark without warning. Referer is browser-controlled and harder to switch off centrally, but it gets stripped by referrer policies and in-app browsers. Use both, in that order, and record which one fired.

Rebuild the URL before you read it

At the edge you usually sit behind a proxy, and the request URL your code sees carries the proxy’s host and protocol rather than the real ones. Rebuild the URL from the forwarded host and protocol headers, and handle comma-separated values — chained proxies append, so the header may read https,http. This matters more than it looks: the query string carries your campaign parameters, and if URL reconstruction is wrong, referral attribution doesn’t error. It silently reads as zero.

Signature lists decay

Whatever user-agent list you start from is a snapshot. Provider agent names change, new agents appear, and some published tokens are robots.txt controls rather than user agents at all — matching those as user-agents classifies nothing. Two design consequences. First, keep the pattern list updatable separately from the code; my SDK compiled the signatures into the library, which turned every pattern update into a release and a customer upgrade. Second, verify current agent names against each provider’s own documentation rather than trusting a blog snapshot — including this one. What stays stable is the category structure: training crawlers, search indexers, and user-triggered fetches that retrieve a page because a person asked about it. Classify into those buckets and let the token list churn underneath.

Two rules that are cheap now and expensive later

• Emit an event for non-AI traffic too.A detection library’s instinct is to report positives. Mine did, until someone asked what percentage of traffic was AI and the answer was uncomputable — no denominator. Send an explicit null classification for ordinary pageviews from day one.
• Store the raw user-agent alongside your verdict. When your patterns improve — and they will — the raw string is the only thing that lets you re-classify history instead of starting your dataset over.

Most of the work is filtering, not detecting

In my SDK, the filtering module grew to roughly six times the size of the detection module. Every page a human loads triggers a swarm of requests that are not pageviews: static assets, hashed bundle files, framework prefetches, API calls. Left unfiltered, they drown the signal — one human landing can produce many prefetch requests that each look like a separate referral, inflating your counts by the prefetch factor. You need layered filters for asset extensions, bundler hash patterns, framework internals, API paths, and prefetch headers, plus deduplication for the prefetch case.

Two cautions. Don’t let filtering run ahead of detection in a way that collapses “not worth recording” into “not AI” — those are different verdicts and you’ll want to tell them apart when the numbers look wrong. And don’t blanket-filter non-HTML paths, because that discards the exact files agents ask for: llms.txt, robots.txt, sitemaps, and markdown docs are AI traffic, not noise.

Verification, and what pattern matching can’t do

Everything above is deterministic string matching, and a determined spoofer can defeat it. The next tier is verification: several providers publish the IP ranges their crawlers use, and some support reverse-DNS confirmation. That work belongs on the server, not in a client-side library — checking an IP against published ranges, holding reputation across requests, and scoring confidence all need state and context a stateless edge function doesn’t have. My SDK stayed deterministic by design and left scoring to the pipeline behind it. One more boundary to know: instrumentation that only watches GET requests misses agents that POST to your endpoints, which matters more each year as agents act rather than just read.

The summary I’d give a past version of myself: observe at the edge, classify crawl versus referral with signals in precedence order, keep the denominator and the raw evidence, spend real design effort on filtering, and treat every signature list — including this article’s framing — as something to verify against provider documentation before you rely on it.