How to tell humans, AI crawlers, and AI agents apart in your traffic

Your analytics say AI traffic barely exists. Meanwhile GPTBot and ClaudeBot fetch your content every day, and a growing share of your visitors arrive from answers those fetches fed. Both readings come from the same site. The gap is instrumentation: AI systems touch your site in two ways, and standard analytics misses most of both.

I built a detection SDK at ZeroClick that classified this traffic at the edge and fed an analytics pipeline, and several of the sharpest lessons came from things that broke. This article is the version I wish I’d had on day one: where the traffic is observable, which signals to trust in what order, and the instrumentation rules that are cheap up front and expensive to retrofit.

Crawls and referrals are opposite events

Two categories get lumped together as “AI traffic,” and they mean opposite things. In the first, a machine fetches your page to train on it, to index it, or to answer a question someone just asked. In the second, a human clicks a citation link inside an AI product and lands on your site. A crawl means your content is being consumed without a visit, and a referral means an AI product is sending you people. Everything downstream depends on keeping the two apart: content strategy, access policy, whether to celebrate or worry.

A third category is growing fast enough to need its own bucket. Agentic browsers such as ChatGPT’s agent mode, Perplexity’s Comet, and Claude in Chrome drive a real browser on a user’s behalf. They execute JavaScript and fire analytics tags, so they look like visitors while a machine does the clicking. Detect them separately: by user agent where the product declares one, by behavior where it doesn’t.

Quick aside on naming: my SDK first labeled the machine case origin, and that was a mistake. In a library that lives at the edge and reads headers, the word collides with the HTTP Origin header and with CDN-origin terminology. I renamed it to crawl the same day, before the first release. Pick unambiguous names before your taxonomy reaches a dashboard.

Where you can observe it

Sites instrumented with only a JavaScript analytics tag report near-zero AI crawl traffic while their content is being fetched constantly. The tag requires JavaScript execution, and HTTP fetchers like GPTBot and ClaudeBot never run your JavaScript. A client-side tag sees referrals and agentic browsers and misses fetchers entirely.

Origin logs have their own blind spot. CDN caching means a crawler’s fetch may be served from cache and never reach your origin server at all. So where can you watch from? The one reliable observation point is the layer every request passes through: edge middleware or the CDN itself.

The product my SDK fed briefly offered a script-tag install path during onboarding. The SDK itself was edge-side from its first spec, because a script tag can’t see fetchers. If your instrumentation runs in the page, the machine half of your AI traffic doesn’t exist in your data.

Running at the edge imposes constraints worth knowing before you build. Worker-style runtimes want zero dependencies, Web Crypto (crypto.subtle) rather than Node’s crypto module, and code that tolerates being cancelled mid-request.

The signals, in precedence order

On day one, my SDK classified referrals by campaign parameters alone, and referral counts from every AI product except ChatGPT read as zero. The other products appended no utm_source to their links. Referer support went in the next day. The ordering below is what that taught me.

1. Campaign parameters on citation links come first. Some providers append a source parameter to the links they cite. Match these by exact equality, never by substring, so a lookalike value can’t pass.
2. The Referer header comes second. It shows up when a human clicks through from an AI product’s web interface. Parse it as a URL and compare the host, because a substring check accepts any hostile domain that merely contains the string.
3. User-agent patterns come last. They are the only signal crawls give you. The string is self-declared and copyable, but it’s the baseline every provider supports.

Classify with campaign parameters first, then Referer, then user-agent, and record which signal fired. Both referral signals are provider-controlled. A provider can drop its campaign parameters unilaterally, and one Referrer-Policy header on its domain silences the Referer for every click. They fail independently, though, so use both.

The Referer rule is one my own code broke. My SDK substring-matched the raw header against values like chatgpt.com, so a referer of https://chatgpt.com.evil.example/would pass the check and get classified as an OpenAI referral. The exact-match discipline held for campaign parameters and never made it to the Referer path. Here’s the shape both checks should take:

type ReferralMatch = { provider: string; signal: 'utm' | 'referer' };

function classifyReferral(
  url: URL,
  referer: string | null,
): ReferralMatch | null {
  // Campaign parameters: exact equality only. A substring
  // check lets a lookalike value pass.
  const utmSource = url.searchParams.get('utm_source')?.toLowerCase();
  if (utmSource === 'chatgpt.com') {
    return { provider: 'openai', signal: 'utm' };
  }

  // Referer: parse and compare the host. Substring matching
  // accepts https://chatgpt.com.evil.example/ as OpenAI.
  if (referer) {
    let host: string;
    try {
      host = new URL(referer).hostname;
    } catch {
      return null;
    }
    if (host === 'chatgpt.com' || host.endsWith('.chatgpt.com')) {
      return { provider: 'openai', signal: 'referer' };
    }
  }

  return null;
}

Rebuild the URL before you read it

At the edge you usually sit behind a proxy, and the request URL your code sees carries the proxy’s internal host and protocol rather than the real ones. Mine broke behind a Shopify storefront’s proxy. The query string carries your campaign parameters, so a misread URL means the source parameter never matches. Wrong URL reconstruction doesn’t produce an error, only a referral count that reads as zero.

Rebuild the URL from the forwarded host and protocol headers, and handle comma-separated values, because chained proxies append and the header can read https,http. My own SDK handled the comma case in only one of its two runtime paths. That’s the kind of gap an audit finds and error logs never will.

// req.url behind a proxy carries the proxy's internal host and
// protocol. Rebuild from the forwarded headers before reading
// campaign parameters off the query string.
function requestUrl(req: Request): URL {
  const url = new URL(req.url);
  const host = req.headers.get('x-forwarded-host');
  if (!host) return url;

  // Chained proxies append, so the header can read "https,http".
  const proto =
    req.headers.get('x-forwarded-proto')?.split(',')[0].trim() || 'https';

  return new URL(url.pathname + url.search, `${proto}://${host}`);
}

Signature lists decay

Google AI crawl counts can read as zero while Google’s fetches hit your site all day. My SDK shipped this bug. Its signature list matched Google-Extended as a user agent, and Google-Extended is a robots.txt control token with no user agent of its own. That signature could never match a real request.

Whatever user-agent list you start from is a snapshot. Provider agent names change, new agents appear, and published token lists mix real user-agent strings with robots.txt-only controls. Two design consequences follow.

• Keep the pattern list updatable separately from the code. My SDK compiled the signatures into the library, which turned every pattern update into a release and a customer upgrade.
• Verify current agent names against each provider’s own documentation rather than a blog snapshot, including this one.

What stays stable is the category structure: training crawlers, search indexers, and user-triggered fetches. The last kind retrieves a page because a person just asked about it. Classify into those buckets and let the token list churn underneath.

Two rules that are cheap now and expensive later

Emit an event for non-AI traffic too, and store the raw user-agent with every verdict. Both rules cost a few lines on day one. Skipping either costs you the dataset later.

• Send an explicit null classification for ordinary pageviews. My SDK left the verdict field off non-AI events, and the gap surfaced when someone asked what percentage of our traffic was AI. Absolute counts have no denominator.
• Store the raw user-agent alongside your verdict. Your patterns will improve, and the raw string is the only thing that lets you re-classify history instead of starting the dataset over.

const detection = detectAi(request); // null when nothing matched

track({
  url: url.href,
  // "Not AI" is a verdict. An absent field is not.
  ai_detection: detection ?? { type: null, provider: null },
  // The raw string lets you re-classify history later.
  user_agent: request.headers.get('user-agent'),
});

Most of the work is filtering

One human landing from ChatGPT can register as several referrals, because framework prefetches carry the same URL and referer. Every page a human loads triggers a swarm of requests that aren’t pageviews: static assets, hashed bundle files, framework internals, API calls. Left unfiltered, they drown the signal.

In my SDK, the filtering module grew to roughly six times the size of the detection module. Detecting AI traffic was the small problem. Deciding which requests count as page visits was the big one. The filters ended up layered.

• Extension lists catch static assets.
• A hash pattern catches bundler output like /app.3f2a9c1b.js.
• Path lists catch framework internals and API routes.
• Prefetch headers from Next.js, Remix, and the other frameworks mark speculative navigations, which get suppressed rather than counted.

The first caution: don’t let filtering run ahead of detection in a way that collapses “not worth recording” into “not AI.” Those are different verdicts, and you’ll want to tell them apart when the numbers look wrong.

The second caution is a mistake my own filter makes. Its asset-extension list includes txt, xml, md, and json, so robots.txt, llms.txt, sitemaps, and markdown docs get dropped unless the consumer opts them back in. Those are the files agents ask for. They’re AI traffic rather than noise.

Verification, and what pattern matching can’t do

A spoofer sending ChatGPT-User in its user-agent is indistinguishable from OpenAI at the string-matching tier. Everything above is deterministic. So what do you do about spoofing? The next tier is verification: several providers publish the IP ranges their crawlers use, and some support reverse-DNS confirmation.

Verification belongs on the server, behind the deterministic edge layer. Checking an IP against published ranges, holding reputation across requests, and scoring confidence all need state a stateless edge function doesn’t have. My SDK stayed deterministic by design and left scoring to the pipeline behind it.

One more boundary is worth knowing, because it’s a limit of my own code too. My SDK only tracked GET requests, so an agent that POSTs to your endpoints was invisible to it. That blind spot matters more each year as agents act rather than read.

If I could send a note back to myself on day one, it would say this. Observe at the edge. Classify crawl versus referral with signals in precedence order. Keep the denominator and the raw evidence. Spend real design effort on filtering. And treat every signature list, including this article’s, as something to verify against provider documentation, because mine still carries a signature that can never match.