The Chart Said Engagement. The Data Said One Person
for several weeks straight, our CTA-click count looked like a real, if modest, engagement trend. Single digits most days, the occasional double-digit day, nothing dramatic but nothing zero either. Read as a top-line number, it looked like the kind of thing you'd point to and say "people are clicking."
Then we broke it out by distinct visitor ID, and the story changed completely. One recurring ID accounted for somewhere between 70 and 90% of every CTA click, every week, for over a month running. Everyone else, the visitors who actually looked like real, distinct people, showed up as one click each, if that.
We hadn't been looking at engagement. We'd been looking at one person, probably internal, probably testing the site, refreshing the page.
Why an Aggregate Number Hides This by Default
A total doesn't know who clicked. "15 CTA clicks this week" is true whether that's fifteen different people each clicking once, or one person clicking fifteen times. The chart looks identical either way. Nothing in a top-line count forces the distinction, and most dashboards default to showing you the total precisely because it's the simplest thing to compute and the easiest thing to put in a headline.
This is exactly what makes the mistake so easy to make without noticing. Nobody consciously decides to read one visitor's repeat activity as market demand. You just look at a chart that says "up" or "steady" and move on, because the chart isn't lying, exactly. It's just answering a question you didn't ask ("how many clicks happened") instead of the one you actually care about ("how many different people are interested"). There's a name for this: insensitivity to sample size, the tendency to treat a small, skewed sample as if it carries the same weight as a large, representative one. Fifteen clicks feels like more evidence than it is, because nothing in the number itself tells you how few distinct sources it actually came from.
We caught it because something was checking the distinct-ID breakdown every single day, not because the aggregate number ever looked wrong. It never did. It looked fine the entire time.
What the Real Breakdown Looked Like
Once we split by visitor, a clear pattern emerged and held for weeks: one recurring ID logged the overwhelming majority of total events, most of the CTA clicks, and, when a form-related event fired for the first time in the entire tracked history, that too. Every other visitor in the same window logged a handful of events each, mostly a single pageview and pageleave a few seconds apart, no CTA, no deeper engagement.
Two or three genuinely distinct visitors a week, most of whom bounced immediately, versus one visitor generating the bulk of every "engagement" metric we had. Read as an aggregate, this was a modestly encouraging chart. Read by distinct ID, it was mostly noise from what looked like internal or test traffic, with a thin trickle of real, if unconverted, visits underneath.
Neither number was false. They were answering different questions, and only one of them was the question that mattered for deciding what to build next.
The Decision Risk This Creates
If you're a post-seed founder using engagement data to decide what's working, an aggregate-only read can point you at completely the wrong conclusion in three specific ways. It's the same structural weakness we've written about when weighing conflicting evidence sources: analytics tell you what happened, not why, and a healthy-looking number can just as easily mean one person is doing the work of fifty.
You credit a feature that no customer actually used. If a CTA button's click count looks healthy because one internal visitor keeps clicking it during testing, you might conclude the button's messaging is working. It might be doing nothing for a real visitor. You'd have no way to tell from the total alone.
You miss that your real funnel is thinner than it looks. Strip out the recurring visitor and the "healthy" click count for a given week might collapse to two or three genuine data points. That's a materially different confidence level than the aggregate implied, and it changes how much weight that data point should carry in any decision.
You stop asking who's actually driving a number. Once an aggregate chart looks fine, the natural move is to stop interrogating it. That's the exact moment a single non-organic visitor can quietly dominate your read on demand for weeks, because nobody's checking underneath the total anymore.
The Check That Would Have Caught This Sooner
Break every engagement metric out by distinct ID before trusting the total. Not occasionally, every time a number is about to inform a decision. If one ID accounts for a large share of any metric, that metric needs a footnote before it goes anywhere near a roadmap conversation. Most platforms, GA4 included, let you flag and filter recurring internal traffic before it ever reaches a report. Use it.
Flag recurring visitors explicitly, don't just note them once. A visitor who shows up constantly, across weeks, with no gap, is worth naming and tracking as its own category, not folding back into "total engaged visitors" every time a new report gets pulled.
Ask what a genuinely distinct visitor count looks like, separately from a genuinely distinct event count. Two or three real visitors a week and fifteen total clicks are both true numbers. Only the first one tells you anything about actual demand.
Treat "the chart looks steady" as a reason to check closer, not a reason to stop checking. The version of this mistake that costs you isn't a chart that looks obviously wrong. It's one that looks fine for long enough that nobody thinks to ask who's behind it.
A single recurring visitor isn't a scandal, most sites have one, usually someone on the team. The mistake isn't that the visitor exists. It's building confidence in a growth signal without first checking whether the signal is coming from more than one person.
We've sat with an uncomfortable number before and had to resist the urge to react to it. Six posts and a 100% bounce rate taught us the same lesson from the other direction: a number that looks bad can be just as unreliable as one that looks good, if you haven't checked what's actually behind it.
If you're deciding what to build next, start with who the evidence is actually from before you decide what the evidence says.
