Calls Said One Thing. Analytics Said Another. Here's What We Actually Do
no single source wins by default. When calls, tickets, and analytics conflict, the fix isn't declaring a winner it's running each signal through seven contextual lenses (who raised it, where it sits in the journey, whether behaviour confirms it, how severe and commercially relevant it is, whether anything else corroborates it, and how reversible a wrong call would be). Here's the worksheet we use to do that.
Three customers told you on calls that onboarding is confusing. Your support queue is quiet on onboarding but full of billing complaints. Your analytics show activation is actually up week-on-week.
The tempting move is to pick the source that agrees with what you already wanted to build, and quietly discount the rest. That's not analysis. That's confirmation with extra steps.
Why 'which source wins' is the wrong question
Each source has a structural bias baked into how it's collected:
▸ Calls over-represent whoever agreed to talk to you often your most engaged or most annoyed users, rarely the silent majority.
▸ Support tickets capture friction sharp enough to make someone stop and complain. Plenty of pain never becomes a ticket; people just leave.
▸ Analytics show what people did, not why. A rising number can mean the thing works or that a confusing step is being brute-forced past.
Nielsen Norman Group puts this clearly in their research on triangulation: all research methods are limited in some way, and a lack of context and meaning is a persistent weakness of quantitative methods used alone. Their prescribed fix applying multiple research techniques so the limitations of one method are offset by data from another is the same move this article makes operational for founders who don't have a research team to run it.
Source: Triangulation: Get Better Research Results by Using Multiple UX Methods
Rank sources universally and you'll systematically over-trust one bias every time. The onboarding conflict doesn't resolve by declaring a winner. It resolves by asking, of each signal: whose problem is this, where does it sit in the journey, how often, how severe, does it fit strategy, and does anything corroborate it?
Evidence is a constellation, not a ladder
Treat your four evidence sources calls, tickets, behaviour, market as points in a constellation around a decision, not rungs on a ladder. What connects each point to the decision is a set of contextual lenses. The same ticket can be decisive or irrelevant depending on whose it is and where in the journey it lands. There is no universal ordering. There's only 'in this context, weighted this way.'
Seven lenses for reconciling a conflicting signal
Run each conflicting signal through these, in order:
1. Identity: Whose problem is this? A target-segment customer and a poor-fit trial user are not equal evidence for a roadmap change.
2. Journey: Where in the customer journey does it occur? A first-run blocker and a power-user nicety carry very different weight.
3. Behaviour: Does observed behaviour confirm or contradict what was said? Said-versus-did gaps are information, not noise.
4. Severity: How much does it hurt when it hurts? Frequency and severity are different axes; a rare catastrophe can outrank a common annoyance.
5. Commercial relevance: Does solving it move something that matters to the business, or just make a vocal user happier?
6. Corroboration: Does a second, independent source point the same way? One signal is a hypothesis; two independent ones is a pattern.
7. Reversibility: If you act and you're wrong, how cheaply can you undo it? Cheap-to-reverse calls can clear a lower evidence bar.\
Applied to the onboarding conflict: if all three calls are target-segment customers hitting a first-run blocker (identity + journey), and the 'up' activation number turns out to be users grinding through a confusing step rather than sailing past it (behaviour contradicts the surface number), the calls suddenly carry real weight. Not because calls beat data — because the lenses reweighted them.
The evidence reconciliation worksheet
Put the conflicting signals in columns and score each lens against them. Filling this in usually dissolves the 'which one wins' argument on its own because the disagreement stops being about the sources and starts being about a specific, nameable lens.\
| Lens | Signal A e.g. Calls | Signal B e.g. Analytics | Signal C e.g. Tickets |
|---|---|---|---|
| Identity: whose problem? | |||
| Journey: where in the flow? | |||
| Behaviour: said vs. did | |||
| Severity | |||
| Commercial relevance | |||
| Corroboration | |||
| Reversibility | |||
| Weighted read |
The bottom row is the judgement call. Because every cell above it is visible, a colleague who disagrees can point at the exact lens where your read diverges from theirs. That's the difference between a decision and an assertion.
What data is actually good at and what it isn't
Build the Right Thing by Haja J Deen makes a point worth keeping close: data is especially useful for ruling out bad choices, and rarely sufficient to select the winner on its own. Deen's framing is humility let the data speak, but don't pretend it decides for you. Analytics can tell you a feature is unused far more reliably than it can tell you which of two promising features to build next.
This lines up with how research practitioners describe triangulation more broadly. The question should never be 'how much time can we spend on research' but rather 'how much risk of problems or failure are we willing to accept?' Expensive choices warrant robust triangulation. Simple, easily reversible decisions don't need as much investment — but checking whether other available data supports the favoured approach is worth at least a few hours regardless. That's lens seven, reversibility, showing up independently in the UX research literature.
Use quantitative evidence to eliminate. Use the seven lenses to choose.
FAQ: Does calls, tickets, or analytics take priority?
None of them do by default. Priority is assigned per-decision by running the conflicting signals through the seven lenses above identity, journey, behaviour, severity, commercial relevance, corroboration, and reversibility not by a fixed hierarchy of evidence types.
A support ticket from a target-segment customer hitting a first-run blocker can outweigh a week of 'activation is up' analytics. A call from a poor-fit trial user complaining about a power-user edge case usually shouldn't move a roadmap at all. The lenses make that call explicit, not instinctive.
Reconciling evidence once is useful. Keeping the reasoning visible so next quarter's team remembers why you weighted the calls the way you did requires traceability: see why a feedback repository is not a decision system.
The reconciled read is exactly what feeds the ranked moves in the Monday product briefing.
Running this process without a dedicated research function? P8 for founders covers how that gap usually gets filled or doesn't.
Terms in this framework feel unfamiliar? The P8 glossary defines them without the jargon.
See how P8 attaches receipts to every recommendation so a reconciled signal doesn't get lost start with your first brief.
This article adapts frameworks from Build the Right Thing by Haja J Deen and reinterprets them for founders handling conflicting evidence, with supporting context from Nielsen Norman Group's research on triangulation. Product capabilities described reflect P8's current approach, not a guarantee of specific features.
