Why Most Product Decisions Are Built on Sand

You've shipped a feature. The team expects it to solve a core pain point. But usage metrics flatline. Support tickets spike. The product manager sighs. Sound familiar?

The hard truth: most teams operate on hunches disguised as data. Surveys, CRM notes, and even direct user interviews can be wildly misleading. As Erika Hall famously put it, asking a question directly is the worst way to get a true and useful answer.

To build products that actually work, we need to move beyond surface-level feedback and understand users across four distinct levels: what they say, what they think and feel, what they do, and why they do it. This framework, originally articulated by Hannah Shamji, is a powerful lens for any developer or product team.

Let's break down each level with concrete techniques you can apply starting tomorrow.

UX researcher observing user behavior on a mobile app prototype with emotion wheel diagram overlay Dev Environment Setup

The Four Levels: From Noise to Signal

Level 1: What They Say (The Most Unreliable)

This is the easiest data to collect — and the most dangerous to trust. People exaggerate, focus on edge cases, and rationalize their behavior after the fact. A user saying "I absolutely need a comparison table" doesn't mean they can't achieve their goal without it.

What to do: Use surveys and CRM data only as a starting point, never as a decision-maker. Treat every stated need as a hypothesis to be tested.

Level 2: What They Think and Feel (Context, but Still Biased)

Interviews and diary studies give richer context, but memory is a faulty filter. Users reconstruct their experience through the lens of how they want to be perceived.

Practical technique: Use the Emotion Wheel by Geoffrey Roberts during interviews. Instead of asking "How did you feel?", present the wheel and ask users to point. It moves the conversation beyond "good" or "bad" to precise emotions like "frustrated" or "curious."

Level 3: What They Do (The Gold Standard)

Behavioral data — clicks, scrolls, hovers, time on task — doesn't lie. But raw analytics can be misleading without context.

Practical technique: Run task analysis sessions. Observe users completing a core workflow without asking them to speak aloud. The "speak-aloud" protocol actually disrupts natural behavior. Instead, watch where the mouse hovers without clicking, where users scroll back and forth, and how long they pause. Only ask questions after they signal they're done or stuck.

# Simple session recording analysis script (conceptual)
# Track hesitation points in a user session

import pandas as pd

# Assume we have a CSV with mouse events
session_data = pd.read_csv('user_session_001.csv')

# Filter for hover events longer than 3 seconds (hesitation)
hesitations = session_data[
    (session_data['event_type'] == 'hover') &
    (session_data['duration_ms'] > 3000)
]

# Group by UI element and count
pain_points = hesitations.groupby('element_id').size().sort_values(ascending=False)
print("Top hesitation points:")
print(pain_points.head())

Level 4: Why They Do It (The Deepest Insight)

This level uncovers underlying motivations, goals, and root causes. It requires repeated, trust-based conversations and real workflow observation.

Practical technique: Use "mirroring" — repeat back what the user said, or ask the same question paraphrased. This often triggers a second, more honest answer with richer context.

Triangulation: Reconciling Conflicting Data

Different levels will often contradict each other. A user says they want X (Level 1), but their behavior shows they actually use Y (Level 3). That's not a problem — it's the signal. Triangulate by combining methods: survey + analytics + observation + interview.

For deeper reading on user research methods, check out this practical guide on uncovering user needs — it covers exposure hours, live UX testing, and helpdesk insights.

Data analyst triangulating user feedback, analytics, and behavioral data on a dashboard Developer Related Image

Common Pitfalls & Limitations

  • Over-reliance on NPS scores: NPS measures sentiment, not behavior. It's a vanity metric. Focus on task success rate and time-on-task instead.
  • Confirmation bias in user testing: Many teams "validate" existing assumptions rather than genuinely exploring. Go in with questions, not answers.
  • Emotions as a distraction: Alin Buda argues that our job is to solve problems, not perform empathy. Use emotional signals as a diagnostic tool, not the end goal.

Next Steps: Build a Research Habit

  • Exposure hours: Every team member should spend 2 hours every 6–8 weeks watching real users.
  • Live UX testing: Invite the whole company to observe sessions. Make user struggles visible.
  • Co-design sessions: Show users prototypes and ask them to rank features. Their priorities will surprise you.

For a deeper dive into how AI and agentic workflows are changing product development, see this analysis on many agents, one team.

Design team conducting a live co-design session with users ranking feature cards on a whiteboard Development Concept Image

The Bottom Line

Real user understanding doesn't come from expensive tools or large sample sizes. It comes from a disciplined practice of observing, diagnosing, and building trust. Stop validating. Start researching.

One action for this week: Schedule a 30-minute session to watch one user complete your core feature. Don't ask questions. Just watch. Then ask yourself: what did I learn that no survey could have told me?

This content was drafted using AI tools based on reliable sources, and has been reviewed by our editorial team before publication. It is not intended to replace professional advice.