(Lenny's Newsletter) How to Get Trustworthy AI Insights from Customer Data

# How to Get Trustworthy AI Insights from Customer Data: A Complete Guide

## Key Takeaways

- **AI confidence doesn't equal accuracy**: Models hallucinate quotes, cherry-pick evidence, and present generic themes with dangerous certainty
- **Verification is non-negotiable**: Define quote rules and verify every citation before using insights in decisions
- **Generic insights are useless**: Force AI to dig deeper with specific context—project goals, product knowledge, and participant backgrounds
- **Choose your model wisely**: Claude excels at thorough analysis, Gemini at video interpretation, ChatGPT at stakeholder communication
- **Four failure modes destroy decisions**: Invented evidence, false insights, weak signal, and contradictions—each with proven fixes

## The Hidden Problem: Why AI Analysis Falls Apart

Everyone's using AI to analyze customer research. But nobody's talking about the crisis hiding inside: **the output always looks confident, even when it's completely wrong.**

You can drop the same interview transcript into ChatGPT and Claude and get two entirely different narratives—different evidence, different themes, different product recommendations. Both models present their answers with equal confidence. Both sound convincing. Neither tells you that their analysis might be fundamentally flawed.

**The real problem isn't the AI itself.** It's that false but convincing-looking insights go directly into decision decks. A team builds a product roadmap on top of them. Money gets allocated. Six months later, the decision falls apart because the customer evidence behind it had "enormous holes."

As Caitlin Sullivan, a user research veteran who trains product professionals at companies like Canva and YouTube, puts it: "These mistakes are invisible until a stakeholder asks a question you can't answer, or a decision falls apart three months later, or you realize the 'customer evidence' behind a major investment actually had enormous holes."

The gap between confident-looking AI output and trustworthy insights isn't a limitation—it's a _workflow problem_. And it has solutions.

## Understanding Why AI Fails at Customer Analysis

Before you can fix AI analysis, you need to understand why models struggle with research data in the first place. The problem isn't one thing—it's that **interviews and surveys are fundamentally hard for language models to parse correctly.**

### Why Interviews Break AI

A 45-minute customer interview is messy. A participant contradicts themselves. They wander into tangents at minute 8, then circle back and reframe everything at minute 35. They hesitate on sensitive topics. They emphasize some moments and downplay others.

Real analysis requires sitting with that mess—noticing contradictions, weighing what matters, catching tone shifts that signal uncertainty. But LLMs handle messiness by _imposing structure too fast_. They find clean themes immediately, extract quotes that fit those themes perfectly, and produce tidy summaries. The output looks like analysis. But it's actually pattern-flattening.

Without explicit guidance, models will:
- Jump to consensus patterns instead of revealing edge cases
- Pull quotes that confirm their emerging themes while ignoring contradictory evidence
- Generate plausible-sounding language that wasn't actually said by the participant
- Treat a tangent as equally important as the main topic

### Why Surveys Are Worse

You'd think structured data—rows and columns—would be easier. But surveys hide complexity that models can't navigate alone.

A CSV with 200 responses to "Why did you cancel?" is just as messy as interview data, possibly worse. In an interview, you remember that someone hesitated, or complained about a specific feature earlier. In a survey, you get "It wasn't for me" with zero context.

Your data might also be messier than you think. Different tools export differently. SurveyMonkey puts question text in headers. Qualtrics uses internal codes. Some exports include metadata columns—timestamps, internal tags—sitting right next to customer responses without clear separation. If you don't tell AI which columns represent the customer's actual voice, it analyzes everything as signal. Some teams have reported AI treating internal notes ("flagged for follow-up") as customer feedback.

Even "structured" columns hide meaning. A header like "Q3_churn_probability" tells a model nothing about the scale, the original question, or whether 5/5 is positive or negative. When models don't understand these nuances, they make wrong assumptions that propagate through the entire analysis.

The four failure modes below hit both data types. Fixing them will 10x your AI analysis reliability and relevance.

## Choosing the Right Model for Your Analysis

Not all LLMs are equal for customer research analysis. After testing analysis workflows across Claude, ChatGPT, and Gemini over 100+ iterations, distinct strengths emerged for each model.

### Claude: Best for Thorough, Nuanced Analysis

**Strengths:**
- Delivers more quotes and covers more ground with less prompting
- Stays more grounded in actual data
- Gives you both breadth and depth without oversimplifying
- More conservative with confidence scores

**Tradeoff:** It gives you the whole "brain dump," so themes aren't always cleanly proven. You get comprehensive coverage, but you'll need to verify which patterns are actually well-evidenced.

### Gemini (Including NotebookLM): Best for Highly Evidenced Themes

**Strengths:**
- Provides fewer themes but with stronger grounding in source material
- Can analyze non-verbal behaviors in video (unique advantage)
- Gives stronger citations

**Tradeoff:** Expect to prompt multiple times for completeness. Quotes are sometimes shorter than helpful. You'll need to dig more to get the full picture.

### ChatGPT: Best for Stakeholder Framing

**Strengths:**
- Most creative with final framing and communication
- Excellent at packaging findings for specific audiences
- Fast iteration

**Weakness:** Least reliable for actual evidence. It combines quotes across context, invents citations, and prioritizes compelling narratives over accuracy.

**Recommendation:** If you have a choice, use Claude for analysis. You get depth and coverage without as much pushing. The tradeoff is managing unfiltered output, but that's better than discovering your LLM missed critical patterns.

## The Four Hidden Failure Modes of AI Analysis

After 2,000+ hours testing customer discovery workflows with AI, four distinct failure modes emerged—and proven fixes for each that work across platforms, data types, and models.

### Failure Mode #1: Invented Evidence (Hallucinations)

Models fail at evidence in two ways:

**Problem:** Completely fabricated quotes attributed to real participants (still happens across all three major LLMs) and "Frankenstein quotes" sewn together from multiple source fragments that somewhat represent what the user said—but aren't actually their exact words (particularly common in ChatGPT).

Both types go unnoticed unless you manually verify every quote. Both are often caused by _how you prompt_. When you ask for "a punchy representative quote (≤12 words)" or "max 100 words for each theme," you almost always trigger models to combine multiple customer statements or generate plausible-sounding language. The model isn't retrieving quotes like a search engine retrieves documents. It's _generating_ statistically likely text given the context.

If you mention "phone-checking frustration" in your context, the model predicts language that fits that pattern. Sometimes it matches the original exactly. Sometimes it's a near-miss. Sometimes it's fabricated but plausible.

**The Solution: Define Quote Rules + Verify**

**Step 1: Define what a valid quote actually looks like.** Add this to your analysis prompt:

QUOTE SELECTION RULES:

• Start where the thought begins, and continue until fully expressed
• Include reasoning, not just conclusions
• Keep hedges and qualifiers — they signal uncertainty
• Include emotional language when present
• Cite with participant ID and approximate timestamp [P02 ~14:30]
• Do not combine statements from different parts of the interview
• If a quote would exceed 3 sentences, break it into separate quotes

This removes ambiguity about what "verbatim" means. The model now knows exactly where to start and stop.

**Step 2: Verify quotes actually exist.** After your initial analysis, run this verification prompt:

QUOTE VERIFICATION

For each quote in the analysis above:

1. Confirm the quote exists verbatim in the source transcript
2. If the quote is a close paraphrase but not exact, flag it and provide the actual wording
3. If the quote cannot be located, mark as NOT FOUND

Output format:

• Quote: [the quote]
• Status: VERIFIED / PARAPHRASE / NOT FOUND
• If paraphrase: Actual wording: [what they said]
• Location: [Participant ID, timestamp, or line number]

This takes 5-10 minutes depending on data volume. But it catches errors that would otherwise end up in your deck, attributed to real participants, influencing major decisions with false evidence.

### Failure Mode #2: Generic and Useless Insights

Models find themes that describe any product in your category, not your specific situation.

**Problem:** You get outputs like:
- "Price is a factor in decisions"
- "People value reliability"  
- "Users want more real-time information"

All probably true. All completely useless for decisions. These themes could come from any competitor analysis, any industry study. They don't tell you whether _your_ users want _this specific feature_ enough to build it, or whether adding it would alienate customers who chose you specifically for being different.

AI defaults to finding consensus because language models are pattern-finding machines. They surface obvious patterns that rise to the top—what multiple participants mentioned. They generate a theme that matches the pattern. Then they move on.

But the most important insight might be something only a few people said that would be noteworthy if shared by more customers. Or it might be the tension between what people say they want and what their behavior reveals. Or it might be an edge case that's non-obvious until you see it clearly.

LLMs also bring priors from their training. If the model has seen thousands of churn analyses where "price" ranks #1, it weights toward price even if your data doesn't support it. And if your prompt mentions "pricing concerns," the model will suddenly code many responses as pricing-related, even if that wasn't the primary issue.

**The Solution: Load Context Strategically**

The standard prompt structure (Role, Context, Task, Format) works, but "context" often means either three vague lines of background or a rambling stream of everything you could think of.

Neither works. Effective context loading has at least four specific components:

**1. Project Context:** What decision are you making? "Exploring whether to add a screen" is specific. "Doing customer research" is vague, so AI defaults to generic analysis.

**2. Business Goal:** What do you actually need to know? "I need to understand whether a screen would attract new users OR alienate existing ones because that changes our priority" tells AI what to weight. Without it, AI assumes you want generic insights.

**3. Product Context:** Domain knowledge matters. "I want to see my data" means something generic. But "I want to see my data on a screenless wearable that's competing against Apple Watch" is specific enough that AI understands the constraints and can weight evidence toward what actually matters for your decision.

**4. Participant Overview:** Who's speaking? "I need real-time data" from someone who switched from Garmin means something different than the same statement from someone loyal to you who's never tried competitors. AI weights evidence correctly only if it knows who's providing the evidence.

Use this context structure for interview analysis:

CONTEXT FOR ANALYSIS:

Project: [What decision are you making? What are the constraints?]
Example: "Evaluating whether Whoop should build a screen. Key constraint:
hardware timeline is 18 months. We can't do both a screen and new sensors."

Business Goal: [What outcome matters most? What decision becomes easier with this insight?]
Example: "Do new users want a screen more than existing users do? This determines
whether we prioritize attracting new customers or keeping loyal ones."

Product Context: [What should the model know about your product/market?]
Example: "Whoop is a screenless wearable. Competitors include Apple Watch (has screen),
Garmin (has screen), Oura Ring (screenless). Our strength is simplicity and battery life."

Participant Overview: [Who are these people? What's their background?]
Example: "Mix of loyal users (3+ years), recent switchers from competitors, and people
who churned. Note: Switchers may have different priorities than long-term loyalists."

This context eliminates vagueness. The model now understands your actual decision, your constraints, and what evidence matters. It will weight findings accordingly and avoid generic pattern-matching.

### Failure Mode #3: Weak or Non-Actionable Signal

Sometimes AI finds patterns that seem directional but don't actually guide better decisions.

**Problem:** You get a theme like "18% of churned users wanted more actionable guidance." This sounds like a job-to-be-done. But it's too broad to act on. Should you focus on clearer metrics? Workout plans? Both? The theme doesn't clarify. Plus, most of it might be unrelated to your actual decision (like a screen).

AI clusters based on surface-level similarity, not on what actually moves your decision forward. This is especially problematic in surveys, where you're starting with sparse responses and AI is guessing at meaning. A response like "It's not for me" could mean:

- Too expensive for the value
- Too technical—I'm not a serious enough athlete
- I don't want another device to charge
- I need a screen and Whoop doesn't have one

Four completely different business implications. Without guidance, AI lumps similar-sounding responses together into meaningless clusters.

**The Solution: Segment and Weight by Decision Relevance**

Instead of asking AI to find clusters, ask it to segment users by what matters for _your specific decision_:

SEGMENTATION FOR DECISION-MAKING

Instead of: "Find themes in churn reasons"
Use: "Segment churned users by whether they wanted a screen feature specifically,
versus other reasons they left. For each segment, tell me:

1. How many people fall into this segment? (estimate percentage)
2. What was their primary frustration? (quote)
3. Would building a screen address this reason? (yes/partial/no)
4. How likely are they to return if we build a screen? (if we know this)"

This forces AI to segment by relevance to your decision, not just surface similarity. You get actionable clarity: "30% of churned users would likely return if we built a screen. 20% never wanted a screen; they left because of pricing. 50% are unclear."

That's something you can actually decide on.

### Failure Mode #4: Contradictory Insights

Sometimes AI finds patterns in one part of your dataset that directly contradict patterns in another part—but presents both as true without flagging the contradiction.

**Problem:** A user might tell you "I love how simple Whoop is, no screen to worry about" in one part of the interview, then later say "It would be more useful if I could see my data in real-time on my wrist." Both are true. But they point to different product directions. If AI extracts these as separate themes without flagging the tension, you might invest in the wrong direction.

In surveys, this gets worse. A user rates "having a screen" as "very important (5/5)" but also wrote "I like that Whoop is simple" in an open-ended question. The contradiction is real but hard to spot across data.

AI doesn't flag these tensions naturally. Models are designed to find consistent patterns, not contradictions. Contradictions look like noise to models, so they either ignore them or present conflicting themes as separate, equally-weighted findings.

**The Solution: Forced Contradiction Detection**

After your initial analysis, add this step:

CONTRADICTION DETECTION

Review the themes and participant quotes above. For each finding:

1. Are there any contradictory statements from the same participants?
   Example: User said "I love the simplicity" but also "I wish I could see my data on my wrist"

2. Do different user segments have conflicting needs?
   Example: Loyal users prioritize battery life; new users prioritize features

3. Is there a gap between what users SAY they want vs. what they DO?
   Example: Users say battery life is important, but three asked about screen features immediately

4. For any contradiction found, mark as TENSION and explain what it means for your decision

Output format:

• Tension: [Description]
• Evidence: [Quotes from different parts of data]
• What it means: [How this affects your product decision]

This forces the model to surface tensions instead of hiding them. Tensions are often where the real insight lives.

## Practical Prompting Workflow for Interviews

Now that you understand the failure modes and fixes, here's the complete workflow for analyzing interview data with AI:

**Step 1: Load Context**

Start with your four-part context (Project, Business Goal, Product Context, Participant Overview). This prevents generic analysis before it happens.

**Step 2: Run Initial Analysis with Quote Rules**

Ask the model to identify themes, with your quote selection rules included. At this stage, you're looking for patterns, but patterns that are verified and specific to your decision.

**Step 3: Verify Every Quote**

Run the quote verification prompt. Flag any "NOT FOUND" or "PARAPHRASE" results. Replace with actual verbatim quotes.

**Step 4: Test Themes Against Your Decision**

For each theme, explicitly ask: "Does this help me decide whether to build a screen?" If the answer is "not really" or "maybe," dig deeper or discard it. Generic themes add nothing.

**Step 5: Detect Contradictions**

Use the contradiction detection prompt. These tensions often reveal the real decision-making insight.

**Step 6: Segment by Relevance**

Break participants into groups that matter for your decision (switchers vs. loyalists, wanted screen vs. didn't, etc.). This reveals whether patterns hold across segments or are specific to groups.

**Step 7: Synthesize with Verification**

Write your final summary using only verified quotes, addressing contradictions, and weighted by segment relevance.

## Practical Prompting Workflow for Surveys

Surveys require different handling because you're starting with less context per response. The workflow is similar but emphasizes clarifying ambiguous responses:

**Step 1: Load Context + Column Clarification**

Start with your four-part context. ALSO specify which columns are customer voice and which are metadata/internal notes. Example:

COLUMN SPECIFICATION:

• Customer voice: response_text, open_ended_feedback
• Participant data: user_id, signup_date, cohort
• Internal only: internal_notes, qa_flag
  Analyze only customer voice columns.

This prevents AI from treating metadata as customer feedback.

**Step 2: Clarify Ambiguous Responses**

For open-ended responses that are vague, ask the model to generate possible meanings:

For each response that's ambiguous or vague (like "It's not for me"):

1. List 3-4 possible interpretations
2. For each interpretation, note how it would affect our screen decision
3. Flag which interpretations are speculative vs. fairly clear from context

**Step 3: Segment by Response Clarity**

Separate responses into:
- Clear and specific ("Screen was 90% of why I left")
- Somewhat clear ("I wanted more features on the go")
- Too vague to act on ("Wasn't right for me")

Weight your analysis toward the clear responses.

**Step 4-7: Same as Interviews**

Quote verification, decision relevance testing, contradiction detection, segmentation, and synthesis—all follow the same pattern.

## The Complete Context Template for Analysis

Use this template to prepare your analysis prompt. It works across all three major models:

PROJECT CONTEXT:
[What specific decision are you making? What are your constraints?]

BUSINESS GOAL:
[What outcome matters most? What would change your decision?]

PRODUCT CONTEXT:
[Domain knowledge: what should the model know about your space?]

PARTICIPANT OVERVIEW:
[Who are these people? What's their background/history?]

DATA SPECIFICATION:
[If using surveys: which columns are customer voice? Which are metadata?]
[If using interviews: any patterns in how these were conducted?]

QUOTE SELECTION RULES:

• Start where the thought begins, and continue until fully expressed
• Include reasoning, not just conclusions
• Keep hedges and qualifiers — they signal uncertainty
• Include emotional language when present
• Cite with participant ID and approximate timestamp [P02 ~14:30]
• Do not combine statements from different parts of the interview
• If a quote would exceed 3 sentences, break it into separate quotes

THEMES YOU'RE LOOKING FOR:
[Optional: if you have hypotheses to test, list them. If not, say "open exploration"]

YOUR TASK:

1. Identify 4-6 distinct themes related to [your specific decision]
2. For each theme: provide 2-3 verbatim quotes following the rules above
3. Estimate what percentage of participants support each theme
4. Flag any contradictions or tensions you notice
5. For each theme, explain how it affects our [specific decision]

## Model Comparison: What You Actually Get

To show you real differences, here's what three models returned when analyzing the same interview transcript for screen feedback:

**Claude:** Found 5 themes with longer quotes, covered more ground, gave conservative confidence scores but didn't always summarize themes consistently.

**Gemini:** Found 4 highly-cited themes with shorter quotes, stronger grounding in source material, required multiple prompts to get completeness, but was excellent at noting source context.

**ChatGPT:** Found 6 themes with punchy framing, excellent for stakeholder communication, but combined quotes from different parts of the interview and occasionally invented supporting evidence.

For analysis work where accuracy matters more than speed, Claude wins. For final framing where credibility matters, ChatGPT works if you verify quotes. For video analysis, Gemini's unique advantage is worth the extra prompting.

## Why Most Teams Fail at AI Analysis

The common mistakes that destroy AI analysis:

**1. Using AI without verification.** Running one prompt, getting one output, and trusting it entirely. This is how false insights end up in decision decks.

**2. Vague context.** Telling the model "analyze our churn data" without explaining what decision you're making or why it matters. This triggers generic pattern-finding.

**3. Skipping quote rules.** Asking for "verbatim quotes" without defining what that means. Models fill the gaps with assumptions that result in fabricated citations.

**4. Ignoring contradictions.** Presenting conflicting themes as separate, equally-weighted findings. The tension is often where the real insight lives.

**5. Mixing all models together.** Using ChatGPT for speed without understanding it's the least reliable for evidence, or using Gemini without recognizing it needs more prompting for completeness.

**6. Treating themes as facts.** Assuming a theme that AI found must be important. Without testing against your specific decision, themes are just patterns, not insights.

## Conclusion and Next Steps

The gap between confident-sounding AI analysis and trustworthy insights isn't a limitation of the technology—it's a workflow problem with proven solutions.

To get insights you can actually trust:

1. **Define your context specifically.** Project goal, business decision, product knowledge, participant backgrounds. Vague context triggers generic output.

2. **Make verification non-negotiable.** Quote rules, verification prompts, and contradiction detection take minutes but catch errors that would otherwise influence major decisions.

3. **Segment by decision relevance.** Ask "does this help me decide?" for every theme. Generic patterns aren't insights.

4. **Choose your model by strength.** Claude for analysis depth, Gemini for video, ChatGPT for framing—but verify quotes always.

5. **Test before you build.** Don't let AI analysis go directly into a decision deck. Verify, test, segment, and detect contradictions first.

The customers who trust you are based real insights, not confident-sounding hallucinations. The product decisions that succeed are built on verified evidence, not generic patterns. Using AI for customer research doesn't mean sacrificing rigor—it means being deliberate about where models need help and verification, then building that into your workflow.

Start with one analysis this week. Apply the four failure modes and their fixes. You'll see the difference immediately.

Original source: How to do AI analysis you can actually trust

powered by osmu.app