What I Learned About User Interviews
Software developers are very good at building things. Unfortunately, this is only useful after they have picked the right thing to build.
A few years ago, I co-owned a startup that developed software for relationship management. It was similar to a CRM, but focused on building a professional network rather than managing a sales pipeline.
We had plenty of ideas. We also had the technical skills to implement them. What we did not have was a reliable method for deciding which ideas deserved to be implemented.
It took us more than a year and several expensive wrong turns to understand some fairly fundamental facts about our product: it would be used for business rather than private relationships, it needed both a mobile and a desktop version from the start, and the right customers could justify paying around 100 Euros per month.
In hindsight, none of these insights required a year of software development. They required better conversations.
TL;DR: Do not ask people whether they like your idea. Ask them how they dealt with the problem the last time it occurred. Talk about their life, not your solution. Treat compliments as politeness and commitments as evidence. Use AI to prepare and analyze interviews, but never mistake a simulated customer for a real one.
I explore several of these ideas, particularly the role of AI, in my German-language chapter “Klicks, Code und Kundenlächeln — Mit KI vom Commit zum Kompliment” in this book. Rob Fitzpatrick’s The Mom Test also influenced the way I think about user interviews, and I strongly recommend it.
Opinions Are Cheap
Suppose I tell you about my brilliant new app and ask:
Would you use an app that helps you maintain your professional relationships?
What are you going to say?
You may say yes because the idea sounds vaguely useful. You may want to encourage me. You may simply want to avoid an awkward discussion. None of this means that you will ever install the app, let alone pay for it.
The question asks you to speculate about an imaginary future. People are remarkably bad at this. They are much better at describing what they have already done.
Useful interview questions therefore point backwards:
- When did you last have this problem?
- What happened?
- How did you solve it?
- What did that solution cost you?
- What was inconvenient about it?
- Who else was involved in the decision?
Past behavior does not guarantee future behavior. However, it is considerably better evidence than a friendly prediction made during a coffee break.
The future tense is where bad product decisions like to hide.
Talk About the Problem, Not Your Idea
Founders naturally want to explain their products. After all, they have spent weeks thinking about them, and possibly months building them. Once they finally have a potential customer in front of them, remaining quiet feels like a wasted opportunity.
However, the moment I start explaining my idea, the interview changes. I am no longer researching a problem, but I am pitching a solution.
In one interview I made for my startup, a user was complaining about a certain feature of the user interface. She clearly didn’t understand the idea behind it. I immediately felt the urge to explain how it’s supposed to work, and how she could benefit from it. But while that might have helped one user at one specific point, it would have been disastrous for the all other users: clearly, there was a deep, underlying problem that made the user interface clumsy and hard to grasp. That lead to a solution of the problem, rather than one more user able to navigate something that didn’t work well enough.
That’s the key: Nobody cares about my idea merely because it is mine. Customers care about their own problems. This sounds obvious, yet it is surprisingly difficult to remember while presenting a feature that took three months to implement.
A good user interview should teach me something even if I never mention the product at all.
Commitments vs Compliments
“That sounds interesting” is not evidence. “I would definitely use that” is not evidence either.
People give compliments freely. They make commitments much more carefully. A meaningful signal therefore costs the interview partner something: time, reputation, access, or money.
It may also be introducing me to a colleague who has the same problem, sharing data required for a pilot, agreeing to another meeting with the decision maker, testing a prototype (that’s basically an investment of time), or paying for the product.
The strongest compliment a customer can give a product is not enthusiastic praise, it is reaching for the credit card.
Now, the interview need not (or rather should not) end in a sale. Early conversations often happen long before there is anything to buy. Nevertheless, asking for a reasonable next step separates general goodwill from genuine interest.
Ask About the Last Time
One phrase has improved my interviews more than almost anything else:
Tell me about the last time this happened.
It turns an abstract discussion into a story. Stories contain the details that general statements conveniently omit.
Someone may claim that organizing contacts is extremely important. When asked about the last time an important contact slipped through the cracks, she may struggle to remember an example. That does not prove that the problem is irrelevant, but it should make me cautious.
Conversely, she may describe a missed introduction, a spreadsheet maintained by three people, two hours of manual preparation before every meeting, and a tool the company already pays for but dislikes. Now I have evidence of frequency, cost, existing alternatives, and urgency.
The workaround is particularly interesting. If people have already invested time or money in an imperfect solution, the problem probably matters. If they have lived with it for five years without doing anything, it may not be as painful as they say.
Prepare the Questions, Then Have a Conversation
I prepare my interview questions in advance. This makes several interviews easier to compare and helps me avoid accidentally turning each conversation into a product demonstration.
However, an interview is not a questionnaire.
The interesting insight is often hidden behind an unexpected remark. If I mechanically proceed to question number seven, I may miss the most valuable part of the conversation. A prepared guide provides direction, but it should not prevent me from asking for additional context like: Why was that difficult? What happened next? Can you give me an example?
Essentially, I want enough structure to compare interviews and enough flexibility to follow reality when it refuses to follow my structure.
Four to Six Interviews Can Change the Picture
In my experience, the first few interviews produce the biggest surprises. This experience is also backed by the “The Mom Test”. After roughly four to six conversations with a well-defined target group, patterns should begin to appear.
This does not mean that six interviews prove a market exists. They do not provide statistical certainty, and they certainly do not replace testing an actual product.
They do something else: they expose obviously wrong assumptions early.
If every conversation produces a completely different problem, this is also an important result. The target group may be too broad. The supposed problem may consist of several unrelated problems. Or I may be asking questions that are too general to reveal a pattern.
More interviews are not always the solution. Sometimes the next step is to narrow the audience.
What AI Is Good For
Generative AI can make user interviews considerably more efficient. I use it as a sparring partner in three places.
First, it can review an interview guide. I can ask it to identify leading questions, requests for speculation, hidden product pitches, and gaps in the sequence. This is particularly useful because a question can look neutral to the person who wrote it while making the desired answer painfully obvious to everyone else.
Second, AI can simulate an interview. With a detailed description of a role, its responsibilities, its environment, and its constraints, it can produce plausible answers and help me practise follow-up questions. It may also uncover assumptions I had not considered.
Third, AI can summarize real interviews. Given anonymized notes or transcripts, it can extract recurring problems, workarounds, decision criteria, objections, and phrases used by the interview partners.
These are valuable uses. They save time and improve preparation.
What AI Cannot Do
A simulated user interview is not a user interview.
An AI model can generate a plausible account of how an A&R manager, a software architect, or a medical professional might work. Plausibility is useful for forming hypotheses. It is not proof that a particular person has a particular problem.
An AI cannot:
- show that someone has recently experienced the problem,
- make a credible commitment,
- introduce me to the budget owner,
- struggle with a real prototype,
- risk money or reputation, or
- reveal the unexpected constraints of one particular organization.
It also tends to be helpful. Unfortunately, helpful interview partners are exactly the ones most likely to validate an idea that does not deserve validation.
I therefore treat simulated interviews as preparation for real conversations, never as a replacement for them. The answers tell me what to investigate. They do not tell me what is true.
There are practical limits as well. Transcripts lose body language, hesitation, irony, and context. Models may reproduce biases from their training data, especially for narrow target groups. And interview transcripts can contain personal or confidential information, so I remove or replace such details before submitting them to any external service.
AI is a capable assistant. It is a very unreliable customer.
What I Do Differently Today
My current process is simple:
- I define a narrow target group and the risky assumption I need to examine.
- I prepare a short set of questions about recent, specific experiences.
- I ask about the problem, current behavior, workarounds, cost, and decision process.
- I avoid explaining the solution until the research part is over.
- I write down exact phrases as well as facts. The customer’s language is often useful later.
- I look for a reasonable commitment or next step.
- I compare several interviews and search for repeated evidence.
- Only then do I decide what to build or test.
This process does not eliminate business risk. Nothing does. It does, however, make wrong assumptions cheaper.
Summary
The greatest challenge in software development is rarely finding 100 ideas. It is deciding which 99 should not be implemented.
User interviews help with that decision, provided I do not ask users to design the product for me. Their job is to tell me about their world: what they tried, what failed, what it cost, and why it mattered. My job is to listen, identify the underlying problem, and decide what to do about it.
Code can tell me whether I built something correctly. A good user interview helps me decide whether I should build it at all.