Don’t ignore post-user testing questions: they often offer the most valuable insights
Post-user testing questions are often a treasure trove of insights for product strategy.
“What’s the unique selling point of your product?” A Lead Product Designer asked me, and my weak answer made me realize the value of post-user-testing questions.
Designing for Business-to-Business (B2B) companies has taught me to dabble in a number of fields beyond my job description. When trying to motivate users to buy a product costing $25,000/mo, a basic understanding of user needs is often insufficient.
However, I realized that one data source often yields important insights into user motivations that I’ve ignored: the pre- and post-user testing questions I’ve used for years.
If you’ve been asking pre- or post-user test questions and not reporting your findings, you might be ignoring a goldmine of insights that Product cares about.
To understand why, we first need to talk about strategy.
Understanding Strategy through value
In simple terms, a business strategy will be outside the realm and scope of most designers.
This is where Product teams, C-level executives, and Principal Product Designers/VPs of Design will make decisions. But understanding Strategy is something that you can implement in your designs.
Understanding strategy as a designer often comes down to a single question: “How do we design a good user experience (with a lot of value) that motivates users to pay for this product?”
As designers, we usually don’t want to consider the business side. However, in lean times like we’re in now, one of the ways to not only keep your job (but also offer value) is to understand user motivations around becoming a customer.
Product designers who are willing to learn about how businesses operate and grow can become trusted business partners and work directly with founders to impact revenue.
It’s one of several good ways for design to “get a seat at the table”, command higher rates / salaries, etc. — Neil Reinicker
The reason is that strategy requires you to think beyond the user experience and toward value. To understand this question, let’s ask a simple question:
“Would you use a beautifully designed application that you didn’t find valuable?”
The answer, for most of us, is no. To see why, you can browse the app store for beautifully designed applications that have been abandoned due to users not using them enough.
This is why the product team centers their work around Aha moments, in which users recognize not a good user experience but something of value to them.
The product team aims to hit these value moments as quickly as possible.
However, how are businesses supposed to find these moments of value to the user? They often rely on customer interactions, such as sales calls and other data gathered. But sometimes, this picture can be incomplete.
This is why the user testing questions we ask around post-test can be beneficial. These include questions like:
What did you like least about using the Product?
What are your job responsibilities (or what you do daily)?
When was the last time you engaged with (a specific task)?
Etc.
These questions probe the user for what they find valuable or what might be those Aha moments for them. Product cares deeply about these things, yet we often don’t think to include these things in our findings.
Our user testing is often centered entirely on usability, which is understandable. We want to understand where the user struggles, where they have frustrations or more. However, what can be equally important to capture are the moments where a user doesn’t say, “Cool, that was a good experience.” It captures moments when the user says, “This is valuable to me.”
We often get so wrapped up in the tiny details of our work (such as button placement or wording) that zooming out and asking about the overall value we provide and what users find valuable can be a great way to align user needs with what the business wants.
After all, if you’re able to show that users aren’t getting to that “Aha moment” where they immediately see the value of your Product (and maybe get out of the checkbook) because of usability issues, they’ll be much more willing to invest the time and resources to help fix that.
However, this is only one of the terms to pay attention to.
Unique Selling Points (USP), or understanding competitor’s designs
Competitive analysis is a weird research method for some designers, but it can be crucial to understanding a new field if you’re unsure what you’re getting yourself into.
The part of a competitive analysis that matters to many designers is the onboarding experience. The idea is to register for an account and explore what the Product offers you from the start to get a sense of what they offer and how they present their Product.
This is often the work of user researchers, business analysts, and senior designers, but that doesn’t mean you can’t learn from your experience. The core takeaway from this experience is the question I started with: “Why would users want to buy from us, instead of our competitors?”
Is there a particular strength or feature that drove them to our website instead of our competitors? If that’s the case, shouldn’t we make those strengths more apparent with our designs?
For example, imagine you’re designing a homepage for your company, and you know real estate is desirable, particularly at the top of the page. What do you choose to put there?
Would you advertise that your Product has a community marketplace when all your competitors do as well? Or would you instead advertise that your community is built around some of the top hospitals in the world?
These are questions that, luckily, you don’t always have to know the answer to. Usually, the Product team, marketing, and perhaps Documentation folks work together to figure this out. Your role as a designer is to do three things essentially:
Understand the Unique Selling Point from the business side
Design a page that emphasizes your strengths and Unique Selling Point
Ask users what they find valuable/compelling and see if it matches your Unique Selling point.
You want to check whether users understand what’s unique or valuable about your product and if it’s the same thing the business thinks is valuable. Assuming the user represents your demographic, this may help your business with “Product-Market Fit.”
This becomes especially important if you’re working with Free and Premium options like many B2B companies are. Knowing precisely what motivates users to keep using a product can help businesses determine what they plan to charge users for and what features may be free.
More importantly, this feedback is relatively easy to get during user testing, especially during the post-test. Some of the easiest ways include asking questions like:
Have you ever used our competitors? If so, what has your experience been like with them?
Etc.
If people seem excited about specific features or find something particularly compelling, they’ll often be willing to gush about the subject.
Your job is to listen to those moments and pass along that feedback.
This brings me to the next subject: Metrics.
Adoption/Engagement/Retention: the significant UX metrics
These terms are technically three separate concepts, but I grouped them because they’re often an essential piece of what matters to the business and how you can assess the Product’s success or failure.
In most companies I’ve worked for, one of these has always been one of the top goals. It’s easy to understand why: they’re bite-sized chunks of several core ideas that most businesses find compelling.
While some designers have complained that they’re vanity metrics designed to appeal to Product teams, they’re nevertheless significant enough to be some of the core metrics around Google’s HEART framework, which defines UX metrics for your team.
More importantly, understanding these three basic ideas and what your business wants from them can help you ask the right follow-up questions to users.
Adoption: You care the most about getting users to sign up for the first time. If this is a top priority, then you care about a product's widespread adoption, such as getting a ton of first-time users, and these are often paired with incentives.
Some of the most common Adoption-focused examples might include Uber/Lyft’s free rides campaign in the past, free templates or books (if you give us your e-mail), or something valuable that you offer for free.
Engagement: You care about a user logging back in and doing things. While 1st-time user engagement can be significant, you often care more about what motivates users to return a 2nd/3rd/4th time and what they do.
This is the most common metric and what many companies care about. After all, many purchases (or other actions companies find helpful) don’t happen with 1st-time users: they happen after the user has logged in multiple times and chooses to interact with the content.
Retention: You care about keeping users who have become engaged users or even turning them into loyal, long-term customers. This tends to be true for more well-established companies or when there’s serious churn (i.e., users cancel subscriptions or fail to stay active).
Understanding these concepts can help influence less-common user interview questions, such as:
How do you envision yourself using this Product?
What do you find helpful about this Product?
Would you recommend this product to a friend?
Etc.
While these questions are less valuable from a user testing perspective, understanding what metric your team is pursuing can help with the testing process and what to design.
Strategy can help find the right design.
“At its core, all business is about making bets on human behavior.” — The Power of ‘Thick’ Data, WSJ
I will fully admit I am not a UX Strategist: I wear the strategy hat sometimes being a Senior Product Designer at a startup.
So I can’t recommend all of the does and don’ts of UX Strategy. However, I would recommend paying more attention to the pre-and post-test questions responses. These are not just standard questions that you always include: these are insights into how users think (and are motivated), which your Product team might care about a lot.
This can influence how you iterate and improve your designs. Let’s face it: sometimes you’re given a list of requirements from Product, and you’re forced to visualize something without any clear ideas of the larger picture.
These questions can help you understand what to design and what users need from your Product. So, while you may not work on the product strategy that much, that doesn’t mean you don’t have much to gain from understanding your user’s motivations through user testing questions.
So, the next time you analyze a user test, focus on the pre-and post-test questions. There are tons of valuable insights that your product team wants from it.
Kai Wong is a Senior Product Designer, Data-Informed Design Author, and Top Design Writer on Medium. His new free book, The Resilient UX Professional, provides real-world advice to get your first UX job and advance your UX career.