The 5 most common arguments against user testing
The 5 most common arguments against user testing
How to argue for user testing when budgets are tight
I've been working in UX for over a decade and have heard about every excuse to try and avoid user testing.
From misunderstandings to outright rejecting the concept, I feel like I've heard every argument posed by the rest of the team.
Unfortunately, with the economy resulting in many organizations having tight budgets, we're once again in a time where it's easy for user testing to fall by the wayside.
However, many of the arguments I've heard against user testing tend to fall into five general categories, and throughout my career, I've learned how to argue against each of them.
If your team is trying to justify not user testing, here are some strategies to deal with their arguments.
The ugly truth around user testing
I have never held the job title UX researcher, but I am a designer with enough graduate school training to lead basic user research.
In some ways, this is part of the problem that I often face. I'm taking on responsibility beyond my job description and advocating for things businesses aren't hiring me for. In addition, most places where I worked didn't have dedicated UX researchers either.
The ugly truth is, unfortunately, many businesses are saying, "We can't afford user testing right now." With tight budgets, many companies are cutting back on user research, and user testing is sometimes collateral damage.
So it's unsurprising that my efforts to help lead user testing sometimes fall on deaf ears. This is usually because of 3 particular reasons:
It's not one of my 'written' core job responsibilities
It was being outsourced, so it's probably expensive
It's unfamiliar, so the organization doesn't know how it works
However, the truth is that we, as Designers, can only design what users need with user testing.
Whether we call it "Customer feedback" or "user validation," formal or hugely informal, someone must ensure that proper user research is done. It often falls to the designers if no other team member can do user testing.
So here are the five most common arguments I've heard against user testing and strategies to deal with each one.
Understand your Product Manager and their arguments
It's often rare to get a whole team on board with an idea, and as a result, we want to ensure we're spending time convincing the right person to get user research approved.
Most often, this will be the Product Manager (although sometimes it can be the executive team or someone higher up). This is because the three factors most common to user testing blockages are things that they often tackle:
Time (Fitting user testing into sprints and managing work hours)
Resources (Setting budgets and resources)
Access (Getting access to specific users, going through approvals, etc.)
As a result, they're the ones I often try to convince. I've heard all manner of arguments for why we can't do user testing, but arguments tend to fall into one of 5 different categories:
We can get 'user testing' data from somewhere else, like analytics
This is nice to have, but building and shipping a product takes priority
We don't have the resources to hire a user researcher or pay for user research/testing
User testing doesn't seem to offer a ton of value/bang for the buck
Fundamentally, we can't give you access to users (due to privacy laws, etc.)
This is not a comprehensive list of all the excuses you'll likely encounter. However, the product manager will often stick to one of these arguments.
Here's how you can respond to each argument.
Walkthrough analytics (or other data) to show gaps in knowledge
If you haven't been around that long, you should know that Analytics was the popular buzzword before Machine Learning or AI.
The idea was that business executives would see a dashboard full of numbers and data visualizations and be able to understand everything they needed to know from the customer. Some even believed all customer feedback could be boiled down to one number: Net Promoter Score.
That didn't happen, but many people still believe that 'data' can tell us everything we need to about the customer. They don't want to understand the technical details of how the numbers got there, just some 'value' they can act on.
What I've often found to be helpful is to use that to your advantage. If you're working with a crowd in love with analytics, go through some analytics visuals and explain what we don't know.
For example, if there's data around Monthly Active Users and it dips during summer, show the graph and bring up that we don't know why there's a dip during summer.
Please don't put the analytics people on the spot, though: instead, talk about why user testing is essential to understand our knowledge gaps.
One way of doing this is by framing arguments around change. Businesses often re-design products or features and change one particular Key Performance Indicator (KPI) to be better. For example, they want to improve user engagement by 15%.
In that case, there are 'easy wins' to be found through user testing by seeing where people are getting tripped up and fixing those usability issues.
The knowledge gap, and the purpose of the user test, is a simple question: Where are users running into problems in the workflow? By showing how user testing can address that question, you can persuade your team that you have gaps in knowledge and know how to address them.
However, there are other arguments you'll run into.
Research isn't just nice to have, but can save time and money
The most valid-sounding argument that many teams bring up is that while user research is nice to have, there are other priorities.
In some ways, this is true. If you're in a startup, and your ability to keep the lights on involves releasing a product, then yes, research isn't the top priority. I've often encountered both Product and Engineering teams pushing back against research because they're trying to get a product to work out the door.
After all, while you're going to seek out, user test, and provide recommendations to your team, it's yet another thing they'll have to tackle right before release.
What I've often found helpful, especially with stressed Engineering and Product teams, is categorizing user testing as something to do during 'Early Access' or 'beta testing.'
For some large companies, this is much too late, as when something is released to the public, it might be released to millions of users. However, many startups and smaller companies often release their product in phases.
Their first version is the Minimal Viable Product (MVP), and it can often be missing key features. In addition, the teams often know there will be bugs they'll need to address with the next one. Lastly, getting access to users before something releases can take time and effort.
You can squeeze user testing and research into this phase and push for gathering 'more bugs' from the user's perspective instead of just technical issues.
One of the best ways to start this process is to schedule a walkthrough of the entire process with employees that haven't seen the product before. Having people in your team walking through the entire process and seeing 'user bugs' (i.e., usability issues) instead of technical bugs often shows that there's a need to seek out people to test with.
To do this, however, you need to prepare user testing tasks, research questions, or an interview script while everything is ready for the first release. That way, all you will need is access to your users, and you can hit the ground running.
Preparing the user testing plan yourself also helps with one of the other primary reasons people reject user testing: they need to understand how low-resource it can be.
You don't need a ton of resources to do basic user testing
One trend I've noticed become increasingly common is businesses outsourcing user research. Whether it's enormous groups like UserTesting or UserZoom, debunked ideas like user-less AI research, or more, some companies seem reluctant to do user research themselves.
I suspect that it comes down to money: they've seen the high salaries that UX Researchers can warrant, and it's cheaper for them to contract an organization to do that work.
Am I saying UX Researchers are not worth the money? Absolutely not: I know some fantastic UX Researchers worth every penny. However, some companies have the perspective that user research can be expensive and might not have the budget for it.
However, this is where DIY user testing can be helpful. Steve Krug, in his book Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems, highlights just how little resources it can take:
Scheduling testing in the morning once a month
Getting a hold of a few participants and asking questions
Having a quick turn-around of the basic user findings by the afternoon
This is always one of the selling points of user testing: it's cheap, relatively quick, and you can get a lot of excellent customer feedback and data while avoiding catastrophic mistakes. These are things that most UX Designers know (including the '5 users' for testing), yet your team might not.
After seeing product demos from fancy user testing companies, some teams might conclude that they need to find an extra $25,000 somewhere to do user testing (so they'll tackle it later).
You can do basic DIY usability testing for $250, which yields a lot of value. Will it give you all the insights a professional company or UX Researcher will provide? Of course not.
In addition, you'd probably need to tackle some of the tedious parts yourself, such as:
Figuring out how to recruit users and where you can find them
Approaching people and asking them to be part of a study
Clearing 'talking with users' with any legal/organizational committees
etc.
However, clearing these hurdles makes it significantly easier to do these things for subsequent user tests.
Showing how low-resource user testing can also help partially resolve another reason: the perceived value of user testing.
User Testing provides value by connecting "What people say" and "What people do"
This reason isn't the sole reason why your team might reject user research, but it's often combined with another reason (usually the first).
If businesses can see what users do through analytics, and they can see what users are saying through feedback like the Net Promoter Score (NPS) or customer support tickets, what value does user research and testing offer?
The main reason is subtle but nevertheless important: it connects what users say and what they do. A user test is a way of observing what users are doing and asking them about it in real-time. That's a method that no other data can provide.
For example, what does it mean if someone goes from a "7/10" NPS score to an "8/10"? What improved the user's experience? That's something that survey data alone can't tell you.
This is not even addressing more complicated topics, such as bias. Customer support tickets, for example, are heavily biased towards the negative: you don't often get people calling in to say your product was okay (or even great).
Asking, "Well, how are we going to hear from people that like our product?" can get the team to focus on the value of user research.
Focusing on this disconnect can often be one of the strongest arguments for user testing if your business is focused on user engagement and retention. If it seems confusing to your business why customers say your product is excellent, but analytics shows very few people are using it, this is how you can argue for user testing.
Because we're not just listening to what users say: or listening to what users do.
Figure out if you have no access to users or is it a misunderstanding
Lastly, there is sometimes a valid argument that seems like a dead end for user testing.
In one case, I was told I'd probably break several federal privacy laws by seeking out patients to talk to when I worked for a federal cancer organization.
However, one thing you want to do when encountering this "No" is figure out if this is a hard or soft "No." I don't mean to act like a pushy salesman: I mean ask your Product Manager why this is the case.
In another example, after digging around a bit further, I found out that my Product Manager felt it would be too intrusive to watch users doing their jobs and that not all of our users would agree.
When I told him I didn't need access to a hundred percent of their users (only a couple) and that I didn't need to bug them for multiple hours, my Product Manager became more receptive to user testing.
So if you encounter a "No," always try to figure out why. It may turn out to be a misunderstanding. Even if it's a hard "No," there are ways to design when you cannot access users.
User testing is also a designer's responsibility
I've been a designer, not a UX researcher, for over a decade. I don't know much about user research besides what I learned in graduate school and my time applying user testing and user research methods to my work.
However, right now, I know that many UX Researchers are facing layoffs, and user research at some organizations can be quite dire. Many companies currently can't afford to invest in user research, and I've found that if I don't bring up user testing, no one will.
Because I'm the person that brings up this idea, when budgets are tight and time is precious, it's no surprise that I've run into resistance. But even though I don't do UX research for a living, I understand the crucial importance of it, and so should you.
And one of the best things you can do during this lean period is show the value of user testing and DIY user research. By implementing it into your workflow, you may not just ensure that your product is usable by your audience: you may demonstrate the value of user research to your organization.
That, in turn, might push them towards digging up that extra money to hire a full-time UX Researcher. So if you've been seeing user testing falling to the wayside in these lean times, use these five strategies to counter some of the most common arguments about user testing.
Kai Wong is a Senior Product Designer, Data-Informed Design Author, and Top Design Writer on Medium. His new free book, The Resilient UX Professional, provides real-world advice to get your first UX job and advance your UX career.