To moderate user research debriefs effectively, understand who your notetakers are
Understanding the difference between 3 types of notetakers
I had a rude awakening the first time I moderated a UX debrief.
I had been a notetaker several times for user interviews, and each debrief afterward had run smoothly. But my first moderated debrief resulted in one person dominating the conversation and running out of time to talk about key issues. I didn’t know what exactly I did wrong until I spent the time to understand what a debrief is supposed to be.
It was only then that I knew that I needed to do two things. The first was that I needed to prepare more. And the second was that I needed to answer a simple question:
What’s the purpose of a debrief?
The difference between research finding presentation and debriefs
Despite what actually happens, teams schedule meetings to make progress through decisions. We schedule meetings to decide how to move a project forward, user concerns, or outstanding issues.
I bring this up to mention that research findings presentations aim to introduce key findings from user research. The team can then begin making decisions on integrating them into Agile sprints or other business processes. I thought that the debrief had the same purpose on a smaller scale. You would decide how to incorporate individual research findings into the research findings. But there’s one key difference: you never want to make decisions during a debrief.
The interview is still fresh in everyone’s minds, which means that no one has had time to process and analyze anything just yet. Therefore, we don’t want to try and decisively answer questions or solve problems at this stage.
Instead, the goal of a debrief is to help your group explore what happened during the research session and what it might mean.
This exploration might include:
Hearing multiple perspectives on the interview,
Key findings each person took away
What they found most important.
But to do that, you need to prepare thoroughly and have a plan. The reason for this is because you want to hear from every notetaker, rather than just the strong voices. Remember, you’re not deciding anything at this point, so you need to make sure louder voices don’t push towards making decisions yet.
Instead, you want to hear what each notetaker observed and took away from a session. This is so that you can reach a consensus on what observations to incorporate into your core findings. To do this, there are two types of debriefs that you can use: One is a simple conversation, and one uses Post-its and a whiteboard. Each of these methods can be effective, but we should use them in different scenarios. But choosing a method is based on a familiar question: who are your notetakers?
Who are your notetakers?
A good piece of UX advice can complicate your debrief, which was one of the primary reasons that I struggled. One of the easiest ways to get stakeholders to see the value of UX research is to invite them to watch (or notetake) a user research session. While that’s true, if you invite them, you often need to provide additional instructions about what they should do to ensure it’s effective.
I’ve experienced a wide range of notetakers, ranging from excellent to those that have written 4 sentences for a 1-hour user interview. So it’s not enough to assume that they know what to capture in the notes.
The debrief is the perfect point to find this out, rather than several days later when it’s difficult to remember things. Debriefs are also great at clearing up misunderstandings or finding out different perspectives. So consider who you’re notetakers are and think about how you will plan things based on that.
Planning a debrief with different types of notetakers
Here are 3 different types of notetakers and how you might think to prepare for them.
UX team members familiar with the project:
Sometimes, you’re working with members of your UX team who have spent a lot of time working with you on this project. You’ve helped develop the facilitator guide, the tasks, and the usability test plan together. As a result, these tend to be the easiest notetaking sessions to plan.
Oftentimes, the notetakers are aware of the tasks, what to pay attention to, and the like. But you still might want to establish some categories and spend a set amount of time talking about them to make sure you get to everything. While it’s possible to do a Whiteboard debrief, it’s often easier to make a Moderated conversation as that’s less formal.
What to plan on doing with these notetakers:
UX Designers not familiar with the project, SME’s with no notetaking experience:
The second category of notetakers requires a bit more planning because they’re not quite as immersed. Sometimes, you don’t have enough notetakers on your team and can recruit other UX professionals working on different projects. Or, you’ll be able to get people who are immensely familiar with the process or project, such as Subject Matter Experts, to notetaker for you.
In either case, they may know part of the equation (either the UX research or the project material) but may not know everything. But the notes they take can provide a unique perspective and often yield many insights if planned well.
As a result, you may need to give a little bit of context to the project, the tasks, and the purpose of the notetaking procedure.
What this entails is catching them up to speed first before jumping right into the notetaking process. This means making sure that they understand what you are hoping to learn from this interview and what you might want them to take note of. This means you have to do a little bit more preparation to get them up to speed. This also means that a Whiteboard debrief may be more useful than a Moderated conversation. Assigning categories (as well as having a parking lot to table discussions) can assist during the debrief.
What to plan on doing with these notetakers:
Send your notetakers an e-mail explaining the steps, itinerary, and more. This is particularly helpful when you are doing field research.
Figure out 3–4 topics that you want to cover (such as pain points, surprises, questions, quotes, or takeaways) and set up timeboxes for them
Tell your notetakers exactly what to do (i.e., “Note down the exact words the user said” or “Note Down what actually happened and where”)
Stakeholders working on a project:
The last group of stakeholders is those I’d call those you invite for a purpose beyond just taking notes. Perhaps they don’t understand what UX does, how users do things differently than what they might think, and other things like that. While the notes you might receive from them might be a unique point of view, you typically want to capture their insights and experience. So, as a result, we want to get their opinions, viewpoints, and other insights into the process. But to do this, we need to do a little more handholding to understand what’s going on and provide insight into what they observe. So we might consider many different things how we can best leverage their input.
What to plan on doing with these notetakers:
Send your notetakers an e-mail explaining the steps, itinerary, and more. This is particularly helpful when you are doing field research.
Tell your notetakers exactly what to do (i.e., “Note down the exact words the user said” or “Note Down what actually happened and where”)
Tell them to watch for a single thing, like Emotion (i.e., “Look for when the participant gets emotional. What was the emotion and why?)
Figure out 3–4 topics that you want to cover (such as pain points, surprises, questions, quotes, or takeaways) and set up timeboxes for them
Tell them what NOT to do (“Don’t interpret what you think is happening, just write down what actually happened.” or “Don’t come up with solutions right now…”)
Debriefing effectively
There’s been more than once where a debrief helped to save our user interview sessions. After interviewing a surgeon at 4 AM, almost all of our notetakers took a nap on the hour-long drive back to the lab (while I drove). If it wasn’t for our mentor pressing us during the debrief soon after we got back, I’m pretty sure we would have lost some of the insights we worked so hard for. And those questions, as everyone was exhausted, nevertheless formed the basis for a new research question and study.
While your debriefs may not save the day, it’s a powerful tool you can use to understand what your notetakers observed during user interviews. You have to understand that you’re not making decisions quite yet.
Kai Wong is a UX Designer, Author, and Data Visualization advocate. His latest book, Data Persuasion, talks about learning Data Visualization from a Designer’s perspective and how UX can benefit Data Visualization.