February 13th, 2020
The Art of Moderating Research
Whether or not you’re a practiced UX Researcher, we can all agree that the role of the researcher in moderated sessions is to provide a natural experience to uncover insights, with the aim to improve or at least understand real-world usability.
Moderating research is kind of like orchestrating a ballet. It requires grace and other artful skill sets like purposeful timing and becoming the script, whilst remaining open and flexible.
One of our research experts, Kacie Wise, compares moderating research to a good improv set because there is a bit of magic involved in making it a seemingly effortless journey.
“There’s a beginning, middle, and an end,” Wise says. “To maintain a natural fluidity in the conversation, you adjust the line of questions based on the responses from your participant. It’s a back and forth, but with structure because success means connecting with the participant, but also accomplishing your research goal. The end should call on key moments from the conversation and tie back to the beginning objective.”
Facilitating a seamless conversation while factoring in a hidden structure is simple, right?! For some, yes. But for the rest of us, we’ve found that great research happens when the researcher applies a few principles and uses a guiding script.
Asking powerful questions
How we ask questions as researchers is key. Using questions that are open-ended in nature is the most important aspect of this principle. It’s difficult to extract insights of any form from yes or no responses. Open-ended questions typically elicit more detailed answers that the researcher can draw further questions out of. Because the point is to evoke what the user is thinking and feeling, asking the user to describe their own account of their internal processing will lead to a more detailed understanding.
Sitting through the struggle
Good researchers are comfortable with the awkward silence (or at least have nailed their poker face) and lean away from filling or fixing. Great insights come from the challenges that the user is experiencing but researchers don’t want to amplify discomfort either. They remain neutral, and they can leave it to their observing colleagues to do the cringing and squirming.
It’s often a struggle to let the conversation flow without being too tangential. It can be tricky to balance the desire to have a session feel natural and explorative, while striving to stick to the point. There’s an innate struggle when you want to “table something” or course correct, but do so while building trust. The best way we’ve found to do so, is to show up as your authentic self to help build rapport. Even thousands of miles away on a video call, participants can feel when moderators are genuine.
If you can manage to stay in an authentic state, you’ll get more honesty. That non-verbal wince, those conflictual pauses, etc. That’s when you can drill in. See if they understood your question or if they are distracted. If the trust isn’t there you could be missing out on some key involuntary responses that are actually really telling.
Including the struggle of getting the wrong participant
There will be some users that simply haven’t participated in research before. There will be some that passed the screener but may not have answered the questions honestly and may have misrepresented themselves. There will be times when their depth of knowledge isn’t great or they try to take over as the creative director. These things happen. Treat the person with respect and dignity. If you’re deep into an interview and the participant feels they aren’t supposed to be there, still acknowledge that you appreciate their feedback. Figure out some key questions you want to ask them, validate their responses, respectfully thank them for their time, and end early. A challenging session can be frustrating and take up precious time, but ending early will give you a chance to grab a quick snack, or walk around the block.
It gets real
You start off as strangers, but end up feeling a sense of connection at the end of a session; it’s a good idea to set it up in a way that makes participants feel safe. When you make it about them by using phrasing such as, “We want your perspective” to honor them as the expert, they will show up at their best.
You may hear their kids cry, see them pet their cats or they may apologize for looking disheveled. It can be a real glimpse into a person beyond the answers.
Beginning the session with general icebreaking questions and light banter can help both you and your participant relax and ease into the session.
It’s not prescriptive
Even though we recommend a script or outline, you don’t have to get through all the questions. The intent of the conversation is the overall guide. Conversations go in paths, but how you sequence your ideas can be more powerful than the script itself.
Beware of spoilers
It’s easy to slip up and accidentally provide clues to the participant by describing the exact steps they need to take to complete a task, or by using the same language in the UI.
Leading questions are inquiries that inherently direct the participant towards an answer and they can inject bias or assumption. Even the most subtle prompts can prevent the participant from fully explaining themselves and skew the result. As our research expert Tabanitha McDaniel puts it, “If you notice a participant is taking a long time to find a button but they’re eventually successful, avoid any assumptions in your follow-up question.” For example she wouldn’t want to say, “I noticed you had a hard time finding X.” Maybe they didn’t have a hard time finding it, and they were thinking about the button copy or something nearby the UI element. A better question is, “What was easy or difficult about X?”
If a user is stuck and asking for help, remind them that you’re interested in seeing how they’d use the tool (product/interface) on their own, as if you weren’t there. It’s okay to directly ask, “What would you do if I weren’t here?” If they say something like, “I don’t know what to do… should I click this button?” And they look to you for help, responding with, “What would you expect to happen if you did?” can lead to some great insights!
Avoiding confirmation bias
“I’ve experienced participants’ desire to please even when compensation isn’t involved,” McDaniel says. Participants want to help, they want to get it right, and they never want you to feel bad about whatever it is you’re showing them. To get past their natural desire to be good participants and lean in the direction they think you want them to go, put on your friendly-but-neutral poker face. Also, avoid using language in the UI and being unnecessarily prescriptive in your questions/comments. It’s always a good idea to get a second set of eyes on your script. Recruit a colleague to help you scrub your questions of any spoilers or bias.
It is natural for the participant to feel like they are making mistakes while testing. Our researchers find it helpful to use signaling language that removes any wrongdoing. Wise likes to support her participants by adding “every answer is the right answer, so no pressure.” And if she thinks the answer is scripted, they’re holding back, or it’s just not clear, she’ll say, something to the effect of “interesting, can you explain a bit more?” to help tease out a more descriptive response. Asking for more clarity to make sure their intended statement is understood shows that their feedback is valued and can lead to some great insights!
We’ll be publishing some more practical tips next week, including some mentioned above — stay tuned!