Moderated vs. unmoderated usability testing: Which one is right for your product build?

There's no right or wrong decision between moderated and unmoderated usability testing. Both UX research methods are highly successful in gaining crucial user experience insights to help you build more user-centric products.

However, both moderated and unmoderated usability testing methods come with their own pros and cons. By the end of this read, you'll be able to identify which method is best for your product team, workflow, and budget.

Moderated usability testing is human-to-human, real-time testing. For this type of testing, a moderator (also called the facilitator) guides participants through the test, and is available throughout.

A moderated usability test enables moderators to record observations, ask or answer questions, and ask follow-up questions.

For example, during moderated user testing, a facilitator can deep-dive into unplanned topics that emerge in-the-moment, and follow organic conversational threads to uncover unexpected insights. Whereas unmoderated user testing only allows testers to answer the questions your product team provides in advance (although you can follow-up retrospectively).

Today, moderated tests can be done remotely or in person, as long as the moderator is able to read participants’ body language and facial expressions.

Moderated user testing has been thought to limit the reach of diverse test participants, as product teams needed to rely on physical proximity or travel to implement their tests. The other limitation is that post-test analysis can be laborious—requiring researchers to manually shift through recorded interviews to find results.

However, today, video conferencing tools like Google and Zoom allow researchers to connect with participants around the world. On the analysis front, remote UX research tools are using AI to advance and streamline moderated test analysis.

For example, Maze’s AI-powered Interview Studies enables researchers to mass-analyze moderated interview recordings, enabling teams to quantify qualitative feedback, summarize interviews, and surface critical insights faster than ever before.

A moderated session’s benefits include:

Some of the downsides of moderated usability tests are:

Maze’s Product Researcher, Matthieu Dixte, shares four ways product teams can run smoother moderated user tests from the beginning.

Changing just one word can influence a user's interpretation of a question or task instruction. Therefore, it’s important to formulate the questions in the same way for each respondent to ensure you’re comparing the same behaviors and learnings.

Keep your initial questions identical between participants—if follow-up questions emerge, consider asking these across the board, too.

During the natural flow of a test, you should adopt a neutral and silent posture. It’s important to let the person complete their task, to search, and struggle—that's where you’ll gain the most valuable insights.

Matthieu explains: "Don’t react to the actions performed. Whether the person easily completes a task or struggles with it, do not show any hints or emotions."

Instead, be curious and eager to understand their behavior: “Observe and give space to the participant. If you help them by giving hints, it's like taking the mouse for them and completing the task yourself.

“If the person feels uncomfortable, remind them that there are no right or wrong answers, no pressure, and that you’re not judging them.”

When asking follow-up questions, be cautious about cognitive biases that might slip in. As with any best practice in exploratory interviews, it’s important to remain neutral and avoid leading questions.

Instead, promote rephrasing and introspection on the respondent's side, rather than suggesting your interpretations. This means letting them frame and vocalize the question on their own, rather than interpreting their struggles and framing their answer for them.

Moderated user interviews can be heavily subjective when noting behavioral trends around body language and facial expressions. What one moderator considers worth noting down, another may not even notice.

What’s more, two individuals’ expressions of the same emotion can differ drastically—be that due to cultural differences, different neurotypes or ages. This impacts both how a participant expresses an opinion, and how a moderator interprets it.

It can be a challenge to avoid accidentally influencing, misinterpreting, or creating bias in the test. Work to create a thorough testing guide (for both moderators and participants) that covers best practices, UX research ethics, and ways to minimize the impact of cognitive biases.

Moderated usability tests lend themselves to a time in the product development process when more explanation and guidance may be needed, from both the researcher about the prototype, and the participant about their answers. For example, during wireframe testing, when a candidate may need live support to complete tasks, or the limited functionality of the prototype may need further explanation.

This means moderated testing naturally slots into the beginning of a design process—when qualitative insights about opinions and overall feel of product can help guide the later functionality and specific UX.

Using moderated testing early on gives UX teams time to return to the participant throughout the build, dive deeper on their responses, and show them renditions after feedback.

This promotes continuous product discovery, providing design teams with feedback from ideal customer profiles and allowing them to check in with participants to see if they’ve delivered on feedback in each product rendition.

Moderated testing can also help UX teams:

Unmoderated usability testing is, unsurprisingly, when the test doesn’t involve a moderator. Users complete the task independently, often using usability testing tools that record their decisions and actions.

This type of test process is self-paced and the test taker is left to complete the tasks and answer questions to the best of their ability—they cannot ask for help. Testers are usually in their own space.

Unmoderated, remote usability testing is commonly used for high-fidelity prototypes that need little-to-no explanations. For example, in the final stages of design when you have a production-ready, interactive app that’s lacking the final tweaks before launch.

Unmoderated usability testing is also used when product teams:

The pros of unmoderated usability testing methods include:

Some of the cons to using unmoderated testing sessions are:

An unmoderated user test also has to be air-tight, leaving no room for questions. This could mean multiple renditions until testers are totally clear on what’s expected of them.

Research tool features like Maze Clips are combating some traditional cons of unmoderated usability testing by enabling researchers to capture short audio, video, and screen recordings from asynchronous prototype and website tests.

This means Clips can help to bring some moderated user testing benefits to unmoderated remote tests, such as enabling product teams to gather qualitative behavioral insights or understand unclear responses and slow interactions by tracking their activities.

Unmoderated usability tests are incredibly valuable at any point in the design and development process. They’re particularly useful when you need a large amount of data, or want to hear from diverse participants.

Unmoderated usability testing is also best used if your team is on a tight budget, or has limited time and resources to dedicate to the task—as you can set up a usability test and leave it to capture responses in the background, while you focus on the next task at hand.

It’s a good fit for any straightforward UX processes where no handholding is required, so is often used further along the design process when you have high-fidelity prototypes to test.

Unmoderated usability tests can also be used to help design teams:

Despite unmoderated usability testing typically coming towards the end of a design process when reviewing final designs or testing prototypes, when participants are briefed well and given an intuitive testing platform, many organizations opt for unmoderated usability tests throughout the design process as a low-cost, high-reward option.

There’s no right or wrong answer as to whether you should use moderated or unmoderated usability tests. Both moderated and unmoderated usability testing methods are valuable in any design process.

Here are some questions to keep in mind when deciding between the two:

Once you’ve got these questions figured out, you’ll be in a much better position to decide which usability testing type you need. In some cases, you may benefit from using both, as Matthieu recommends:

“I use a mix of the two methods. It's not necessarily one or the other in confrontation, but a combination of the two approaches, which are complementary to take advantage of each other.”

Thankfully, Maze has you covered no matter which method (or mix) you choose. With Maze, you can use AI-supported Interview Studies to streamline and manage moderated tests, or opt for remote unmoderated usability tests to gather high-quality data at pace.

Either way, your team can download customizable research reports, collaborate with team members, and access data that helps you understand user behavior in context.

Both solutions eradicate fears and concerns product teams typically have about each, while empowering them to run both moderated and unmoderated user tests harmoniously.

In the next chapter, we’ll cover the steps to conduct live website usability testing, alongside insights from industry experts, so you can apply the learnings to your process.