Understanding remote usability testing and how to conduct it

This chapter looks at the remote approach to usability testing. Learn how to run effective remote tests and get actionable insights to make informed decisions.

A couple of decades ago, every usability test involved a consultant inviting a participant to complete tasks on a clunky desktop computer, then observing and noting their behavior. There was no such thing as 'in-person' or 'remote' usability tests, as everyone had to be in the same room to do any kind of UX research.

But since then, a steady flow of technological innovations and new testing tools has made remote usability tests a viable option. Nowadays, it's not uncommon for remote testing to be the preferable route to take.

Let's look at what remote usability testing is and how to best use this UX research method for improving user experience.

Remote usability testing is any testing that happens when the participant and the researcher are in separate locations. Remote testing has increased in practice with the advancement of technological innovation and is facilitated by online tools. The sessions are normally carried out through a usability testing platform such as Maze that records people completing the test, collects data, and generates insights that you can put into action right away.

Taking the remote approach to usability testing comes with some big pros:

Remote tests are easy to arrange as participants can take them whenever and wherever they want. You could share a link to a test on social media or in an email campaign and get actionable feedback on your design in minutes. This is a massive benefit over in-person testing, as you can test a lot more people in a lot less time—and with way less effort.

Remote user testing also works well with quantitative testing because it allows you to get data from a large sample size of test participants. You can measure the performance of your design or track usability metrics easily as the data can be recorded automatically, without the hassle of in-person sessions.

There’s also an argument that people behave more real when taking a test in their natural environment—so your insights will be based on a scenario that’s closer to real-life.

One of the drawbacks of remote usability tests is that you have less control over the test environment and procedure overall. For example, when running remote unmoderated testing, if a task is unclear, you can’t be there to explain it in more detail and move them in the right direction.

To minimize these risks, write some introductory context and guidelines at the start of the test, so participants know what to expect. If you’re doing a moderated remote test, inform the participant of the session's goal, and make sure they have every bit of information necessary to understand the context.

Finally, it’s crucial to conduct a pilot for unmoderated remote tests to make sure the way you wrote the tasks is crystal clear. Remote usability testing isn’t less work than in-person testing, it often requires more planning and more communication. So, plan accordingly and make sure the tools and devices you’ll be using are working properly.

The short answer is any time during the design process. Remote tests are faster and easier to conduct than in-person tests, so you don’t need to think as hard about whether they’re a worthwhile investment.

Once you start working on your design, conducting short, sharp remote tests to validate specific design features ensures your iterations are always backed by data. And since they’re easy to create and distribute, unmoderated tests are particularly useful for frequent testing if you’re short on time or money.

For the design team at Tiendanube, switching to unmoderated testing with Maze cut their testing process from weeks to days:

By sharing a link to their test in user testing WhatsApp groups, they can test their designs with users around the world—and get results fast to support their design process.

To really see the benefits of remote usability testing, you have to set up your test in a way that’ll get actionable results. Remote testing doesn't mean less preparation or planning. On the contrary, it requires setting up the test in the best way to get effective results that will help you learn and act on the data collected. Here’s our top tips for making a test that gets the data you need—fast.

There are two types of remote usability testing: moderated and unmoderated.

Moderated tests work much the same as traditional in-person usability tests, except the moderator and the user aren’t in the same room. Instead, the moderator observes the user on a video call using Zoom or similar software.

The advantage here is that you can still ask the user as many follow-up questions as you want, which means you can potentially get more varied answers from each session. Also, the user doesn’t need to trek to your office.

The downside is that it still requires a level of commitment on the user’s end—they have to do the test at a certain time with a specific setup, and the extra questions make it a more time-consuming experience all round. So you’ll probably get fewer takers, and they all need to be in your timezone. Unless conducting user tests at 3 am is your thing.

Unmoderated tests don’t have any back-and-forth between you and the user during the test. You create the tasks and write the questions in a usability testing platform like Maze, send them to users, and then they complete the test alone. The platform feeds the results back to you when they finish.

The good thing about unmoderated tests is that they only take a few minutes for users to finish, so it’s way less hassle for them. This potentially means a lot more people completing the test. You also don’t need to schedule or attend the test yourself, so more tests can be completed in less time. All this adds up to faster results for your team with less effort.

But since no one watches over unmoderated usability tests, you can’t be there to make sure that users reach the end—or that they don’t wander off for a Twix halfway through. So choosing the right scope for your test is key.

The scope of your test can make or break its chances of success. Obviously, you’ll want to test every inch of your product at some point. But with remote testing, it’s better to test specific flows than to throw everything into one mega test. There are a couple of reasons for this.

First, focusing your test on a few hypotheses will make your results much clearer one way or the other. The more designs you try to test at once, the longer your test—and the more blurry your results. Giving people fewer options lets you pinpoint design decisions and test them more rigorously.

Second, the shorter the test, the more likely the user is to finish it. This is especially important for unmoderated tests, as you can’t be there to guide their progress. And if they don’t make it to the end, you get distorted results. We recommend seven or eight tasks for unmoderated remote usability tests. The good thing is that distributing them is as easy as sending a link, so you can run small tests more frequently.

Moderated tests can be a little more complex, as you can have full conversations about what people are finding difficult and make notes on how they get stuck. And since the tests take more work to schedule, you might want to dig a little deeper to make the most of each one. Still, you should always keep tests on the short side to respect people’s time.

Or even better, you already started looking. Because while it’s definitely easier to find people willing to take remote usability tests than in-person ones, it still takes time, especially if you need users with a particular background or job title. Remember that one of the main benefits of remote usability testing is being able to test with large sample sizes. So the more users you can find, the better.

The earlier you find the right target audience, the earlier you can start testing in the design process. This could save a lot of pain undoing your hard work later down the line, as your product will be user-centric right down to its foundations.

Here’s a few places to start your search:

Whatever method you try, start building a pool of users ready for testing as soon as you need them. If you leave it to the last minute, finding people can become a painful blocker in the usability testing process.

Clarity is super important when you create tasks for a remote usability test. Well-written and structured usability tasks will get you more accurate results.

Here’s our top tips for task creation:

Questions are a way to get more feedback out of your remote usability test. Even if you’ve gone for unmoderated testing, usability testing software like Maze lets you ask research questions before and after each task and at the end of the test.

Follow up with participants after each task to get people’s opinion on specific design elements, or ask general questions at the end of the test to get qualitative feedback.

Since you’ll need to write the questions in advance for a remote test, they need to be word-perfect. Here’s a few pointers:

And here’s a few examples of well-written usability test questions:

If you want some more inspiration, you can take a look at the questions on the System Usability Scale (SUS). It’s a tried-and-tested usability survey that’s frequently used to measure product usability. The SUS even gives your product a usability score at the end.

You can also ask demographic questions to segment your users by age, occupation, education, etc. This is useful for spotting usability trends for different groups of people, but keep in mind that asking people personal questions can make them feel awkward. So ask carefully, and make demographic questions optional—or you could risk people bouncing before they start.

Great researchers know that you have to test everything. And that includes testing your usability test. The last thing you want is to send your remote test to 100 people, then realize there’s a task-destroying typo in the first sentence.

So share a pilot test with colleagues and get their feedback on what could be improved. Get people from different teams to try it out, as copywriters will see it with different eyes to the customer success team. Colleagues outside your own team will also experience it totally fresh, so their perspective will be closer to your users’.

If you're running a moderated test session, make sure your internet connection works well and your setup is ready. You have probably already prepared everything in the user testing tool of choice, but also make sure other tools like video conferencing or note-taking apps are set up properly before you start the testing session.

If you're running an unmoderated test, send it in batches—not all at once. There’s a good chance you’ll realize something is off after the first batch or two. This way, you’ll be able to fix it and avoid any major embarrassment.