A Beginner's Guide to Usability Testing

Usability testing is a proven method to evaluate your product with real people. In this complete guide to usability testing, we share everything you need to know to run usability tests and get actionable insights to create better user experiences.

Learn by doing, start usability testing now

Run surveys, validate design ideas, and test your prototypes. Maze makes it easier to make your product truly customer-centric.

Chapter 1

What is usability testing?

Let’s start right away with a definition of usability testing. Usability testing is the process of testing your product with real people by getting them to complete a list of tasks while observing and noting their interactions. The goal of conducting usability testing is to understand if your design is usable and intuitive enough for users to accomplish their goals. 

Good usability testing takes a holistic look at the potential audience that you can serve and the range of intended and unintended uses of a product.

Behzod Sirjani, Founder of Yet Another Studio

However, the benefits of usability testing extend beyond testing functionality and usability. More than that, it helps you really grasp the use cases of your product, understand your audience, and build a good product.

Behzod Sirjani, founder of Yet Another Studio and former Head of Research & Analytics Operations at Slack, explains that "Good usability testing takes a holistic look at the potential audience that you can serve and the ranges of intended and unintended uses of a product to understand how you're enabling people to do what they want with your product while preventing misuse."

A brief history of usability

Before we dive into usability testing specifics, let's define what usability means in the context of the human-centered approach to design. Human-centered design is a creative approach to problem-solving, established and developed by IDEO, which places the user front and center in the design process.

The approach combines quantitative and qualitative UX research methods for developing products, such as ideating, rapid prototyping, the jobs-to-be-done framework, and more. Usability testing is one of those methods and a key principle for building user-centric products.

No matter how much time you spend on ideating and prototyping your design solution, you should always test it with real people. The feedback from the testing session will help you understand what part of your design requires improvements.

Nick Babich, Editor-in-chief of UX Planet

The modern approach to usability testing started in the ’80s as work and personal computers became more prevalent. At that point, most people had very limited exposure to computers, so getting familiar with new technology was usually a very long and frustrating process.

To make sure computers were usable and intuitive, programmers needed to understand users' thought processes for completing a task. In 1981, two researchers–Karl Ericsson and Herbert Simon–published an article called Verbal Reports as Data, which popularized the “think aloud” method in which the participant literally speaks their thought process out loud.

More than a decade later, the ISO 9241 definition of usability—published in 1998—defined usability as “the effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments.”

These developments meant that instead of guessing how a product is used by people, you could observe them complete tasks, learn, and improve usability problems in real-time. That’s where the usability testing process comes in. It gives you a direct look into what the participant is thinking and how they use your product.

Types of usability testing

In this guide, we explore the (often-confusing) world that is usability testing. Before we get started on specific usability testing methods, let’s set out the groundwork for the different kinds of methods and approaches. Broadly speaking, all usability testing methods fall into three categories:

  • Qualitative or quantitative
  • Moderated or unmoderated
  • Remote or in-person

Types of usability testing method

Qualitative and quantitative

All user research you do will fall into either qualitative or quantitative. There are a few key differences between qualitative and quantitative usability testing: namely, the data they deal with. Where qualitative research uses words and meaning to uncover the ‘why’, quantitative research dials in on numbers and statistics to answer ‘what’, ‘where’ and ‘when’. When it comes to usability testing, certain methods of testing may lend themselves better depending on your research goals.

Moderated and unmoderated

Deciding between moderated and unmoderated usability testing is an important step in setting up your usability test. All methods of usability testing will fall under either a moderated or unmoderated approach, and some methods may be able to function with either approach. Generally speaking, the goals of moderated versus unmoderated usability testing are the same, only the presence of a facilitator (moderator)–and sometimes the environment–changes.

Remote and in-person

Considering whether you want to embark on your usability testing with a remote or in-person approach largely depends on what you are testing. Physical products may naturally fall under an in-person usability test, whereas software or online tools will test just as accurately with a remote application. As with moderated/unmoderated testing, you may find your usability testing will work equally well with remote and in-person usability testing, in which case you must weigh up the pros and cons of both approaches.

Find out more about these usability testing types (and others) in Chapter 2.

What’s the difference between user testing and usability testing?

While usability testing and user testing both involve product designers and developers interacting with their target user, the purpose is what differentiates them.

User testing

User testing typically comes before usability testing. The goal is to determine whether future users need the product, tool or service in question. User testing can also be used by marketers to test target audiences and better understand their needs and frustrations.

Usability testing

Usability testing can happen at any and every stage of the design process by testing wireframes or even high-fidelity prototypes. Usability testing typically also happens with each iteration of the product. The aim is to review how users use the product, tool or service, and whether they can navigate it successfully to achieve their goals

Across the industry, the two terms—user testing and usability testing—are often used interchangeably. At Maze, we use 'usability testing' to refer to the practice of testing your design or product with users by getting them to complete a list of tasks. When trying to define what method you and your team should use—focus on identifying your goals, your research questions, and the stage of the process you need to test at. Our advice is to test early, often, and to always keep your target audience and customers in mind when creating experiences for them.

What usability testing is not

Usability testing is not simply gathering information or opinions about a product—that's marketing or user research. It's not even showing users a rough draft of the product and asking them if they know how to use it.

Unlike generative research—which is used to get a deeper understanding of users' goals, motivations, and behaviors—usability testing is the process of testing your product with real people by getting them to complete a list of tasks while observing and noting their interactions.

There are other UX research methods and techniques to test user experience but don't qualify as usability testing:

  • A/B Testing: this is a method of testing design variations to determine which changes improve performance. While usability testing observes and investigates user behavior, A/B testing can't tell you why a version is better than the other.
  • Surveys: you can use surveys to gauge user experience or as part of a usability test, but as they don't allow you to observe users' actions in real-time, they're not actually usability testing.
  • Focus groups: during a focus group, a moderator asks a group of people about their behaviors, preferences, and experiences on a specific topic. With focus groups, the goal is to gather opinions about a product, not to test how people use it.

The benefits of usability testing

Usability testing is important because as much as we like to believe that we are building products for ourselves, each person is unique. We’ve all had unique backgrounds, lived experiences, perspectives, preferences, and abilities. All of these things influence how we understand, approach, and experience a product.

Behzod Sirjani, Founder of Yet Another Studio

Have you ever noticed on cooking shows–when possible–chefs have others taste components of a dish while they’re still cooking? Maybe it’s a sauce or some other ingredient. Their counterpart might say, “needs more salt” or “a little too spicy.” What they’re doing is running a very small-scale user test.

When you’re too close to something–like a chef may be to a dish–it’s very easy to miss things. We tend to overestimate how likely someone else is to share in our opinion. The psychological term for the phenomenon is the false-consensus effect.

The only way to combat false-consensus effects when designing is with usability testing. This allows you to get another point of view, and most importantly, a perspective from the actual users who’ll be using your product in real life.

Usability testing is also relevant for long-term customer success. A broader perspective means you can design better for your audience, and make sure more people can be successful with your product. Forbes reports that companies investing in UX have a lower customer acquisition cost, lower support cost, and higher customer retention rates.

Usability testing is one of the most fundamental ways to test the success of a design. Ensuring customers can find, understand, and use a solution contributes to the overall success of a product and business.

Taylor Palmer, Product Design Lead at Range

To give a more concrete usability example, Shopify reports that one of the most common reasons a customer abandons a shopping cart is due to a complicated check-out process. When you employ usability testing during the design process, you can find and fix usability issues and complex flows before you launch your product, and ensure the user experience is up to standard.

Product tip ✨

Maze allows you to test your prototype with users and get actionable data that helps you make informed design decisions. Learn more in the complete guide to using Maze.

When should you do usability testing?

There are multiple times when you and your team should run usability tests during the design process. In most cases, it’s something you should do more than once, and early and often in the process to inform your design decisions.

Usability testing often happens towards the end of the product development process and is seen as a way to “catch bugs” in the product. That’s not usability testing, that’s QA. Usability testing needs to happen at a point where you can not just learn how usable your product is, but when you can take those learnings and make improvements so that your product is more usable.

Behzod Sirjani, Founder of Yet Another Studio

Here are some general guidelines you can follow when deciding when to do usability tests and creating your test plan.

Before you start designing

If you want to start a redesign, chances are there’s something about your current site that’s not working. You can run a usability test before you start any redesign to discover users’ pain points and bottlenecks with the current design. If you have that data as a jumping-off point, you will be able to benchmark your new design against the existing one to make sure it’s a better version.

Additionally, one of the best times to start testing a product is in the product-discovery phase. If you’re designing a new product from scratch, you can do usability testing with competitor products to observe how people use similar tools and services in real life and understand their mental models when using products like yours. The test results can be a benchmark for your UX design when the time comes to test it.

Once you have a wireframe or prototype

With the initial research phase completed, you can start in on your new design. It’s important to test your designs as you go through the iterative design process. You can start testing as soon as you have your initial ideas sketched in a low-fidelity prototype or even a paper mock-up.

I think usability testing is applicable at every stage of the design process. I try to test at an early concept phase when I have a low fidelity wireframe to get some early input from real users. Once we get the first results, it’s important to use those results throughout other design phases and re-test with users.

Vaida Pakulyte, UX researcher and Designer at Electrolux

This type of early testing is called formative testing, and it helps you test and validate ideas as you move through the process, so you don’t get to the end without any user feedback.

For this stage, you might want to do card sorting or tree testing to test your website or app's information architecture. Generally, if you’re starting from scratch, card sorting is recommended first to understand how your target audience groups and associates information. That, followed by a round of tree testing, ensures that the content on your website or app is easy to find, and the web pages follow an intuitive flow.

Product tip ✨

If you have a Figma, InVision, Sketch, or Marvel prototype ready, you can import it into Maze and start testing your design with users immediately. Get started for free.

Prior to the launch of the product

Once you’re at the end of the development process, you’ll probably have a fully-sketched high-fidelity prototype. That’s where summative testing comes in. Summative testing is a usability evaluation you do at the final stage to prevent any major issues and test if the end-users can complete tasks with the user interface.

When running summative testing, you should aim for a large sample size to get statistically significant results from a large pool of test participants. Usually, the test results can be benchmarked against any data you collected throughout the testing process, so ease of use and user satisfaction can also be measured before launch.

At regular intervals after launch

Just as products aren’t static, neither are user bases. Since they’ll also continue to change and evolve, it’s good to run test sessions every so often to make sure everything is still working as intended. Consider regularly checking after launch to make sure your product stays in tip-top shape, and your actual users aren't experiencing any issues.

5 best practices for usability testing

This section was written by Behzod Sirjani, founder of Yet Another Studio and former Head of Research & Analytics Operations at Slack.

There are lots of things to keep in mind when you are usability testing. That said, here are some of the most important ones:

1. Get consent

You should get consent at two separate points during usability testing from your test participants. First, at the beginning of the test and then at the end. Here's why. At the beginning of the test, you should ask for their consent to record the test and the results. Then at the end of the test, you should ask for their consent to keep the test results and use the data. It's important to ask for consent twice because a participant only has a loose idea of what they are consenting to in the beginning.

Being intentional and inclusive in your testing audience will help you build better products because you will learn from people who are different than you and who see and do things that you do not.

Behzod Sirjani, Founder of Yet Another Studio

2. Be inclusive

You should be intentional about the people you want to participate in your usability test, and you should make sure to include people who may have different perspectives on your product. This includes people of different abilities, demographics, usage behaviors, market segments, etc. Being intentional and inclusive in your testing audience will help you build better products because you will learn from people who are different than you and who see and do things that you do not.

Discover more best practices for designing products and experiences that are truly inclusive in our Inclusive Design Guide.

3. Run a pilot test

Before you ask for any of your customers or potential customers to spend their time giving you feedback, make sure to pilot your usability test with someone inside your organization, ideally someone not on your team. Among other things, this allows you to identify aspects of the study or product that are obvious to you, but that may be missing from your study experience.

4. Establish evaluation criteria

You should know what success looks like for your product testing and have clear ways to identify if and when people were (not) successful. Bonus points if you have easy ways to keep track of that in your notes.

5. Be mindful of length

Your usability test should be only as long as it needs to be for you to have confidence in your results. People do not exist to give you product feedback, and very few people want to spend all day evaluating the usability of your product. If you worry that your test is too long, pilot it and gather feedback. It’s much better to run multiple separate tests than to gather feedback from participants who are cognitively exhausted because you’re asking too much of them.

In the next chapters, we'll learn more about the different usability testing methods and about the different types of usability testing, such as in-person user testing versus remote usability testing, or unmoderated versus moderated sessions.

Learn by doing, start usability testing now

Run surveys, validate design ideas, and test your prototypes. Maze makes it easier to make your product truly customer-centric.

Frequently asked questions about usability testing

How do you perform a usability test?

If you want to run a successful usability test, you should create a usability testing script. Start by clearly outlining the goal of your research, the type and number of participants, and the specific tasks you want them to complete. If you are looking for more information or examples, check out "How to write an effective usability testing script."

What are the main characteristics of usability testing?

Five main characteristics define usability testing: the primary goal is to improve the usability of a product, the participants represent real users, the participants do real tasks, the researchers observe and record the participant's activities, the researchers analyze the findings, diagnose problems, and recommend changes.

How do you write a usability testing report?

At the end of the usability testing process, you should create a final report to share your results with the rest of the team. The report should be easy-to-understand and well structured, helping you and your team to see what works well, prioritize what needs to be fixed, and plan the next steps. Learn more about how to analyze and report usability test results in the final chapter of this guide.

What is usability testing?

Usability testing is the process of testing your product with real people by getting them to complete a list of tasks while observing and noting their interactions. The goal of usability testing is to determine whether your design is usable enough for users to navigate and achieve their goals.

When should you do usability testing?

The best time to conduct usability testing is before finalizing or launching the product. However, there are several opportunities to do usability testing earlier in the design and development process, and ideally you should conduct multiple usability tests at different stages. These stages include:

  • Before any major design decisions are made
  • When it’s time for the product’s next iteration
  • After launching the product
  • When you’re uncertain about the design

What is an example of usability?

There are several famous examples of usability testing. Some common markers of usability testing may be a user being able to locate an item, add it to a basket then navigate to checkout, or whether users are able to turn a product on and off without assistance.

What measures of usability are there?

Measures of usability, or rather– evidence your product is usable–include beingintuitive (the product is easily navigable and easy to learn how to use), **cause and effect **(the product does what users expect of it and elements respond how they expect), and error clarity (errors are clearly communicated, and present in relation to their magnitude).

What is the difference between usability testing and user testing?

The difference between usability testing and user testing lies in their goal: user testing determines whether users need the product in question, whereas usability testing determines how easily the user can use the product. However, in the industry the two terms are broadly used interchangeably.

What are the three types of usability testing?

Broadly speaking, there are three categories usability testing can fall into:

  • Moderated vs. unmoderated
  • Paper vs. digital
  • Explorative vs. comparative