A Beginner's Guide to Usability Testing

Picture this: Your team has spent countless hours ideating and developing your product with the perfect design and functionality. But a nagging question lingers in your mind: "Will users find it as intuitive as I do?".

Usability testing is the unsung hero that reveals authentic user experience. Even if you don’t have any doubts in your mind about how intuitive your product is, it’s crucial to validate your assumptions. Plus, if you return to usability testing throughout the product lifecycle, you’ll catch issues quickly and see how user behavior changes over time. It’s a win-win.

Usability testing is a type of user research that evaluates the user's experience when interacting with a website or app. It helps your designers and product teams assess how intuitive and easy-to-use products are.

The usability testing process identifies problems with your product you may have otherwise missed—by asking real users (instead of developers or designers) to complete a series of usability tasks on the product. The results, success rate, and paths taken to complete the tasks are then analyzed to identify potential issues and areas for improvement.

The ultimate goal of usability testing is to create a product that solves your user’s problems and helps them achieve their objectives with a positive experience.

Before we can delve into the specifics of usability testing, we need to dispel some misconceptions. User research, user testing, and usability testing are often used interchangeably—but they aren’t necessarily the same.

User research refers to the overarching act of gathering feedback and insights from product users, and using this to inform product decisions. Usability testing is a specific kind of user research done to assess the usability of a product or design.

To help clarify, here are some UX research methods and techniques that don’t qualify as usability testing:

The kind of test you want to run will help you choose the right usability testing method. All product research and testing broadly falls into three main categories:

If you prefer to learn in video form, take a look at this handy video for a rundown on the different types of usability testing.

Any user research will fall into the category of qualitative or quantitative. You ideally want usability testing to gather both kinds of data, to provide a rounded evaluation of the user experience.

Qualitative usability testing focuses on the ‘why’; understanding users' experiences, thoughts, and feelings while using a product. For example, you could conduct a think-aloud study where users verbalize their thoughts while using your product to complete usability tasks. Qualitative data can be gathered from observation, interviews, and surveys.

Quantitative usability testing focuses on collecting and analyzing numerical data like success rates, task completion times, error rates, and satisfaction ratings. It’s about identifying patterns, making predictions, and generalizing findings.

Moderated and unmoderated usability testing are two different approaches to usability. In moderated usability testing, a moderator guides the users through the test (in-person or remotely). They answer any questions participants may have, ask follow-up questions, and record observations during the test.

Unmoderated usability testing, as the name suggests, doesn’t involve a moderator. Users complete tasks independently, typically using usability testing tools that record their actions and responses.

Research can be done remotely or in-person, depending on the type of product you're testing and your research goals.

Remote usability testing can be moderated or unmoderated, and is done using online tools or software that allows users to share their screens, record their activity, and provide feedback. It’s useful because your team and test participants can be based in entirely different locations.

In-person usability testing, on the other hand, is conducted in a physical location, usually a usability lab or other research facility. For that reason, it can be more expensive, time-consuming, and limiting in terms of sample size and geographic reach. Many researchers opt for remote research, however in-person testing may be necessary for products that require safety considerations, supervision during use, or physical testing.

Usability testing provides insights into user preferences, motivations, and goals. To help you get the lowdown, we spoke to experts in the research and product space to get their perspectives on how to do effective usability testing.

First up, here are some key benefits:

When you run usability tests, you save time and money by avoiding costly development mistakes. For example, if you find out users struggle to navigate a specific feature, you can fix this before launch. It's significantly cheaper to make changes before launch than after a product has already been released.

“We all have unique backgrounds, lived experiences, perspectives, preferences, and abilities. All of these things influence how we understand, approach, and experience a product,” says Behzod Sirjani, Founder of Yet Another Studio. So, even if you understand your product, your users might approach it differently.

By talking to users directly and observing how they experience your product, you can better understand their needs and tailor the product to work for them—ultimately serving their needs and solving their problems more effectively.

Accessible products are designed and developed to be usable for as many people as possible—including those with physical, visual, auditory, or cognitive requirements.

Of course it’s important to comply with accessibility standards and regulations, but your product will also benefit from prioritizing accessibility. When you include users with diverse abilities and needs in the usability testing process, you’re contributing to and promoting a more equitable digital landscape.

“Usability testing is one of the most fundamental ways to test the success of a product. Ensuring customers can find, understand, and use a solution contributes to the overall success of a product and business,” says Taylor Palmer, Product Design Lead at Range.

Usability testing enables product teams to identify potential issues and make improvements before releasing a new product or feature. This can lead to better user experiences, more consistently—creating a loyal user base, and reflecting your overall brand reputation positively.

Our brains love to create shortcuts to make quicker decisions or inferences. They’re just trying to be efficient, but this can lead to subconscious beliefs or assumptions, called cognitive biases.

Usability testing helps combat biases like the false-consensus effect by providing objective feedback from real people, ensuring that design decisions are based on authentic user behavior rather than assumptions or opinions of those already-familiar with the product.

You need to carry out usability testing continuously to make sure your product stays relevant and solves your user’s most pressing problems throughout its lifecycle. Here's a quick overview of when to do usability testing:

You can identify user needs, expectations, and potential pain points early on through preliminary research, surveys, or even testing competitors' products. With that data as a benchmark, you'll start the design process with a solid foundation for creating a user-centric product that's even more intuitive to use.

"I think usability testing is applicable at every stage of the design process. I test at an early concept phase, and once we get the first results, we use them throughout other design phases and re-test with users," says Vaida Pakulyte, UX researcher and Designer at Electrolux.

For example, you could use card sorting and tree testing on your wireframe or prototype to discover how users would approach the product—or make sure the existing design is easy to navigate.

At this stage, you already have a working, or almost-working, product. Now it’s time to evaluate its overall effectiveness through summative testing. The goal of summative testing is to measure how well your product performs via specific usability heuristics like efficiency, effectiveness, and user satisfaction. Each testing task should represent typical user scenarios.

Even after your product goes live, it’s crucial to ensure it remains user-friendly, accessible, and relevant over time. With a research tool like Maze, your team can create and run live website testing and discover issues caused by browser updates, third-party integrations, or changes in user behavior patterns.

H2: What to do before your usability test
Usability testing starts with careful planning and preparation. Consider these questions before delving into your testing period:

Once you’ve thought about the above questions, it’s time to move on to the final decision—and what we’re focusing on in the next chapter:

What usability testing method are you using?

Regardless of your resources or budget, it's possible to set up and run usability tests. Following a consistent process is key to getting started:

The point of testing is to deliver a product focused on providing a positive user experience. To do that, you need to understand who your users are. Start by asking questions like:

Once you have your answers, formulate a problem statement that makes it clear what insights you want to gain. For example: "We want to assess the usability of our product for 20-30 year old customers to complete checkout within 3 minutes." This will help you determine the right research methods and metrics for evaluating the users’ experience.

It’s worth investing in a tool to facilitate your team to carry out fast, scalable usability testing. We recommend choosing one that supports them to test early and often in the process, including after launch.

This continuous research can inform your design decisions, keep you updated on evolving user needs, and will ultimately help retain users. For example, a tool like Maze supports testing throughout the product development lifecycle, and after launch.

Here’s what you and your team can do with Maze:

How do you know if your team is evaluating the right thing? Establishing your evaluation criteria will help focus your team on specific elements of the user experience.

Depending on the goal of your test, consider evaluating usability metrics like:

For example, to measure user accessibility during a checkout process, you can review metrics like the time it takes for users to complete the checkout and overall success rate (i.e. percentage of successful purchases). It’s also a good idea to establish benchmark test results using earlier prototypes or competitor products, so you can track user performance as your product evolves over time.

A well-crafted usability testing script outlines the tasks to be performed, instructions to be given, and questions to be asked. It serves as a roadmap for the test participant and provides an opportunity to identify any potential problems or areas of improvement. Head to chapter five for an in-depth walkthrough on creating your usability tasks and script.

Keep your tests short and sweet. When it comes to test duration, it depends on factors like the product's complexity, the scope of the testing objectives, and participant engagement level. But, as a general guideline, usability tests are an average duration of 15–20 minutes with 5—10 tasks. It should be short enough to maintain participants' focus, while giving your team sufficient time to gather meaningful insights.

If you want to nail usability testing, getting your team to do a pilot run is a smart move. Ideally, your test participant should be someone inside your organization, but not directly involved in your project, as they might have preconceived notions about how the product works. Otherwise, you could test a small batch of participants who represent your target user.

The pilot test is where you evaluate the clarity and effectiveness of your testing materials, like the script, tasks, and instructions. You can also validate the feasibility of your chosen testing methods, tools, and environment. Think about it as the usability test for your usability test.

“No matter how much time you spend on ideation and prototyping your design solution, you should always test it with real people,” says Nick Babich, Editor-in-chief of UX Planet. You must recruit participants that represent your target audience and have similar backgrounds, preferences, and experiences as your intended users.

Here's how you can hire participants in three easy ways:

When recruiting participants, consider their availability and willingness to participate in the session. Screen participants based on your criteria and avoid those who aren’t a good fit for the test.

Usability testing should feel like an interactive conversation, not an interrogation. To get the most out of your usability test, be sure to brief your team on how to interact with participants and reduce bias. Your researchers should avoid leading questions that might influence participants' behavior—and each participant should feel comfortable providing honest feedback.

Take a look at this video for some usability testing best practices:

When you transform raw data and participant feedback into actionable insights, you'll be able to identify user pain points, prioritize UX improvements and communicate them effectively within your team.

Some metrics you should be tracking include:

Usability testing is an ongoing process that requires iteration, feedback loops, and revisions throughout the product lifecycle. You can test different versions of a design and analyze the results to determine which one works best. As your product and audience grow, it's important to re-evaluate your usability testing process and adapt it to accommodate new goals, user segments, and market conditions.

Usability testing can help you create user-centric products that deliver exceptional experiences. When you understand user behavior, preferences, and needs, you can create intuitive products your customers will love.

When done correctly, usability testing can help your team:

If you're unsure about how to implement usability testing, try using a continuous product discovery tool like Maze. Your team can create or upload existing prototypes, assign tasks and questions to testers, recruit participants, and even download reports with actionable insights—all from one central hub.

Head on to chapter two to dive into the difference between moderated and unmoderated usability testing–and when to use each type.