How to analyze and report usability test results
In this chapter, you'll learn how to analyze and report on usability testing results, what to include in your usability testing report to make sense of the data collected, and devise a plan of action.
Run usability testing with Maze
Maze generates instant, immediately-actionable reports, including detailed survey and user flow analytics. Get started for free.
How to analyze your usability testing results
1. Organize the usability issues you identified
You probably have multiple data points by now. The first step to understanding them is to organize the issues you recorded during usability testing.
The most straightforward method is to write down every single issue along with a comprehensive description of where in the design it happened, how it occurred, and the task the user was performing, plus all of the details that describe the problem.
Alternatively, you can arrange the data in a table format according to one of the methods presented by Lewis and Sauro in Quantifying the User Experience: Practical Statistics for User Research. On one side, include each task with its respective issue. On the other, include a list of all your usability test participants so you can mark down every user who’s experienced the same issue.
2. Prioritize issues based on criticality and impact
Not all issues discovered during usability testing are equally important. You need to categorize the problems based on the severity of the issues identified. Ranking your findings will make them more actionable and help you classify the issues you need to handle according to their priority.
After this exercise, you will know which ones to prioritize based on criticality and your team’s resources. For instance, a usability issue that makes it hard for users to find the call-to-action on the page might be a more critical issue than a typo.
According to NNGroup, we can categorize usability issues based on five severity levels:
- 4 - Critical: In this category, write down all problems that interfere with the user experience and impedes users from completing their tasks. These issues need to be fixed by all means necessary before a release.
Examples: The user didn’t receive a confirmation message after making a payment or they can’t sign up for an account.
- 3 - Serious: In the Serious category, include issues that have to be fixed as soon as possible as they slow down a user’s experience with your website or product
Examples: The user can’t navigate to the next page in their flow or can’t reset their password.
- 2 - Medium: In this category, include issues that have a medium severity. They might not interfere with the user experience often, but can still be frustrating for users to experience.
Examples: The user has to scroll a lot to find a category in a dropdown or the text is too small on the Pricing page
- 1 - Low: Include cosmetic issues that are necessary to fix, but don’t hinder the user experience as much as critical or serious issues. These issues might affect your brand image, such as a spelling error, but don’t severely impact your UX.
Examples: a misspelled word in the heading or a logo that isn’t updated on the About us page.
- 0 - No issue: This situation occurs when a user reports a problem that turns out to be a non-issue or a feature request, so you don’t need to take further actions right now.
3. Discuss the findings with your team
The fundamental goal of usability testing is to build a product that your customers will be able to use. That’s why involving different team members from the product team in the process of usability testing is important. Every team member will benefit from understanding the issues users experience and help make decisions to improve them.
I’ve seen the most success with usability testing when designers, engineers, and product managers are involved in understanding how people actually use or don’t use a product. You want people who have the ability to make decisions to be involved, so you can make sure that all of your research is building a case for the important decisions that you want to make, whether that’s as a high level as “do we release this product?” or as focused as “Is this a more usable menu?”
Behzod Sirjani, Founder of Yet Another Studio
What to include in your usability testing report
At the end of the usability testing process, you should create a final report to share your insights and findings with the rest of your team. A usability test report should be easy-to-understand and actionable, helping you and your team plan future design iterations with clear next steps.
Research results need to be organized, synthesized, and analyzed, preferably in a partnership or group to broaden the perspective.
Taylor Palmer, UX Director at Lucidchart
A usability testing report displays the findings in a structured way that helps the team come up with action plans, see what features work well, and most importantly, prioritize the issues that need to be fixed. Here’s what you should include in your usability test report.
First, include an overview of the usability test you ran. Here, describe the design you tested, the location of the tests (in-person vs. remote), the usability testing tools you used, the time and date of the tests, information about the moderator (if any), etc. This provides a concise overlook at the tests to anyone who wasn’t part of the process including other team members and stakeholders.
Next, mention the main goals behind your usability testing in a few sentences. While the general goal of usability testing remains to test and assess how usable, intuitive and easy to use a design is, here are five examples of specific objectives you should opt for:
- Test the new design of an upcoming feature
- Increase conversion rates on the sign-up flow by 30%
- Test your new user flow against the old one to confirm a design change
- See if on-screen instructions are clear enough
- Make it easier for people to complete goals in less time
In this section, mention the different usability testing methods you used to run the tests. This includes information about the type of data collected (qualitative vs. quantitative) and insights on the types of tests that were conducted. For the latter aspect, specify if you turned to unmoderated or moderated usability testing and the location where tests were conducted (in-person vs. remote).
Next, share the methods used to select the participants for your test. Here, focus on sharing screening information on how participants were selected, the number of user testing participants, and why target users were selected.
Here, provide demographic information about each test participant who took part in the tests, including their age, profession, experience with similar technologies, and any other relevant data based on the target audience you defined in the testing plan.
A recommended best practice is to make sure you don’t mention their names to guarantee respondent privacy. Fictional names can be used instead to talk about a scenario or group of people.
This section is the core of your usability testing report as it includes all the findings identified after you’ve analyzed your usability testing results. We’ve already learned how to analyze the results based on the severity of the issues above. Once you prioritize the issues into categories, include the most critical findings in the report.
Here, you can complement the main findings with usability metrics such as completion rates, time spent, or bounce rates for individual tasks. This data will help you back-up the findings, and make it easy for stakeholders to understand the problems. You can also highlight user comments or satisfaction rates to provide relevant context around an issue.
If relevant, include recommendations for things that need to be changed. Ideally, those will be discussed and planned with members from your product team and implemented in subsequent design iterations.
Once you finish compiling all findings into a usability test report and discussing the results with your team, planning the next steps is the next important part of any usability testing. No report is useful unless the results are acted upon.
Analyzing the report’s findings and coming up with new solutions to tackle the issues identified is the responsibility of the entire product team—whether that involves discussing the feedback or retesting a design for new results. Try to keep your usability testing report short and the findings actionable, and focus on delivering something your team can act upon.