Usability testing is divided into two fundamental approaches: formative and summative. Both are essential for product development, but serve different purposes and are used at different points in the product development process.
TL;DR:
- Formative testing occurs early in the design process and is all about iteration—this is when you identify what doesn’t work, fix it, then test again. It’s driven by the need to shape a product based on real user interactions and feedback.
- Summative testing occurs toward the end of product development, and assesses the finished product’s effectiveness and overall user satisfaction. It confirms whether the product meets its intended goals and is ready to launch.
These testing methods are also known as formative and summative evaluation—whether you want to call it testing or evaluation, the terms are interchangeable. and mean the same thing:
In this article, we look at how each type of testing contributes to the product development cycle; improving design decisions and a product’s user experience. We also cover when and how to use the right testing approach, with practical examples of formative and summative testing in action.
Formative vs summative usability testing: An overview
Formative usability testing | Summative usability testing | |
|---|---|---|
What it is | Early-stage testing to refine product features based on user interactions and feedback. | Final-stage testing that evaluates the overall functionality and readiness of the product (often against a benchmark, like feedback collected before the redesign). |
When to use it | During the initial concept and design phases, especially when prototypes and wireframes are being developed. Also crucial during the early stages of user interface creation and immediately after initial user feedback sessions. | At the end of the development process. Typically just before or after the official launch, or during late-stage beta testing with a larger user group. |
Why to use it | To catch usability issues early on and redesign based on qualitative feedback, ensuring the product aligns with user expectations. | To measure if the final product meets the specified usability criteria and user satisfaction. Essential for products that must perform flawlessly to gain user trust. |
What is formative usability testing?
Formative usability testing is used during the early stages of product development to identify and resolve usability issues. The primary goal is to improve the design by gathering qualitative feedback from users as they interact with the product in its initial stages. This process helps designers understand the ‘why’ and ‘how’ behind user behaviors and preferences.
This method involves small groups of five to eight users or research participants who interact with the product in its preliminary forms, such as wireframes or prototypes. These participants provide qualitative feedback through methods like think-aloud protocols, where they verbalize their thought processes while navigating the product.
Product tip ✨
Boost your usability testing with Maze Clips. Record user interactions and verbal feedback effortlessly, gaining real-time insights into their experiences. Share clips with your team to quickly identify and address usability issues.
What metrics do you look at when conducting formative usability testing?
Depending on the different usability testing methods you run, you’ll likely have different metrics tailored to each development phase.
Here are some key usability testing metrics commonly used in formative usability testing:
- Task completion rates: This measures whether users can complete specific tasks without assistance. It helps you understand how well a design supports user activities.
- Time on task: Recording how long it takes users to complete tasks can indicate the efficiency of the interface. Shorter times typically suggest better usability.
- Error rates: Counting the number and types of errors users make during testing provides insights into potential areas for improvement.
- User satisfaction: Often measured through post-task usability testing questions or interviews, this metric assesses how satisfied users feel with the overall experience and specific features.
- System usability scale (SUS): A quick, reliable tool to assess usability. It consists of a 10-item questionnaire with five response options for respondents, from Strongly Agree to Strongly Disagree, providing a global view of subjective assessments of usability. If you’re using Maze, you can speed through setup with this system usability scale template.
When to conduct formative usability testing
Formative usability testing is ideally conducted early in the product development cycle to guide and refine a product's design.
Here are some of the different stages of product development when formative usability testing is particularly beneficial:
- During concept testing: Concept testing ensures the product's direction aligns with user expectations and market needs. It involves presenting potential users with concept sketches, mockups, or storyboards to gather feedback on the product's proposed value and usability. This stage is crucial for market validation, as it confirms whether the concept has potential appeal and solves a real problem for users.
- During prototype testing: Prototype testing allows for a deeper exploration of the design and functionality, helping to identify usability issues and user interface improvements. This stage is more detailed than concept testing, involving interactive elements that users can engage with, and providing more concrete feedback and actionable insights.
What are the benefits of formative usability testing?
Both summative and formative usability testing have their benefits. Some of the pros of formative user research include:
- Early problem identification: By engaging with users early in the UX design process, formative testing uncovers usability issues before they become too costly to fix. This helps avoid major revisions later in the development cycle, saving you both time and resources.
- Iterative design concept validation: Formative testing supports an iterative design approach, where feedback is used to continuously refine and improve the product. This cycle aligns the product more closely with user needs and expectations for a more user-centric design.
- Better user experience: Since formative testing involves detailed observation of how users interact with the product, it provides deep insights into user behavior, preferences, and difficulties
- Risk reduction: Understanding what works and what doesn’t from the user's perspective helps ensure market fit; and that the final product will be accepted and valued by its target audience
- Supports product discovery: Formative usability testing is integral to the product discovery phase, helping to identify what features are necessary and how they should function. It also supports the UX workflow by providing actionable data that guides design decisions throughout the new product development process.
How to conduct formative usability testing + example
Let’s take a look at the steps involved in conducting effective formative usability testing with an example featuring a fictional mobile app, HealthTrack, designed to help users manage their fitness routines.
Step 1: Define objectives and scope
This stage is about aligning your usability testing efforts with broader business goals and user needs. It determines what aspects of the product will be examined, and focuses resources effectively so your usability testing plan is directly relevant to the product's success.
For example, should the test cover only basic functionalities, or should it delve into complex user interactions? This decision is often influenced by factors like project timeline, available resources, and the areas of the product that need improvement—as identified during preliminary research.
You also need to specify who your target users are, what UX research methods will be used (such as UX surveys, task-based testing, or user interviews), and how results will be measured and analyzed.
📌 Follow the example
Objective: Your team aims to optimize the user experience for HealthTrack, focusing on the setup process and daily usage efficiency to enhance user engagement and retention.
Scope of testing:
Features to be tested: Concentrate on user registration, personal workout plan creation, and activity tracking functionalities
Aspects to examine:
– User interface intuitiveness and interaction flow
– Clarity of instructional content
– Performance and accuracy of the tracking features
Target audience: You will target fitness enthusiasts of varying expertise, from beginners to intermediate athletes, to ensure the app appeals to a broad demographic within the fitness community.
Methodologies to be used:
– Surveys and interviews: Collect qualitative data on user expectations and experiences with fitness apps
– Task-based testing: Observe users as they register in the app, set up workout plans, and log activities to identify usability barriers
Step 2: Develop a testing plan
Your testing plan specifies how the testing will be executed, detailing methodologies, participant selection, tasks, and scenarios. It ensures that all team members have a clear understanding of their roles, required usability testing tools, the sequence of activities, and how results will be captured and analyzed. Essentially, it translates the broad objectives into a concrete action plan that can be systematically followed to gather meaningful user insights.
📌 Follow the example
Methodologies:
– Surveys: Gather initial perceptions and expectations from new users
– Task-based usability testing: Observe users as they complete specific tasks within the app, such as registering an account, setting personal fitness goals, and logging a workout session
– Think-aloud protocol: Encourage users to verbalize their thoughts and feelings as they navigate through the app for insights into their cognitive process
Participant criteria:
– Demographics: Target users who are fitness enthusiasts aged 20–40, who frequently use technology to assist with workout tracking
– Experience levels: Include a mix of beginners and experienced individuals to capture a wide range of feedback on the app's usability across different levels of fitness and tech familiarity
Tasks and scenarios:
– Task 1: Create a user account and set up a profile, complete with personal health metrics
– Task 2: Plan a week's workout using the app's planning tools
– Task 3: Log daily workouts and mindfulness sessions and review the generated summary reports
Data collection methods:
– Screen recording and audio: Capture user interactions and verbalizations during the task completion
– Post-task interviews: Conduct brief interviews after each task to delve deeper into the user's experience and gather qualitative feedback
Test environment:
Remote testing: Conduct the test remotely, using screen-sharing and video conferencing tools
Timeline and budget:
– Preparation phase: Two weeks for recruiting participants and preparing testing materials
– Testing phase: One week to conduct all sessions
– Analysis phase: Two weeks to analyze the data and compile a report
Budget: Allocate funds for participant incentives, software tools for remote testing, and resources for data analysis
Step 3: Create usability testing scripts and questions
Usability testing scripts include an introduction to the session, detailed instructions for each task the participants will perform, and specific user research questions to probe deeper into the user experience. These questions are designed to gather detailed, actionable feedback and can be both task-specific and general.
If you’re stuck on writing usability testing questions, take a look at the Maze Question Bank—our open-source question repository with 350+ researcher-approved questions for different research and testing scenarios.
📌 Follow the example
Sample usability script for HealthTrack:
"Welcome to our testing session today. We’re testing the HealthTrack app, and your feedback will help us improve its usability. Please remember: we're testing the app, not your abilities. We ask that you perform a few tasks while thinking aloud, describing your thoughts and actions as you go. So we can look back on this later, we’d like to record the call. Are you comfortable with that?”
"First, please create a new user account and set up your profile by entering your personal health goals."
Usability testing questions:
General questions: Ask at the beginning or end of the session to get overall impressions
– "What was your initial impression of the app when you first opened it?"
– "How do you feel about the amount of information requested during sign-up?"
Task-specific questions: Follow up on specific actions within the tasks
– After registration: "Was there any part of the sign-up process that felt overly complex or unnecessary?"
– After setting health goals: "How intuitive did you find the process of setting up your health goals? Could you describe any challenges you faced?"
– During workout logging: "As you logged workout, were all needed options readily accessible?”
Step 4: Recruit participants
When recruiting research participants, you need to identify and select individuals who represent your product’s target audience. Often, recruitment is facilitated by using existing customer databases, social media, professional recruitment firms, or user research platforms.
One option to simplify your recruitment process for formative usability testing is with the Maze Panel. Tap into a global pool of over 280 million participants to find the perfect match for your user profile personas. You can quickly filter participants based on demographics, user behavior, or specific needs. Plus, it seamlessly integrates with your testing setup, simplifying the logistics of user recruitment and testing.
📌 Follow the example
Participant criteria:
– Age range: 20-40, capturing a broad range of adult users
– Fitness level: From beginners to advanced fitness enthusiasts
– Tech savviness: Include both tech-savvy users and those who are less familiar with using mobile apps to manage health
Recruitment methods:
– Social media campaigns: Use targeted ads on platforms like Facebook and Instagram, which allow for demographic-specific targeting
– Partnerships with gyms: Collaborate with local fitness centers to engage users who are already invested in fitness
– User research platforms: Use platforms like Maze’s Panel to quickly reach a broad audience
Incentives: A $30 gift card (of their choice) or a free one-month subscription to the app
Step 5: Conduct the formative usability test
Conduct sessions where participants interact with your product while performing predetermined tasks. It's important to ensure the environment is conducive to open feedback and that all interactions are carefully documented—this means considering user interview best practices, and ensuring your UX research tool is intuitive and easy to use.
📌 Follow the example
Technical setup:
– Ensure all participants have a stable internet connection and the video conferencing tool is functioning correctly
– Confirm that screen sharing and recording functionalities are set up to capture the session for later analysis
Session introduction and consent:
– Clearly explain the purpose of the test
– Ask for consent to record the test and share the data
– Set out any guidelines or help available
Task execution and guidance:
– Task 1: "Please create an account in HealthTrack. Start by entering your personal details and fitness goals."
– Task 2: "Now, let’s set up your first workout. Choose your preferred workout days and the type of exercises you want to include."
– Task 3: "Lastly, please log today’s workout using the tracking features of the app."
Encourage participants to think aloud and to express their thoughts and feelings about each step
Observation and documentation:
– Observe the ease or difficulty with which participants complete each task
– Take notes on any navigation issues, user errors, or areas where participants seem confused or frustrated
Session wrap-up and debriefing:
– Thank participants for their time and valuable contributions
– Ensure participants know how their feedback will be used to improve the app and confirm their consent for using the data collected
– Distribute the incentive or give information on how to redeem it
Step 6: Analyze results and iterate
Analyzing the results of formative usability testing is a critical step that involves synthesizing the data collected during the testing sessions to identify trends, common issues, and areas for improvement. This analysis helps translate user feedback and observed behaviors into actionable insights, which can then be used to iterate on the product design.
📌 Follow the example:
User satisfaction:
– Initial score: 3.2/5 based on user feedback regarding the app’s interface and functionality
– Action taken: Based on specific feedback, you simplified the navigation and improved the visual appeal of the workout logs
– Resulting score: More formative testing found that user satisfaction improved to 4.0/5, indicating a positive response to the changes
Ease of navigation:
– Initial observation: Users often failed to find the hydration tracking feature
– Action taken: You introduced more intuitive icons and a more prominent placement for the hydration tracking feature within the app
– Post-iteration feedback: The changes reduce navigation issues, and feedback shows that new users can find the feature 30% faster
Frequency and types of errors:
– Initial findings: Users made frequent input errors when entering their fitness goals
– Action taken: Added tooltips and dropdown menus with examples of typical fitness goals
– Impact: The additional features led to a 50% reduction in input errors
User task success rate:
– Pre-iteration: Only 65% of users could successfully set up a workout plan on their first try
– Action taken: You revamped the tutorial section and simplified the workout plan setup process
– Post-iteration success rate: 85% of users were able to successfully set up their workout plans without additional help
That’s all for our section on formative usability testing—now let’s move onto summative.
What is summative usability testing?
Summative usability testing is a technique used to evaluate the final usability of a product before it goes to market. This type of usability testing typically occurs at the end of the development process.
Both quantitative and qualitative data can be collected in summative usability testing.
Although summative testing is often more quantitatively focused (as we’ll see from the metrics listed below), qualitative data like user feedback from interviews or questionnaires can help provide deeper insights into user satisfaction and the context of usability problems. This data helps explain why certain issues may occur and how they affect the user experience.
What metrics do you look at when conducting summative usability testing?
You use many of the same metrics in summative testing as you do formative testing:
- Task completion rates: Measures the percentage of users who are able to complete predefined tasks successfully. This is a direct indicator of whether the product allows users to achieve their goals.
- Time on task: Assesses the amount of time users spend to complete tasks, which helps you understand the efficiency of the product interface and the complexity of tasks.
- Error rates (number and severity): Error rates are typically categorized by their severity, which can range from minor annoyances that do not affect task completion, to critical errors that prevent users from completing the task at all. A good target for error rates depends on the complexity and context of the product, but generally, lower error rates indicate better usability.
- User satisfaction ratings (quantitative surveys): These surveys collect numerical data to measure how satisfied users are with the product's overall experience. Typically administered through rating and Likert scales ranging from one for Very Dissatisfied to five for Very Satisfied.
- Pass/fail rates of specific tasks: Unlike general task completion rates, this measures the percentage of tasks that are completed correctly to a required standard; providing a clear pass or fail outcome. This measure helps verify essential functionalities within the product work as intended.
When to conduct summative usability testing
Here are the typical phases of product development when summative testing is most appropriate:
- Post-development: After the product is developed and all features are implemented, summative testing can assess the complete system to verify it meets the predefined usability standards or usability heuristics necessary for launch
- Pre-launch: Just before the product launch, to ensure all central usability goals have been achieved and the product performs well under real-world conditions
- Post-launch: To benchmark the product against previous performance metrics or competitor products—this helps position the product in the market based on its usability strengths
- To check regulatory compliance: In some industries, like healthcare or finance, summative testing may be required to meet regulatory standards before the product can be marketed or sold
What are the benefits of summative usability testing?
The most notable benefits of summative user research are:
- Objective measurement: Summative testing helps you objectively measure a product's usability through predefined criteria. It validates whether the product meets the usability goals set during the development process.
- Quantitative data: This type of testing provides quantitative data that can be used to benchmark the product against usability standards or previous versions. Metrics relating to effectiveness, efficiency, and user satisfaction are typically collected and analyzed.
- Final verification: Conducting summative testing before a product's release ensures any remaining defects are identified and corrected. It acts as a final check to confirm the product is ready for its end-users.
- Regulatory compliance: For products in regulated industries, summative testing may be required to comply with specific usability regulations before the product can go to market.
How to conduct summative usability testing + example
Summative usability testing is conducted after formative testing and near the end of the product development cycle. Referring back to our example of the HealthTrack app, here are the steps to conduct summative usability testing effectively, especially after insights have been integrated from earlier formative evaluations:
Step 1: Finalize test objectives
Finalizing test objectives in summative usability testing involves specifying what you intend to measure regarding a product’s usability. This step is critical as it sets clear benchmarks for effectiveness, efficiency, and satisfaction that the product must meet before it can be deemed a success. The goals should be clearly defined, measurable, and directly linked to the user requirements and UX research objectives identified earlier in the development process.
📌 Follow the example
Specific goals for HealthTrack:
– Ensure that at least 90% of tasks are completed successfully by new users without assistance
– Aim for an average task completion time that does not exceed two minutes for daily activity entries
– Achieve a user satisfaction rating of at least 4 out of 5 in post-test surveys
Step 2: Define and select user groups
Even though you might have recruited participants for formative usability testing earlier in the development process, it is common and often necessary to define and select new or additional user groups for summative usability testing.
Here’s why:
- Formative testing generally aims to uncover usability issues during the development phase and involves iterative testing. This might use a smaller group of users to explore different aspects of the product.
- Summative testing aims to validate the final product, and needs larger numbers of participants or representation that accurately reflects the target market or end user
📌 Follow the example
Specific user groups for HealthTrack:
– Demographic variability: Wide age range from young adults (ages 20+) to older adults (ages 40+)
– Fitness levels: Users with varying fitness levels, from beginners to advanced athletes
– Tech familiarity: From tech-savvy users to those with minimal digital interaction
Step 3: Prepare test materials
Test materials typically include detailed task scenarios, instructions for participants, questionnaires for pre- and post-test assessments, and any scripts for guiding the testing process.
Consider creating a research repository to organize and archive these materials. This repository can serve as a centralized hub where all test-related documents are stored, including scenario descriptions, user instructions, and standardized questionnaires.
📌 Follow the example
Task scenarios:
Scenario 1: "Set up a new workout plan"
Instructions: "Imagine it's your first time using HealthTrack. Please create a workout plan targeting strength training for the upcoming week. Select exercises, set goals, and schedule your sessions."
Expected time to complete: 5 minutes
Scenario 2: "Log Daily Workout and Wellbeing Activities"
Instructions: "Log any physical activities you've done today using the app. Then, record any wellbeing activities such as mindfulness or meditation."
Expected time to complete: 3 minutes
Questionnaires:
Pre-test questionnaire:
– "How often do you use fitness tracking apps?"
– "What features do you consider essential in a fitness app?"
Post-test satisfaction survey:
– "On a scale of 1 to 5, how would you rate your overall satisfaction with the app?"
– "Did you find the process of logging activities simple?"
Scripts for moderators:
– Introduction script: "Thank you for participating in this session. Today, you'll be performing a couple of tasks that typical users of the HealthTrack app might do. Remember, we're testing the app, not your abilities."
– Gather consent: “So we can look back on this later, we’d like to record the call. Are you comfortable with that?”
– Task guidance: "Please begin with the first task of creating a workout plan. Remember to think aloud as you navigate through the app, describing what you find intuitive or challenging."
– Debrief questions: "What was the most challenging part of using the app today? Were there any features you found particularly useful or felt were missing?"
Step 4: Conduct a pilot test
Conducting a pilot test is a preparatory step in the summative usability testing process. It involves running a trial session with a small number of participants from within your organization or a select group of external users who are not part of the main test. The purpose is to identify any issues with the test (not the product), such as setup, materials, and methodology before the full-scale testing begins.
Step 5: Execute the testing sessions
Next, you need to carry out structured tests under controlled conditions to validate the product’s usability. These sessions typically include live website or app testing where real users complete predefined tasks while their interactions are observed and recorded.
Step 6: Collect and analyze data
UX reporting and analysis involves data collection and interpreting quantitative and qualitative data to assess the product's usability and user experience. This step is the culmination of all your work—time to see what users think, and validate the design before final release.
Effective data analysis helps identify whether a product meets the established usability goals and benchmarks. It's an integral part of UX research, where the focus shifts from exploratory (as seen in formative testing) to confirmatory, aiming to confirm your research hypothesis about the product's usability through structured testing and rigorous data analysis.
📌 Follow the example
Task completion and pass/fail rates:
We found a 92% task completion rate for workout planning, which was above our target of 90%. However, wellbeing logging had an 85% completion rate, indicating a need for interface improvements.
Time on task:
The average time taken to set up a workout plan was 4 minutes, slightly above our target of 3.5 minutes. This suggests the need for a more streamlined process.
Error rates:
A notable error was users incorrectly logging activity types, occurring in 30% of sessions, highlighting the need for clearer categorization and instructions
User satisfaction ratings:
The overall satisfaction rating averaged 4.3/5, indicating strong user approval. The lowest scores were typically associated with the wellbeing-logging feature.
Comparison with benchmarks:
HealthTrack performed on par with industry benchmarks for workout planning but fell short in wellbeing logging efficiency and error management
Step 7: Report recommendations and get stakeholder buy-in
The final step in summative usability testing is compiling the findings and recommendations into a report and presenting this information to stakeholders. A thorough research operations process optimizes the way these usability issues and insights are structured and communicated.
With a user research platform like Maze, you can transform usability testing results into customizable, easy-to-understand automated reports. Track key metrics like usability scores, mission paths, and success metrics automatically. These actionable insights make it easier to demonstrate the value and impact of UX improvements when you’re looking to get stakeholder buy-in.
Conduct decision-driving usability testing with Maze
Conducting formative and summative usability testing requires a methodical, dual approach to ensure that products meet design expectations and align with user needs, enhancing overall satisfaction and functionality.
Maze is a user research platform that makes user insights available at the speed of product development. It’s an ideal solution for teams that want to simplify these usability testing phases.
With research methods like Prototype Testing, Card Sorting, and Feedback Surveys to gather detailed user insights—plus ample integrations with design tools like Figma, Sketch, and Adobe XD for a smooth workflow—Maze makes it easy for teams to conduct formative and summative research on designs, ensuring your product is always meeting the mark.
Frequently asked questions about formative vs. summative usability testing
What is the difference between formative and summative usability testing?
What is the difference between formative and summative usability testing?
Formative usability testing is conducted early and throughout the product development process to identify issues and guide design iterations. It focuses on collecting qualitative feedback to improve the product's usability as it evolves.
Summative usability testing occurs at the end of the development cycle. It assesses the finished product's usability against predefined criteria, using structured and quantitative methods to validate that it meets specific usability standards before release.
Can research studies include both formative and summative usability testing?
Can research studies include both formative and summative usability testing?
Yes, research studies can include both formative and summative testing. This approach is common in mixed-methods research, where both types of evaluations are used to gain insights across different stages of a project.