Evaluative UX research: Methods to validate and improve UX

How do you know if your product actually works for users? You test it.

Evaluative UX research helps you assess designs, find usability issues, and improve the user experience with real data. It ensures your product meets user expectations before it goes live—saving time, money, and user frustration.

This chapter breaks down evaluative research key methods, like usability testing and A/B testing, plus real-world examples of how UX teams have refined products based on user feedback.

Evaluative research is a research method used to test and assess (aka evaluate) a product, feature, or concept to determine its effectiveness, usability, and overall user experience. It helps teams identify what works well, what needs improvement, and how users interact with a product in real-world conditions.

Also known as evaluation research or program evaluation, this kind of research is typically introduced in the early phases of the UX design process to test existing or new solutions until the product becomes ‘final.’

UX research happens at different stages of product development, and each type of research serves a purpose:

Here’s a quick overview of generative vs. formative vs. evaluative research:

Evaluative research is important because it helps you:

There are three types of evaluative studies you can tap into: formative research, summative research, and outcome research.

Formative research is conducted early and often during the design process to test and improve a solution before arriving at the final design.

Running a formative evaluation allows you to test and identify issues in the solutions as you’re creating them, and improve them as you go based on user feedback.

A summative evaluation helps understand how a design performs overall. It’s usually done at the end of the design process to evaluate its usability or detect overlooked issues.

You can also use a summative evaluation to benchmark your new solution against a prior one, or that of a competitor.

Although summative evaluations are often quantitative, they can also be part of qualitative research.

Outcome evaluation research assesses the effectiveness of a design by measuring the changes it brings about in specific user behaviors and satisfaction.

This type of evaluation focuses on the short-term and long-term impacts on users, such as improved task completion rates or increased user engagement. Outcome evaluations help understand the real-world effectiveness of a product design and support data-driven decisions for future improvements.

The difference between generative research and evaluative research lies in their focus:

Generative and evaluative research are both valuable decision-making tools for researchers, and should both be used throughout the product development process to assess the product and get the evidence you need.

Evaluative research methods collect quantitative and qualitative user data through:

Generative research methods collect qualitative user data through:

Some research methods, such as UX surveys and user interviews, can be used in both generative and evaluative research, depending on the context and goals.

When creating your research plan, study the competitive landscape, target audience, needs of the people you’re building for, and any existing solutions. Depending on what you need to find out, you’ll be able to determine if you should run generative or evaluative research—or both.

“Evaluation research can start as soon as you understand your user’s needs,” says Mithila. Here are five typical UX research methods to include in your evaluation research process:

User surveys can provide valuable quantitative and qualitative insights into user preferences, satisfaction levels, and attitudes toward a design or product. By gathering a large amount of data efficiently, surveys can identify trends, patterns, and user demographics to make informed decisions and prioritize design improvements.

There are many types of surveys to choose from, including:

Some example questions for your next survey could include:

Unlike open or hybrid card sorting, closed card sorting uses predefined categories to evaluate specific usability issues in an existing design.

By analyzing how participants group and categorize information, researchers can identify potential issues, inconsistencies, or gaps in the design's information architecture, leading to improved navigation and findability. The results offer quantitative insights into users' mental models and expectations.

Closed card sorting allows researchers to:

Consider asking these questions when conducting closed card sorting:

Tree testing, also known as reverse card sorting, is a research method used to evaluate the findability and effectiveness of information architecture. Participants are given a text-based representation of the website's navigation structure (without visual design elements) and are asked to locate specific items or perform specific tasks by navigating through the tree structure. This method helps identify potential issues such as confusing labels, unclear hierarchy, or navigation paths that hinder users' ability to find information.

When comparing tree testing vs. card sorting:

Tree testing allows researchers to:

Consider asking these questions during or after a tree testing session:

Usability testing involves observing and collecting qualitative and/or quantitative data on how users interact with a design or product. Participants are given specific tasks to perform while their interactions, feedback, and difficulties are recorded. This approach helps identify usability issues, areas of confusion, or pain points in the user experience.

There are different types of usability testing, including:

Some usability questions to consider include:

A/B testing, also known as split testing, is an evaluative research approach that involves comparing two or more versions of a design or feature to determine which one performs better in achieving a specific objective. Users are randomly assigned to different variants, and their interactions, behavior, or conversion rates are measured and analyzed. A/B testing allows researchers to make data-driven decisions by quantitatively assessing the impact of design changes on user behavior, engagement, or conversion metrics.

Ask questions like these to understand the why behind user behavior in A/B testing:

Across UX design, research, and product testing, evaluative research can take several forms. Here are three examples of evaluative research:

This way to conduct evaluative research involves conducting usability tests with participants to compare the performance and user satisfaction of two or more competing design variations or prototypes.

You’ll gather qualitative and quantitative data on usability metrics like task completion rates, errors, user preferences, and feedback to identify the most effective design option. You can then use the insights from comparative usability testing to inform design decisions and prioritize improvements based on user-centered feedback.

Cognitive walkthroughs assess the usability and effectiveness of a design from a user's perspective.

You’ll create evaluators to identify potential points of confusion, decision-making challenges, or errors. You can then gather insights on user expectations, mental models, and information processing to improve the clarity and intuitiveness of the design.

Conducting diary studies gives you insights into users' experiences and behaviors over an extended period of time.

You provide participants with diaries or digital tools to record their interactions, thoughts, frustrations, and successes related to a product or service. You can then analyze the collected data to identify usage patterns, uncover pain points, and understand the factors influencing the user experience.

In the next chapters, we'll learn more about quantitative and qualitative research, as well as the most common UX research methods. We’ll also share some practical applications of how UX researchers use these methods to conduct effective research.