Evaluative Research: Definition & Methods
In the last chapter, we learned what generative research means and how it prepares you to build an informed solution for users. Here, we look at evaluative research and the benefits of this type of research.
What is evaluative research?
Evaluative research, also known as program evaluation or evaluation research, is a type of research you can use to evaluate a product or concept and collect data that helps improve your solution.
“Evaluative research should definitely start early–before anyone gets too attached to a specific design or implementation of a concept.”
A vital part of the product development process, evaluative research is typically introduced in the early phases of the design process to test existing or new solutions and continue to be employed in an iterative way until the product becomes ‘final’. According to Nannearl LeKesia Brown, Product Researcher at Figma, “With evaluative research, we’re making sure the value is there so that effort and resources aren’t wasted.”
However, evaluative research doesn’t stop when a new product is launched. For the best user experience, solutions need to be monitored after release and improved based on customer feedback. For Nannearl, this ensures the solutions continue to add value and meet user needs:
“There may be times when I'd want to do research on a solution that has already been made available. For example, if my team created a website, after a certain amount of time, it’d be necessary to make sure that this site is still achieving the team’s goal or to find an even better way for it to achieve that goal than it already does.”
According to Mithila Fox, Senior UX Researcher at Stack Overflow, the research process evaluation includes various activities, such as testing for accessibility, testing the content, assessing desirability, and more. Mithila also mentions that evaluative research can also be conducted on competitor products to understand what solutions work well in the current market, before you start building your own. She explains:
“Evaluative research can start as soon as you understand your user’s needs. Even before you have your own mockups, you can start by testing competitors or similar products. There is a lot we can learn from what is and isn't working about other products in the market.”
Balancing evaluative and generative research
Use generative research to bring forth new ideas during the discovery phase. And use evaluative research to test and monitor the product before and after launch.
Generative and evaluative research are both valuable decision-making tools in the researcher’s arsenal. They should be similarly employed throughout the product development process as they both help you get the evidence you need.
When creating the research plan, study the competitive landscape, the target audience, the needs of the people you’re building for, and any existing solutions.
Depending on what you need to find out, you’ll be able to determine if you need to run generative or evaluative research. However, both methods will be useful throughout the research process to get different types of evidence.
Mithila from Stack Overflow explains the benefits of using both research methodologies:
“Generative research helps us deeply understand our users and learn their needs, wants, and challenges. On the other hand, evaluative research helps us test whether the solutions we've come up with address those needs, wants, and challenges.”
Types of evaluative research studies
Run formative research to test and evaluate solutions during the design process, and conduct a summative evaluation at the end to evaluate the final product.
There are two types of evaluative studies you can tap into: summative and formative research. Although summative evaluations are often quantitative, they can also be part of qualitative research.
A summative evaluation helps you understand how a design performs overall. It’s usually done at the end of the design process to evaluate its usability or detect overlooked issues. You can also use a summative evaluation to benchmark your new solution against a prior one or that of a competitor’s and understand if the final product needs assessment.
On the other hand, formative research is conducted early and often during the design process to test and improve a solution before arriving at a final one. Running a formative evaluation allows you to test and identify issues in the solutions as you’re creating them, and improve them based on user feedback.
Choosing an evaluative research method
Part of the research design process is choosing the type of evaluation that best suits your project, and then determining which research method you will use. These can be anything from usability testing to focus groups, but it's essential first to identify what fits the project.
Mithila and Nannearl rely on evaluation methods such as usability testing, surveys, A/B testing, card sorting, and more to test the features that the product teams at Figma and Stack Overflow build.
Surveys help with data collection and gathering useful feedback from users and tap into a large volume of responses for detailed data analysis. Mithila, Senior UX Researcher at Stack Overflow, explains the value of survey research as part of the evaluation process:
"Surveys can help test your users’ specific feelings on a certain feature, and if you continue to run them continuously, you can establish a baseline of user satisfaction. By understanding this baseline, you will be able to see how the changes directly impact your users."
Other types of evaluation include usability testing, which allows you to test your product with real people by having them complete a list of tasks. Check out this complete guide to usability testing to learn more.
For Nannearl, Product Researcher at Figma, selecting the correct method goes back to coming up with the right research questions and hypotheses at the beginning of the research project. She explains:
“This is the value of having a UX research plan before diving into the research approach itself. If we were able to answer the evaluative questions we had, in addition to figuring out if our hypotheses were valid (or not), I’d count that as a successful evaluation study. Ultimately, research is about learning in order to make more informed decisions—if we learned, we were successful.”
No matter the method used, the important thing is to test with people who are part of your target audience:
“Regardless of the type of evaluation, my goal is to help my team build confidence in what we decide to make available to our users. If the update isn’t understandable or useful to those we’re striving to help, then it’s really not worth the time or effort at the end of the day.”
In the next chapters, we'll learn more about quantitative and qualitative research, as well as the most common research approaches, and share some practical applications of how UX researchers use them to conduct effective research.