Special thanks to the many talented researchers and leaders who shared their experiences with me for this article: Kurt Blancaflor, Jenna Hammer, Alison Meier, Cristen Torrey, and Andrew Warr.
In the current age of belt-tightening in tech, with all disciplines facing unprecedented layoffs, research teams are looking for ways to better prioritize, streamline, and demonstrate impact.
UX research teams have historically been responsible for conducting all types of research that drive product development, including evaluative and generative work—both of which are necessary and valuable for making products that serve users well.
But could we improve on this status quo?
In this article, I’ll explore whether there may be more efficient or successful ways to organize generative and evaluative research in a product organization. How might researchers better serve the needs of product teams, improve the impact of our research, and ultimately help create more user-centered products?
A recap of generative vs. evaluative research
One of the fundamental ways we categorize UX research is generative versus evaluative. As Erika Hall, co-founder of Mule Design Studio, puts it, “Generative research dives into ‘What problem might we solve?’, while evaluative research asks ‘How well is our solution working?’”.
Generative research (also known as strategic, foundational, or exploratory research):
- Enables a deep understanding of people’s motivations, needs, and behaviors
- Is used primarily to identify opportunities for the business and inspire innovation
- Can be illuminating for all stakeholders in product development, but targeted more at leadership
- Common methods include user interviews, contextual inquiry, and diary studies
- Examples include persona development, Jobs To Be Done, user journeys, etc.
Meanwhile, evaluative research considers how people engage with a particular product or solution:
- Researchers observe and ask questions to participants
- The goal is to improve a product or solution to better meet user needs
- Helps product teams make product decisions with confidence
- Illuminating for all, but targeted at designers and PMs making tactical product decisions
- Common methods include usability testing, concept tests, A/B tests and content testing
Simply put, generative research is used to understand people, while evaluative research is used to gather feedback on a solution.
Both types of research are crucial to user-centered product development—and traditionally, both are conducted by an organization’s research team.
The two research types are certainly related; the skill sets overlap quite a bit, but they are fundamentally different activities—with different methods, timeframes, stakeholders, and goals. Is having a single team conduct both types of research the most efficient way to get this work done?
Increasing capacity for evaluative research
I’ve spent the last five of my 23 years in the industry scaling research, developing democratization programs at Dropbox and Airtable.
Towards the end of my tenure at both Dropbox and Airtable, the respective Heads of Research asked me to expand my work beyond democratization, to oversee evaluative research more generally.
I wasn’t looking to move beyond the democratization programs I ran, but the need for more evaluative research resources emerged organically at each organization. Both companies had relatively high research maturity, with excellent, well-resourced teams of embedded and specialized researchers. And yet, we were still releasing experiences based on inadequate research.
Something wasn’t working.
We needed to not only expand our capacity for evaluative work, but our efficiency, and our focus. So I began work to create an internal evaluative research studio, staffed by contract researchers who would devote all their time to evaluative studies.
Seeing this need for more evaluative research come up organically at two different organizations led me to question the norm I had seen throughout my career.
Why is it that the same research team is usually responsible for both generative and evaluative research? Why are some embedded researchers disinterested in evaluative research? And could specialization help organizations increase their research capacity?
The evaluative–generative tension
I spoke with several researchers and research leaders in preparation for this article, and they all agreed that evaluative research is valuable and impactful, but companies aren’t doing enough of it. Why is this?
Many researchers buy into the narrative that generative work enables innovation, while evaluative work is less impactful. Some professional researchers can be dismissive of evaluative work. They don’t think of it as interesting, fulfilling, or impactful—instead, they view it as a distraction from the ‘more important’ strategic and generative research they really want to pursue.
I think that’s a huge oversimplification.
Evaluative research empowers product teams to make tactical user-centered design decisions in the short-term, and leads to visible results by moving metrics and/or revenue. Evaluative impact is often immediate and tangible. Ultimately, it’s much easier to draw a direct line between evaluative research and product impact—for example, an A/B test that reveals conversion rate increases with a change of button placement.
Generative research can have huge influence on product strategy, when a company or team is ready for it and knows how to use it. The impact can be massive, but it typically takes longer to manifest, and is harder to track.
So, on the one hand, we have researchers looking to sink their teeth into bigger-picture, strategy-shaping research. And on the other hand, we have product teams consumed by the urgency of the next release, who want to meet KRs and ship more confidently—naturally leading them to prioritize and value evaluative research.
Across the board, research leaders concur that evaluative work is important and impactful, but their teams aren’t doing enough of it. Orgs want more evaluative work to get done, but they have a hard time convincing researchers to spend time on it.
Christopher Nash
Staff Researcher & Research Program Manager
Share
Ultimately, the tension comes from differences in priority, resource, and interest. Each kind of research is important, valuable, and can be impactful. Product teams need plenty of both in order to succeed: generative to identify business opportunities, and evaluative to improve product solutions.
So, how might we better achieve that?
How to bridge the evaluative research gap
Many companies have experimented with ways to address evaluative research needs—I’ve interviewed research leaders at companies including Airtable, Dropbox, Figma, Google, Instagram, Microsoft, Uber, and YouTube—they’ve all found the same need for additional evaluative capacity.
Some of the models they tried include:
- Rapid research programs focused on evaluative work
- UX research teams who only did strategic research as a matter of policy
- Armies of contractors who only did evaluative studies
- Evaluative studies farmed out to vendors
- Individual researchers who specialized in evaluative work
- Democratization programs focused on evaluative work
- Democratization programs focused on generative work, reserving all evaluative work for the professional researchers
Yet no one felt they have solved the problem. They saw some success, and learned a lot along the way, but no one was fully satisfied with their solutions.
Perhaps they haven’t gone far enough. Why not fully segment research into separate teams that can each fully focus on evaluative or generative research?
What would that look like in practice?
Creating a targeted research team
In this scenario, there are two separate teams dedicated to each type of research—generative research works to influence company strategy, while evaluative research supports more tactical product decisions.
For the sake of simplicity, I’ll refer to these hypothetical teams as ‘Strategic Research’ (generative) and ‘Design Research’ (evaluative).
There are multiple ways teams might organize themselves. Here are two options:
- Design Research and Strategic Research are specialized subteams of the UX research team. While Strategic Researchers are deeply embedded on product teams, Design Researchers could execute on evaluative studies across the company.
- E.g. Design Research could take research requests from all disciplines. Any PM or PD or PMM could request research support from this team directly.
- Or Strategic Research could be the primary ‘clients’ of the Design Research team. Embedded researchers would determine what evaluative studies should be done, and pass them off to the Design Research team to execute.
- Design Research and Strategic Research are completely separate teams. The forward-looking Strategic Research team reports to a high-level Product Lead or is allied with a BizStrat team, keeping them closer to company strategy. Meanwhile, the Design Research team reports to the Head of Design, integrating them deeply with iterative design processes and continuous discovery.
The benefits of specialized research teams
In addition to efficiency, there are multiple potential benefits to a specialized team structure, including:
- Improved user experience: If more evaluative work gets done, fewer experiences will launch without usability testing or other user feedback. This in turn leads to better UX, more satisfied customers, higher engagement, decreased churn, and increased revenue.
- Decreased cognitive bias in evaluative studies: There are real benefits to having a fresh set of eyes evaluating a product. An evaluative researcher who wasn’t involved in developing a product idea will be more objective and better equipped to anticipate and test for problems that users may encounter.
- Improved alignment to skill sets: A successful generative researcher must develop strong ongoing relationships with their stakeholders and become an integral part of developing the product roadmap. Meanwhile, folks drawn to evaluative work tend to focus on unblocking teams. They are more systems-thinkers and educators—people who are driven more by process and immediate impact. In specialized teams, researchers can focus on their preferred type of research and hone those skills.
- Increased career fulfillment and development: A Design Research team would be an excellent training ground for early-career researchers. The rapid pace of iterative evaluative research provides ample opportunities to develop and hone research skills. The team could serve as a development tool, training younger researchers who can then go on to join the Strategic Research team in the future, if that’s the path that they want.
Considerations for specialized teams
Of course, implementing a new UX team structure doesn’t come without challenges. Before making the leap and splitting your team, there’s some important considerations.
To have the smoothest transition for your research team, here’s some potential pitfalls (and how to avoid them):
- Make your knowledge management airtight: Creating, maintaining, curating, and championing research repositories is already a major challenge for research teams. If insights are being developed by multiple research teams at additional pace, this could exacerbate these challenges. Work with your research operations team to ensure any research libraries, guidelines, or best practices are updated before the changes come into place.
- Give evaluative specialists time to ramp up: When an embedded researcher embarks on an evaluative study, they already have a ton of context about the users, use cases, the product team’s goals, and the politics and personalities within the product team. The first time you bring in an evaluative specialist who is not embedded, allow additional time for them to establish the context necessary to succeed.
- Circulate insights from both teams: Both evaluative and generative insights inform context for understanding one another. By dividing researchers into strategic and evaluative teams, you run the risk of improving efficiency—but creating siloes. Build in time to consciously share insights and research reports, and make knowledge-sharing a valued ritual for each team.
- Invest in your ReOps efficiency: Increasing the number of evaluative projects means significantly increasing your participant recruitment efforts, from scheduling to payment and tracking. Doubling down on evaluative work might also impact the types or number of UX research tools a team needs. It’s all going to put extra stress on your Research Operations function. Spend time understanding your ReOps processes, and invest in additional capacity to reflect the new workload.
Specialized research teams could build a stronger future of research
Ultimately, generative, strategy-focused research, and analytical, evaluative research are both important, and neither type can do its job fully without the context of the other. We can evolve our organizational structures to better support both disciplines.
In the current economic climate, efficiency is crucial to the success of research teams everywhere. Sorting our practice into specialized teams, focusing on their respective strengths, is one way we can encourage more research at increased efficiency, while creating a streamlined, more targeted research function at the same time.
By expanding research capacity, honing focus, and allowing researchers to pursue their chosen discipline—we may better serve researchers, product teams, and users in the long run.
