Continuous Research Report: Trends to Watch in 2023
Continuous Research Report: Trends to Watch in 2023
How product teams are informing decision-making through continuous discovery and research
In partnership with
With support from experts at
A look at the research industry
User insights are the driver for innovation. The best product organizations today are the ones that learn as they build, putting customer needs at the forefront of product development to deliver constant value and react effectively to changes in the market.
To better understand how product teams learn, we surveyed more than 600 designers, product managers, and researchers. What we found is that more and more product teams see the value of research as a powerful tool to inform decision-making at each stage of the product life cycle. Research as a discipline is becoming better established, but there are still challenges to conducting more frequent, better research.
At the end of the report, we identify three trends that will continue to shape the research industry moving forward, and we look at how Atlassian—a customer-driven product organization—adopts a continuous research mindset. We hope this report will inspire new thinking on how we can best empower entire product teams to continuously learn from customers and innovate with them.
Jonathan Widawski
Co-founder & CEO at Maze
About this report
We surveyed over 600 product professionals to find out how they conduct research to inform their decision-making. The majority of respondents who weighed in are designers (42%), followed by product managers (27%) and researchers (26%). Most of our audience works at SMBs and middle-market companies with 51-200 (26%) and 201-1000 employees (28%). Read more about the study methodology.
Key trends and results
Trend #1
Continuous research is becoming a well-established practice
83% of respondents say research should be conducted at every stage of the product development cycle
82% of companies in our audience have at least one dedicated UX researcher
78% of our audience believe their company doesn’t do enough research or could do more studies
78% of our audience believe their company doesn’t do enough research or could do more studies
83% of respondents say research should be conducted at every stage of the product development cycle
82% of companies in our audience have at least one dedicated UX researcher
78% of our audience believe their company doesn’t do enough research or could do more studies
83% of respondents say research should be conducted at every stage of the product development cycle
Organizations understand the value of research
82% of companies in our audience have at least one dedicated UX researcher. As Lucy Denton, Head of Design at Dovetail, highlights, "it's great to see how far the industry has come in terms of understanding the value of research. Companies are investing more in research which means they’re investing more in customer centricity and building for the customer.”
77% of our audience conducts a new research study at least monthly. In particular, 9% of respondents report running a new research study daily, 38% monthly, and 23% quarterly or yearly. Orgs are investing time and resources in research. Yet, there's still work to do to adopt a continuous product discovery mindset—as only 39% of product teams report having weekly touchpoints with their customers.
At what stages of the product development process does your team gather user insights?
Product teams are putting their customers first and are involving them continuously throughout the product development process
Product teams are involving customers from the start of the product development cycle and continuously throughout the process. In particular, 59% of product teams in our audience conduct research at the problem discovery stage, 57% at the problem validation stage, 55% at the solution generation stage, and 53% at the validation and testing stage.
These results align with what respondents say about the most impactful tasks in their job. These are understanding the customer journey (according to 48% of our audience), discovering unknown/unmet user needs (45%), and validating design assumptions (43%).
“Product organizations need to truly understand their customers to be competitive. With continuous research, they can constantly course-correct to work on the right problem and provide the right solution.”
Head of User Research at Pitch
Only 36% of product teams conduct research studies after launch
83% of product professionals in our audience agree that research should happen at all stages of the product development process, yet only 36% of product teams conduct research studies after launch. When product managers are involved in conducting research at organizations, the number of product teams running after-launch research increases to 44%, which indicates that PMs are a key driver of this type of research.
”Many research resources are being put into the design process, problem discovery, and solution testing, but not so much into post-launch reviews,” says Rachel Lynch, Research Manager at Productboard. “I think that’s a huge gap and a missed opportunity because you only know if something will work once you launch it. You should be using those same research tactics to try and figure out if you were successful once something has gone live.”
“We can use different approaches to research to minimize the guesswork before, during, and after we launch a product. Of course, there are constraints, but I would love to use more of the design research toolbox.”
Manager, product manager
“I feel we have so little knowledge about our users, how they interact with our products, and if we're building useful things that I would love to do testing more often.”
Individual contributor, designer
"At every stage, there are possibilities to gain new insights that can affect the solution or problem statement. Being able to see them coming and changing course is key to delivering quality results."
Individual contributor, designer/researcher
“You can always get user inputs and validate ideas through continuous discovery and research.”
Individual contributor, designer
“We can use different approaches to research to minimize the guesswork before, during, and after we launch a product. Of course, there are constraints, but I would love to use more of the design research toolbox.”
Manager, product manager
“I feel we have so little knowledge about our users, how they interact with our products, and if we're building useful things that I would love to do testing more often.”
Individual contributor, designer
78% of product professionals believe their company doesn't do enough research studies or could do more studies
Despite the high demand for research, there are several challenges hindering product teams from running more studies, the main ones being time limitations (according to 64% of respondents), budget limitations (46%), recruiting respondents limitations (31%), and tooling limitations (25%). Unsurprisingly, time limitation is even more constraining when teams don't have a dedicated UX researcher (78% vs. 64%).
“More research isn’t necessarily going to solve the problem of getting the right insights for decision-making," says Rachel Lynch, Research Manager at Productboard. "It's more about quality than quantity and feeling confident that we know what type of research should be used, when, and for what purpose."
64%
time limitations
46%
budget limitations
31%
recruiting respondents limitations
25%
tooling limitations
“We moved beyond trying to convince people of the value of research and now we’re trying to educate people about how to use research in the best way.”
Research Manager at Productboard
Trend #2
The industry is moving towards the democratization of research
One researcher to two designers to ten developers
64% of respondents report that their org has a democratized research culture
Three researchers to eight People Who Do Research
Three researchers to eight People Who Do Research
One researcher to two designers to ten developers
64% of respondents report that their org has a democratized research culture
Three researchers to eight People Who Do Research
One researcher to two designers to ten developers
The increased demand for research insights has outpaced the bandwidth of most research teams
To understand how product teams are organized, we asked respondents about the number of UX researchers, designers, developers, and People Who Do Research (PWDR) at their organizations.
Our data suggest that the most typical ratio in organizations is one UX researcher for two designers and ten developers. It’s important to note that our survey respondents are largely people who work in SMBs (51-200 employees) and middle-market companies (201-1000 employees) and that 82% of companies in our audience have at least one UX researcher.
If we compare the median number of dedicated UX researchers to the median number of People Who Do Research, we obtain a ratio of three UX researchers to eight PWDR.
Even though many orgs in our audience are investing in research, these results suggest that the high demand for user insights has outpaced the bandwidth of most research teams and that organizations are scaling the impact of research through democratization.
According to the audience we surveyed, in addition to researchers, designers (69%), product managers (54%), marketers (28%), and engineers (10%) conduct research at their companies.
64% of respondents report that their org has a democratized research culture
To the question "In your opinion, does your organization have a democratized research culture?" 64% of our audience answered "Yes."
While 37% of companies with one or more dedicated UX researchers say their UX research team is centralized, 63% of companies have a decentralized or hybrid UX research team. This shows that product orgs understand the value of having researchers embedded across teams and enabling those teams to run their own research.
“It's really encouraging to see so many crafts engage in research practice. While I strongly believe that dedicated research professionals are a necessary component for a successful tech company, there will never be enough researchers to cover the range and volume of research needs within an organization. We will always need our teammates to engage in some form of research—whether collaborating with a dedicated researcher or running their own studies.” - Caitlin McCurrie, Research Lead at Atlassian
User interviews and surveys are the most frequently used research methods
User interviews are the most frequently used research method among respondents, with 65% saying they have used them systematically or often in the past 12 months. This is followed by surveys (59%), product analytics (58%), competitive analysis (54%), and moderated usability testing (51%).
Product teams prefer to have a direct dialogue with their users. User interviews are the most used method, but we also see that moderated usability testing is more used than unmoderated usability testing (51% vs. 37%).
“I think these results depend on the method, the rigor, and how easy it is to be adopted by those who are doing research as a small component of their job," says Roberta Dombrowski, VP of User Research at User Interviews. “Many orgs lead with qualitative research first and 1-on-1 interviews when they’re introducing research to their teams, PMs, and designers, and that’s because it’s easier to talk to a customer—it's something we naturally do. It's much harder to do a full card sort study.”
Primary vs. secondary research
Qualitative vs. quantitative research
Evaluative vs. generative research
Qualitative research, primary research, and evaluative research are the most frequently used research types
The most frequently used research types are qualitative research (63% of our audience has used this research type systematically or often in the past 12 months), primary research (54%), and evaluative research (53%).
Primary vs. secondary research
When it comes to primary and secondary research, 54% of respondents frequently or systematically use primary research versus 46% for secondary research. The ratio is more similar among researchers (58% for primary research vs. 55% for secondary research).
A possible explanation is that researchers have a better understanding of where to find good secondary research sources and how to consume and synthesize existing data. When it comes to effectively scaling and democratizing research, improving access to primary and secondary research across the org is key.
Qualitative vs. quantitative research
Qualitative methods are more frequently used than quantitative methods, with 63% of our respondents saying they always or often use qualitative methods, compared to 51% for quantitative ones.
While qualitative and quantitative research yields different types of data, it's important to remember that these methods are not mutually exclusive and that using them in combination is often the best way to build the right solution.
Evaluative vs. generative research
Evaluative research is more common than generative research (53 vs. 33%). While 44% of researchers say they conduct generative research always or often, this percentage decreases to 28% for both designers and PMs. On the other hand, the percentage of respondents who frequently run evaluative research is 62% for researchers, 50% for designers, and 49% for PMs.
“Many teams often get their designers and PMs to run their own evaluative product research, while a researcher's skills are often better placed in doing more generative, strategic research. Research democratization is about enabling the entire product organization to run and observe research, analyze data, and consume the output of research—which creates a healthy way to include the customer's voice in product decisions.”
Head of Design at Dovetail
Trend #3
Continuous research enables more effective decision-making
Design, product, and engineering teams are the main consumers of research
74% of respondents believe research is partially effective (3) or effective (4) in determining decision-making
Product teams who conduct research weekly or daily report more effective decision-making
Product teams who conduct research weekly or daily report more effective decision-making
Design, product, and engineering teams are the main consumers of research
74% of respondents believe research is partially effective (3) or effective (4) in determining decision-making
Product teams who conduct research weekly or daily report more effective decision-making
Design, product, and engineering teams are the main consumers of research
Designers are the main consumers of research
Product organizations are not only doing research but are also using the findings to inform their decisions. We asked respondents which teams in their orgs rely on product research to make decisions. Unsurprisingly, design (74%) and product teams (67%) are the main consumers of research, followed by engineering (40%), marketing (34%), data (31%), sales (17%), and customer support 16%).
"Today, research is really focused on understanding the user experience,” explains Lucy Denton, Head of Design at Dovetail. "It would be interesting to see how companies evolve to focus more on the customer in other functions like marketing, sales, or customer support. They all talk to customers and need to understand the customers to do their job. I think they are doing research in different ways—they just might not call it research."
“Research at its core is learning. The more research you do and the faster you learn, the faster your decision-making will be.”
VP of User Research at User Interviews
Research positively impacts product adoption, activation, and customer satisfaction
According to our audience, product research has a positive impact on customer satisfaction scores (57%), product/feature adoption (55%), active users (46%), and revenue/profitability (42%).
When asked where product research has the most impact in their organizations, 25% of respondents selected product/feature adoption, 17% customer satisfaction scores, 14% active users, and 12% conversion.
74% of product people believe research is effective or partially effective in determining decision-making at their organizations
We asked our audience to rate how effective product research is in determining decision-making at their orgs on a scale from 1 to 5. When looking at the overall distribution of the effectiveness of decision-making, 74% of respondents report an effectiveness score of 3 (partially effective) or 4 (effective).
In addition, 14% of product professionals in our audience believe research is highly effective in determining decision-making, 9% say it's not effective, and only 3% think it's not at all effective.
As Rachel Lynch, Research Manager at Productboard, explains, "research is not just about what impact it has on your product metrics. It’s about the different decisions it will help companies make—from the big strategic directional decisions to improving usability and those behavioral metrics."
To better understand how orgs inform their decisions based on research findings, we asked respondents whether they agreed or disagreed with a number of statements on a scale from 1 to 5. Here’s what we found:
42%
of respondents agree that in their team, all product decisions are informed by data (agreement score = 3.2/5)
49%
of respondents agree that their organization makes strategic decisions based on product research findings (agreement score = 3.4/5)
60%
of respondents agree that recommendations from research findings inspire new product opportunities (agreement score = 3.4/5)
Product teams who conduct research more often report more effective decision-making
In our survey, we invited respondents to indicate how often their team conducts a new research study and to rate how effective research is in determining decision-making on a scale from not at all effective (1) to extremely effective (5).
When we correlate the frequency of conducting research vs. the effectiveness of research in determining decision-making, we find that product teams who conduct research more often, weekly or daily, report more effective decision-making compared to product teams who conduct research less often, quarterly or yearly (effectiveness score = 3.65 vs. 3.24).
Product teams who conduct research weekly or daily are more likely to report a high effectiveness score compared to product teams who conduct research monthly or less often (56% compared to 48% and 44%). This confirms that conducting research more frequently results in more effective decision-making.
On the other hand, product teams who conduct research quarterly or yearly are more likely to report a low effectiveness score than teams who run research monthly or more often (23% compared to 10% and 7%). This suggests that a lower frequency of research is correlated with less effective decision-making.
Effectiveness score: [1]: not at all effective, [5]: extremely effective
“Once you have more insights on a certain topic and more information about the pros and cons of the options you're facing, there’s a higher chance that the whole team knows more and has higher confidence in decision-making. These results are definitely a good sign that people are not only doing research but also using the findings to inform their decisions.”
Head of User Research at Pitch
The Atlassian Way: Research is a marathon, not a sprint
Written by
Caitlin McCurrie
Research Lead at Atlassian
Atlassian has a world-class research team, with nearly 100 researchers and a wide diversity of perspectives and backgrounds–from anthropologists to psychologists to survey experts and more. Even with such a large and ever-growing team, there are not enough researchers to address the many questions from the organization across our product development cycles. To extend the power of our research team, we collaborate with our cross-functional colleagues to conduct research as a team and empower Atlassians to conduct their own research when they might not have a research partner yet.
Despite such diverse backgrounds engaging in research, we’re consistent in how we orient our research process to the lifecycle of a product. Atlassian research is aligned with the Atlassian playbook for creating products or services: The Atlassian Way.
The Atlassian Way has four stages: Wonder, Explore, Make, and Impact. Across these stages, we continuously collect insights using different methods to keep the customer at the center of our decision-making across the development journey and build confidence that we are on track to deliver a quality experience.
Wonder
Insights here are critical to defining the problem space. Without a clear understanding of the customer needs or pains, you risk investing significant development time and resources into building a product that doesn’t deliver value. Typical research activities include semi-structured interviews, observational studies, and surveys.
Wonder
Explore
Research in this phase helps ensure that you create solutions that actually address your customer’s challenges. Conducting research, such as concept or usability testing, during your exploratory phase maximizes the likelihood you find the best solution that offers utility for your customers.
Explore
Make
It's important to assess the utility and ease of use of your final solutions since these are strong predictors of usage and adoption. Explore activities like usability studies or a diary study of your beta participants. Also, try to triangulate qualitative data with behavioral analytics to strengthen the quality of your insights.
Make
Optimize
Collecting insights at this stage will help you understand the “why” behind the success (or failure) of your newly released product. Atlassian researchers work closely with their analyst peers to offer context to better understand behavioral analytics and identify improvements for later iterations.
Optimize
Inspired to implement The Atlassian Way in your research practice? Whether you’re implementing your first research practice or optimizing the existing process, we have two key messages for you from our experience of maturing research at Atlassian.
Research is a team sport.
Product managers, designers, and product marketers all participate in research at Atlassian. While the researcher brings best practices, each team member brings their unique perspective to our insights and what they mean for the product we’re developing. We encourage all crafts to lead their studies. While this can feel intimidating for those new to research, not all studies need not be lengthy and complicated. For example, for Atlassians-who-do-research, we recommend exploring open comment fields of a CSAT survey, creating a poll in a community space, or walking through paper prototypes with five customers.
Research is a marathon, not a sprint.
Research isn’t confined to one stage of the product development process, such as discovery or validation after shipping, but is a marathon that extends across the entire product life cycle. Rather than thinking of individual studies that generate customer insights, we create continuous learning cycles that contribute to an ever-growing knowledge base of who our customers are and how we best serve their needs. The Atlassian Way ensures that we continuously deliver insights that allow teams to course-correct as they build.
Methodology and audience insights
The report survey was created using Maze and distributed between June 21 and July 31, 2022. During that period, we shared the survey on our Maze website, social media accounts, in our monthly newsletter, and via email to our contacts, who also shared the survey with their professional networks and in research and product communities.
We partnered with ADPList, who promoted the survey in their global community of over 8000 mentors and learners. We collected responses from 603 product professionals with experience conducting research (researchers, designers, and product managers). Other 564 respondents took the survey but did not qualify based on our criteria.
Which of the following most accurately describes your role?
Which of the following most accurately describes your role?
How many people work at your company?
How many people are in your product org?
Next Steps
1
Continuous research is here to stay
Product teams understand the value of research and the importance of building products with their customers. Yet, the biggest obstacle keeping companies from running more research is a lack of time. The challenge is no longer getting buy-in but how to conduct research more often and throughout the product life cycle.
1
Continuous research is here to stay
2
Democratizing research will help scale the impact of research
The demand for customer insights has outpaced the capacity of most research teams. As a result, companies are starting to amplify the impact of research by empowering more teams across the organization to access, collect, and consume research insights to make more informed decisions.
2
Democratizing research will help scale the impact of research
3
Conducting research early and often improves decision-making
Product teams report that research positively impacts product adoption, activation, and customer satisfaction and helps make more effective decisions, especially when conducted weekly or daily. We think research findings will continue to be a key factor in informing decision-making at each stage of the product life cycle.
3
Conducting research early and often improves decision-making
3
Conducting research early and often improves decision-making
Product teams report that research positively impacts product adoption, activation, and customer satisfaction and helps make more effective decisions, especially when conducted weekly or daily. We think research findings will continue to be a key factor in informing decision-making at each stage of the product life cycle.
3
Conducting research early and often improves decision-making
1
Continuous research is here to stay
Product teams understand the value of research and the importance of building products with their customers. Yet, the biggest obstacle keeping companies from running more research is a lack of time. The challenge is no longer getting buy-in but how to conduct research more often and throughout the product life cycle.
1
Continuous research is here to stay
2
Democratizing research will help scale the impact of research
The demand for customer insights has outpaced the capacity of most research teams. As a result, companies are starting to amplify the impact of research by empowering more teams across the organization to access, collect, and consume research insights to make more informed decisions.
2
Democratizing research will help scale the impact of research
3
Conducting research early and often improves decision-making
Product teams report that research positively impacts product adoption, activation, and customer satisfaction and helps make more effective decisions, especially when conducted weekly or daily. We think research findings will continue to be a key factor in informing decision-making at each stage of the product life cycle.
3
Conducting research early and often improves decision-making
1
Continuous research is here to stay
Product teams understand the value of research and the importance of building products with their customers. Yet, the biggest obstacle keeping companies from running more research is a lack of time. The challenge is no longer getting buy-in but how to conduct research more often and throughout the product life cycle.
1
Continuous research is here to stay
About Maze
Maze is the user research platform that makes products work for people by making user insights available at the speed of product development. Built for ease of use, Maze allows designers, product managers, and researchers to collect and share user insights when needed most, putting them at the center of every decision.
2023 Continuous Research Trends Report