Product experimentation: How to conduct and learn from experiments

By now, you have conducted product research to help inform the product strategy. You know exactly what your users’ pain points are and how the existing options on the market are solving them. It's now time to test your product hypotheses and determine the most effective solutions. Learn what makes a successful product experiment, when to run one, and how to implement the results.

Product experimentation is the process of testing different hypotheses for product improvement through quantitative and qualitative data. These improvements can range from small design and color changes to new features. You can run many possible experiments to optimize your product, including A/B tests, usability tests, multivariate testing, focus groups, or user interviews.

Product experimentation is a key part of the product development process. You can build a culture of product experimentation by running experiments continuously throughout the product lifecycle. The goal is to make small changes to your product’s design or functionality to ensure you’re always delivering the best possible experience.

To conduct successful product experiments, you should focus on:

After you’ve completed the product research process, consider conducting product experiments on your live product. Here's how to do it:

You need to set goals for your research and make sure they’re aligned with your product strategy and overall business objectives. Focus on running experiments that influence bigger goals. For example, if one of your business KPIs is to increase time on page, think about how your experiments put you closer to reaching the target.

Take a look at these example goals:

⛔ Implement a new website banner based on A/B testing results
⛔ Test three different ‘Learn more’ button placements on the mobile app

✅ Increase time on page by experimenting with different banner designs
✅ Increase the number of most qualified leads (MQLs) coming from mobile devices by testing multiple ‘Learn more’ button placements

The first two goals might end up driving an increase in time on page or MQLs, but unless you specify that aim, you may not be able to capture the real impact of your experiments.

To narrow down your results and keep your experiments on track, try setting a North Star goal and counter metrics.

For example, if your North Star goal is to reach 100 new daily sign-ups, you might use traffic rates, time on page, or 'learn more' button clicks as counter metrics. Pay close attention to your counter metrics while conducting experiments, as any significant alterations could result in a drop in your North Star metric.

Remember, hypotheses aren’t siloed ideas. They’re well-researched statements that you’re trying to validate or debunk. “Well-defined goals should guide the experimental design and analysis—and having a clear and testable hypothesis is crucial for meaningful results,” says Sonal Srivastava, Senior UX Researcher at Amazon.

When you formulate a hypothesis for your product experiments, come up with a clear description of what you expect to happen based on previous product, market, and user research. Here’s how to do it:

For example, a hypothesis could look like this:

If we move the 'Learn more' button to the top right corner, then we expect users to click on it more. That's because findings from usability tests showed that most users clicked on the upper right side of the website when asked to sign up for the newsletter.

Based on your goals, hypothesis, and resources, you should determine how you’ll conduct and measure your experiments. Use a mix of quantitative and qualitative data to make decisions. As Sonal explains, “Qualitative insights shed light on why users may prefer one option over the other, even if the quantitative results appear similar.”

The most common methods for product experimentation are:

You could design a low-fidelity paper sketch or a high-fidelity clickable design to run your experiment. Work closely with your design team to ensure previous user insights influence each variation. The method you're using for this experiment will determine the number of variations you'll test—and the teams that need to be involved in the design. Examples of potential experiment versions include:

Use prototyping tools like Figma, Sketch, Marvel, InDesign, or AdobeXD to create these clickable versions of the experiment. Then, bring those designs to a testing tool like Maze to conduct research on those experiments.

Netflix is an example of a company that is constantly testing innovations. Juliette Aurisset, Director of Product Experimentation at Netflix, explains that "if every single Netflix member is, on average, in 20 experiments and each experiment has three variants, that's 3.5 billion Netflix experiences." So, everyone goes through a unique Netflix experience, and you can do the same at a smaller scale with your product.

When testing your experiments, you want to get a representative sample of your actual and potential users. It's essential to determine who you'll be testing and how many people you'll need to ensure your experiment is viable. Try to gather a diverse group of research participants to avoid leading a biased study.

A good way to conduct legitimate research is by assigning tests randomly. “Randomly assigning users to different experimental conditions helps minimize bias and ensures a fair comparison,” says Sonal. “Sufficient sample size is important to obtain reliable and statistically significant results. And, employing appropriate statistical methods to analyze the collected data helps for accurate interpretation.”

In product experiments, unlike other types of research, you want to use your users as a primary source of information. So, make sure you conduct experiments using your current users as your audience.

Once you've developed your designs and know who you're testing, you can start conducting experiments. You can run studies on your live website or use mockups or simulations. Launch your experiments on your testing tool and let them run for some time before reviewing the data.

If you're testing your live website or product, make sure you review your counter metrics frequently to avoid letting experiments hurt performance. Also, avoid having all users test a new idea at the same time. Carrying out controlled experiments will guarantee that you don't affect the experience of most of your users in case they go wrong.

When you build a culture of experimentation, you can monitor different metrics depending on the various types of experiments. Choose metrics that will determine the success of your experiment and help you decide which solution works best for your audiences, all while keeping the bigger goal in mind.

"I look beyond the primary metric to consider secondary metrics that may reveal additional insights. Analyzing user behavior, engagement, or other relevant aspects often uncovers nuanced differences between the options," says Sonal.

When multiple diverse audiences use your product, you can experiment with algorithms to offer a personalized product and experience. As Sonal explains, "While overall results may be similar, variations can exist among specific user segments. Understanding these differences lets you tailor the product to better meet the needs of different user groups."

Once you've completed your experiments, review the results. "Based on user interactions, behavior, or other desired outcomes, use statistical analysis to compare variations and assess the experiment's impact on the chosen metrics," explains Sonal. Look at the metrics in context and review the results with statistical comparisons.

If the results aren’t conclusive, meaning they’re too similar, you can either:

Your penultimate step is to set up a meeting with stakeholders to go through your findings. Explain the goals, methods used, audience size, and results. Then, list all the action steps for each team. Open the room for feedback, and populate the backlog with new action items for sales, marketing, development, product, and research teams.

This phase is crucial to reflect on what you've achieved, keep all stakeholders in touch with the users’ preferences, and reinforce the importance of making user-centric product and business decisions.

When you follow a continuous product discovery approach, product experimentation isn’t a one-time thing. Remember the Netflix example: this company frequently conducts product experiments to keep innovating and offering the best product experience possible.

Continuous product experimentation becomes a competitive advantage and can help you grow or stay at the top of your industry by listening to your customers first and often. "Product organizations need to truly understand their customers to be competitive," says Xiangyi Tang, Head of User Research at Pitch on the 2023 Continuous Research Report.

Iterate and continue experimenting with your product to build solutions for your users. "With continuous research, product organizations can constantly course-correct to work on the right problem and provide the right solution," says Xiangyi.

Typically, you should do product experiments after you’ve conducted initial product, user, and design research. This way, you’ll have a better understanding of your target audience and confirmation that your concepts and solutions work. Now, it’s just a matter of identifying how and what works better.

Skipping previous research on your initial design or early minimum viable product can lead you to spend valuable resources testing ideas that aren’t aligned with your user base. Consider running experiments on your product when:

Running experiments on your product helps you create products that serve your customers better and drive superior business growth. Additional benefits include:

1. Increase customer retention rates: By continuously listening to customer feedback, innovating on your product, and taking the time to understand the customer’s journey, product experimentation can boost your retention rate

2. Gain a competitive advantage: Running product experiments allows you to stay one step ahead of your competitors as you constantly evolve based on market and users' demands and industry trends

3. Shorten decision-making loops: When you focus on the user and make evidence-based decisions after experiments, you speed up the decision-making time and take innovative solutions to the market faster

Product experimentation also allows you to grow your organization's research maturity by implementing a culture of continuous improvement where the user is front and center.

To test fast and often, you need a product discovery tool that can support your research needs. With Maze, you can recruit from a diverse panel of participants and use different research methods to collect high-quality customer insights.

Continue learning about product research and how to conduct competitive product analysis to gain insights about the market and improve your product.