Understanding confidence intervals for better user research insights

After spending considerable time and effort planning and conducting a user research project, the last thing you want to be is uncertain on the validity of your results.

Unless you’re conducting a study with large sample sizes, how can you be sure your findings apply to the vast majority of your users?

Confidence intervals offer a solution.

The concept is simple: you can’t interview all your users, so you conduct research with a manageable sample size, then use confidence intervals to validate the accuracy of your findings when they’re applied to a larger population.

Read on for the specifics of how to calculate confidence intervals in your own research.

A confidence interval is the probable range for the true score of an entire population (in most cases, your product user base). When conducting UX research, you can’t study all your users—it’s just not feasible.

The confidence interval formula presents the probability that the findings from a small dataset will hold true when applied to a large sample.

Confidence intervals come in two parts—a confidence percentage (the probability findings will hold true), and an interval—with an upper and lower limit—that you can confidently say most users fall within. In addition to, you’ll have a confidence level which dictates how accurate your interval is (more on this below).

For example, say you're testing a new website interface and want to discover how quickly users can find Feature A on the site. You choose a small sample size of 40 users to test with. Your analysis concludes that users take 50 seconds on average to find Feature A.

Using a confidence interval formula, you calculate with 95% confidence that the average time for all users to find this feature is between 45 and 55 seconds.

When calculating confidence intervals, it's important not to confuse them with confidence levels. Let's break down the two concepts in more detail to clarify the differences.

The biggest difference is that confidence intervals are probabilistic; they tell you the likelihood that your findings remain accurate when applied to a larger population—there's no guarantee that the interval will include the true mean, but there's a good chance it does.

This chance depends on the confidence level—if you're 95% confident the true mean is within the range, there’s still a 5% chance that it’s not.

If you wanted to analyze how long it takes users, on average, to complete the checkout process on an e-commerce store. You collect sample data from 100 users and discover that it takes eight minutes to complete this process.

Your analysis presents a 95% confidence interval of 45–55 seconds. That means you can confirm with 95% certainty that the average time for the entire population is 45 and 55 seconds.

On the other hand, confidence levels convey how sure you can be of finding the true mean within the interval you’ve calculated. It’s the percentage of times you’d expect to get the same or similar interval estimation if you run the experiment with different samples or resize the population.

For example, if you repeat the checkout process analysis several times, an 80% confidence level means you'll get the same or similar average checkout time in 80% of all experiments.

Confidence intervals make UX data analysis more reliable. They indicate the variability of your mean value if you repeat the study. In doing so, these intervals show a realistic representation of your results, and how your mean will fluctuate within the population.

Let's look at how confidence intervals improve UX research.

UX researchers work with small sample sizes because it’s not realistically possible to study all users. Confidence intervals predict how accurately your findings from a small dataset apply to the entire population, revealing an accurate representation of real users. The interval also prevents overgeneralization and cognitive biases based on average values, since the range of values shows how user behavior differs.

Confidence intervals acknowledge that users don’t behave identically and there’s variation in how users perform certain actions. UX teams can better understand and prioritize user needs based on this variability. Confidence intervals paint a picture of how all users experience or interact, ensuring you’re building and implementing solutions that work for everyone.

A narrow confidence interval means your estimate is more precise and reliable. You can add more credibility to your research by adding this interval. It shows stakeholders that you’re not relying on an average point estimate. Instead, your findings are based on a range of data points and apply to a wider population.

Another huge benefit of finding confidence intervals for UX research is to make more informed, data-backed decisions. You can capture a range of user behavior and responses to any decision. As a result, you can make more nuanced UX design choices based on the breadth of user responses to every product and update.

The bottom line: Confidence intervals offer clarity into how applicable your findings are to your entire user base. You can apply this understanding to make strategic decisions and improve the user experience.

You’ll commonly see a 95% confidence interval for UX research. But is this the confidence interval to aim for?

A 95% confidence interval means there’s a 95% chance the true population mean is within the given interval, with only a 5% chance that it doesn’t include the real mean.

This sounds good on paper, but it’s important to know that there’s no ‘ideal’ confidence interval in UX research. Sometimes you’ll want a smaller confidence interval, sometimes you’ll want a larger one. The range will vary, depending on the context of your research.

Let’s consider when it’s ideal to have a smaller and larger confidence interval.

A smaller confidence interval gives you a narrower range of values which contain the mean value. This indicates higher confidence in the accuracy of the estimate.

For example, a confidence interval of 78-82% gives just a 4% range, which means researchers are confident the true mean value is between the respective upper and lower limit. In contrast, a confidence interval of 70-90% has a 20% variance, so is less precise and shows lower confidence.

A smaller confidence interval is good when you want to:

A large confidence interval is broader and suggests lower confidence in the accuracy of the interval estimation. However, it’s still useful in different scenarios in UX research.

UX researchers leverage confidence intervals for making design decisions based on user data—intervals can eliminate guesswork and give you a realistic estimate of your population to make informed product decisions.

Alongside UX, confidence intervals are used across the board for different business needs, such as:

Let’s face it: you can’t study every user as a part of a research survey or interview. Since it isn’t always possible to study a large group of users, due to time and budget constraints, product teams rely on confidence intervals to see how their findings from a small sample may apply to the overall user base.

In the context of UX research, confidence intervals present more accurate data to inform design roadmaps, build reliable UX reports and make decisions.

Now we’ve covered the essentials of confidence intervals in UX research, let’s break down the exact steps for calculating them.

Start by collecting and organizing data from your research study. You can use any quantitative research method to gather data on your research subject (bear in mind that you can’t calculate confidence intervals for qualitative information).

The most important point to note in step #1 is building your research sample. You want to choose participants that represent your overall user base, instead of focusing on any specific segment.

Platforms like Maze Panel make it easy to recruit research participants for your study, and Maze Reach enables you to build a database of testers that you can return to—this can be helpful in validating your confidence level

Once you’ve got your testers, it’s time to run the research. There are ample user research methods and ways to get customer insights. Consider your needs and choose a UX research method that gets the quantitative data you need, such as usability testing or user surveys.

After collecting your data, it’s time to find the sample mean (x̄). This is the average score of all the participants in your sample size.

You’ll use this figure when calculating the confidence interval for larger samples—it represents the average experience or response of your sample group.

You can find the sample mean by summing up all the scores in your survey and dividing by the number of responses.

Now, you need to calculate the standard deviation (s). The standard deviation explains how spread out the scores are around the mean—helping understand the variability in your data.

To do this, you have to subtract the sample mean (x̄) from individual scores/responses and write the square root of the difference for each data point. Then, add all the squared differences and divide this total by the number of scores minus one (N-1) to find your standard deviation (s).

The next step is to choose your preferred confidence level (typically between 90% and 99%) and find its corresponding z-score value. You’ll find this value in a standard z-table. For example, a 90% confidence level corresponds to a z-value of about 1.645.

You’re almost at the summit of the math mountain we’ve just climbed! Now, it’s time to put all these values in the formula to calculate the confidence interval.

This formula calculates the range of values that’s likely to include the true average for the entire population.

The final step is to interpret your results and contextualize the confidence interval within your research.

For example, by now you might know that it takes six–eight minutes on average to complete the checkout process on your product, with a 95% confidence level. But you also need to communicate this finding in a clear and meaningful narrative.

Here’s what you need to know to effectively convey the results:

Remember to document these findings for every research project and store them in a research repository for future reference.

Calculating confidence intervals is an intimidating task, but it’s critical to the reliability of your research. Being able to effectively estimate the results of a wide user base helps you validate your research at scale, without need for huge budgets or endless time.

If you’re ready to calculate confidence intervals, kick-start your next research study with Maze to find participants effortlessly.

Reach allows you to conduct a highly targeted search with 400+ filters to find ideal candidates aligned with your research theme, then send personalized invites and manage all participants in a database—like a CRM for your UX research.

You can then use Maze’s suite of research methods to conduct your research—whether it’s usability testing, prototype testing, user surveys, or more—to start calculating confidence intervals and understanding the reliability of your research.