Chapter 10

Usability testing questions: How to write and ask effective UX questions

Asking the right questions in usability testing is vital. In this chapter, we share examples of usability testing questions to ask before, during, and after the test, and best practices to keep in mind to get effective results.

There are two ways to gather info with a usability test: observing participants completing pre-defined tasks, and asking participants usability testing questions.

While the former serves as the core of your usability test, questions are a great way to get feedback on specific elements and pain points of your design. Often with user testing, there are one or two aspects in a test prototype that as a designer you’re particularly keen on validating. Asking questions can prompt a user to reflect on certain parts of the user experience and voice their feedback.

And outside the test itself, usability survey questions allow you to know the person behind the user. Which demographic they’re in, how familiar they are with technology, and what they think of your product overall are useful bits of info that add context and detail to your results.

TL;DR
If you can't wait to get to the questions, here's the kind of research questions you'll want to ask during usability testing:

  • Background questions about demographics and product useage
  • Questions about the usability tasks
  • Specific questions about the design, navigation, language
  • Reflective questions on the test and tasks
  • Questions on overall opinions
    Let's get into some examples.

What’s the goal of usability testing questions?

Like with all research and testing, usability questions help you understand the what, why, and how of your product—and its user. Some of the ways usability questions can help are:

  • Uncover errors or bugs
  • Gain feedback from real users
  • Understand your audience
  • Learn whether the product works as expected
  • See how user-friendly your product is

Ask effective usability questions with Maze

Maze is a remote user research platform that enables you to run surveys, validate design ideas, and test your prototypes.

All this is vital for getting the most out of each testing session. Since finding participants is always a challenge, every test counts.

So even if you can’t test as early and often as you’d like to, asking good questions is a path to getting the insights you need. In this chapter, we’ll break down what to ask, when to ask it—and most importantly, how to ask usability testing questions in a way that gets accurate results.

Let’s start with pre-test questions.

Questions to ask before the usability test

Finding participants for user testing can be tough. So it’s tempting to take people up on their offer to participate as soon as you find them.

But remember: not all test participants will give you equally relevant results. One usability testing best practice is to test people who resemble your target audience, as this will give you an accurate idea of how to improve your design from real potential users.

To find out more about your participants, you have two types of user testing questions at your disposal: demographic questions and background questions.

Demographic usability testing questions

Demographic info gives your results more context, allowing you to spot usability trends across different demographics, ages, nationalities, genders, income groups, etc. Ask carefully though—demographic questions can touch on sensitive topics for some people. Here’s a few examples of well-worded demographic questions to ask as part of your user research.

  • What age group are you in? 18-24, 25-30, 31-40, etc.
  • How do you describe your gender?
  • What’s your relationship status?
  • What’s your household income? (Provide ranges as options)
  • How do you describe your ethnicity?

Asking demographic questions carefully avoids making assumptions about people by mistake. It also gets you the context you need—without making people uncomfortable with overly direct or specific requests for personal info. Getting off to a positive start will help your participants feel relaxed when it comes to taking the test.

Background usability testing questions

Apart from finding out demographic info, it’s also smart to ask some screening questions related to people’s product habits and preferences. If someone’s using your competitors’ products, for example, you should take this into account when analyzing their results. This kind of info is also useful when thinking of post-test questions to ask specific people.

On the other hand, a participant who’s completely new to your product might spot usability issues in your design that are intuitive to long-time users. So it’s important to find out before you start the testing session.

Here are some background questions to ask your participants:

  • How often do you use our product?
  • Which features do you use most?
  • How experienced/comfortable are you with using an application to do [X action]?
  • What type of product do you use to do [X action]?
  • Have you used any products in the [X industry] before?
  • How often do you use [X type of product] to do [X action]?
  • What device do you normally use for [X action]?

By deploying the right demographic and background questions pre-test, you make sure you choose the right test participants, and you’ll have more information with which to analyze your results afterwards. This can help explain potential anomalies in results, and also gives you info to make your design more accessible to a wider variety of people.

Questions to ask during the usability test

Asking questions during a usability test helps pinpoint design issues by probing a little deeper. However, it’s also important not to ask too many questions so the participant can complete the test with minimal distraction. To avoid having to rely on questions too much, make sure you write some great usability tasks.

The number one rule of usability test questions is no leading questions. A leading question influences that participant to respond in a certain way, skewing your results. Instead, you should carefully word your questions so they’re neutral and open.

Here’s a few examples of questions you should not ask during a usability test:

  • What made completing this task a good experience?
  • How simple and clean was the interface?
  • Was the language in the mobile app clear and straightforward?

By using words like ‘simple’ and ‘clear,’ you can unintentionally plant ideas into the participant’s head. So avoid adjectives whenever possible, and phrase questions in a way that invites participants to share their thoughts openly.

Here’s how you should phrase the questions above:

  • How was your experience completing this task?
  • What do you think of the user interface?
  • How is the language used on this page?

Notice that as well as not leading the participant, the questions are open-ended. Exclusively asking questions that require a ‘yes’ or ‘no’ answer isn’t ideal for a couple of reasons. First, your results will lack the detail and depth you need to make improvements—if a participant is struggling with specific tasks, you need to know why. Second, people might be tempted to answer ‘yes’ just to avoid having to explain themselves, even if in reality they’re having problems.

As inspiration, here’s some more examples of well-written questions to ask during the test:

  • What’s your opinion on the way features and information are laid out?
  • How did you find navigating to [X] page?
  • Which parts of the website do you use most often?
  • What did you think of the explanations on the page?

Taylor Palmer, Product Design Lead at Range, shares examples of usability testing questions they ask during a session:

  • What do you think [feature] is trying to communicate to you?
  • How have you seen [content] presented in other diagrams, domains, or standards?
  • Based on the previous task, how would you prefer to do this action instead?
  • Thinking about how your team likes to work, how would [feature] change the way you work, if at all?

The features we test are often data-driven and highly technical. Because of that, we’re usually asking about a participant's understanding of the content, how they would like to adjust it, and their ability to navigate it.

Taylor Palmer, Product Design Lead at Range

Taylor Palmer, Product Design Lead at Range

In-person moderated vs. remote unmoderated usability testing

While every usability study used to take place in person, these days technological advances have made unmoderated and remote usability tests a viable way to get results fast.

The type of questions you should ask are generally similar regardless of your testing process. But there are a few things you should be aware of depending on the approach you choose.

Asking usability testing questions in a moderated test

If you’re doing a moderated usability testing, you have the opportunity to follow up on anything the participant does that you find interesting. Use this to your advantage by asking specific questions based on user behavior:

  • I see that you [X]. Can you explain why you did that?
  • Did you notice that there was an alternative way to do [X]? Why did you go with that option?
  • You seemed to hesitate on the last step. What was going through your mind?

However, just because you’re moderating the test, that doesn’t mean you should bombard the participant with questions throughout the test. It’s important to let the user feel relaxed and complete the test in their own time, so test conditions more accurately reflect a natural situation.

Kara Pernice from Nielsen Norman Group recommends following these steps when considering whether to speak to a user:

  • Figure out if the user was actually addressing you with a question, or if they were just thinking out loud
  • Decide if you can tell what a mumble or noise means, or if a proper explanation is needed
  • Weigh up whether asking further questions will actually get more useful info, or if observing the user is enough to form a conclusion

As a general rule, take your time working out exactly what you want to ask someone, how to phrase it correctly—and whether it’s worth asking at all. Jumping in at the wrong time could catch the user off-guard, potentially compromising their progress for that part of the test.

Writing usability testing questions for unmoderated tests

On the other hand, the unmoderated approach requires questions to be written in advance. It’s important to test your questions on people before you send them to users, as you won’t be there to clarify if people get stuck.

And while you can still ask some open-ended questions, providing scales or multiple-choice options for people to respond prevents participants from having to write long answers. Here’s a few examples:

  • Which of these two approaches to completing the task did you prefer? (Multiple choice)
  • On a scale of 1 to 10, how was your experience with the interface? (1 = not user-friendly at all, 10 = super user friendly)
  • How was the process of making an order? (Super unintuitive, unintuitive, intuitive, super intuitive)

Finally, using simple and straightforward language is always best practice with any usability test—but it’s especially important when you’re writing questions for unmoderated testing. If people interpret a question differently, it’ll impact your results in an unpredictable way.

So avoid jargon and remove any internal placeholder terms your design team has been using. And if in doubt, ask a copywriter.

What if the research participants ask you a question?

Sometimes your test users may ask you a question before, or during, the usability test—how (and whether) you answer any questions is crucial, to avoid accidentally providing one research participant with information that others won't have.

Questions may be about the practialities of the test, like 'how do I select my answer?' or 'how long is left?'—while answering questions like this may seem harmless, you risk impacting how the user progresses through the test compared to others, who don't receive the same advice. This is why it's critical that you prepare for your usability test with clear, detailed instructions that answer any questions users might have, and provide them with all the information they'll need.

If you're using a usability testing platform, this can also mitigate questions about completing the test itself, as a good testing tool will have built-in tool tips or help functionality that's accessible to every participant.

Other questions from users might be more in-depth, such as asking for additional detail on the product, your opinion or guidance. In this situation, it's even more important that you assert a boundary and avoid answering with anything that could impact the test results or create bias. It's worth coming up with a standard answer you can give to all participants if questions come up, such as:

'I'm afraid I can't help you with that as it might impact the test, but please follow the instructions and do what you think is best."

Questions to ask after the usability test

Once the participant has made it to the end, it’s a good time to ask broader follow-up questions about their overall experience. Since you’ve already asked about specific design details as they came across them, these questions can be more open-ended and could even include opportunities for the user to suggest improvements.

Here’s a few examples:

  • What did you think of the app/website overall?
  • What did you like the most/least about this product? Why?
  • How would you describe your experience with this product?
  • Would you use such a product to do [X activity] in real life?
  • What features would make you more likely to use this product more?

Depending on the type of testing you’re doing, you might also want to conduct a more formal post-test UX survey. Using a standardized format with rating scale questions is a good way to gather quantitative data on the overall usability of your designs—particularly important if you’re at the summative testing stage.

Industry-standard scoring systems like the System Usability Scale (SUS) offer a straightforward approach, as you can use an already existing survey template. And since the SUS is used to measure the usability of many products, basing a post-test survey on it allows you to compare scores with other similar products—or even earlier versions of your own design.

Finally, after you’ve thanked the user for participating, give them an opportunity to air any final remarks they might have. You never know—they might say something that sparks an idea for the next big feature.

Best practices for successful usability tests

Now we’ve covered the questions to ask, let’s look at some best practice tips for usability testing questions.

The below dos and don’ts will ensure you communicate in the best possible way with participants, and gather the most valuable user insight.

  • Do: Ask participants to think out loud
    Understanding the in-the-moment thought processes of users, and their opinions, can help you understand their state of mind at each step. It also provides you with valuable insight to ask further questions.
  • Do: Ask open-ended questions
    Open-ended questions (a question which can’t be answered with ‘yes/no’) are best for getting in-depth feedback from users. A simple way to do this is by trying to ask questions with ‘what’, ‘why’, ‘how’, rather than ‘did’ or ‘do’.
  • Do: Utilize online tools or software
    Remote usability tests can be just as useful as in-person ones, especially if you have the right equipment to assist you. Make the most of the tools at your disposal to gather as much information as possible, for example—consider conducting follow-up questions over video call, rather than email; or use testing platforms that automatically process data for you and format them in an accessible way to maximize the results info available to you.
  • Don’t: Ask leading questions
    Similar to our tip on open-ended questions, leading questions are a no-no. Questions should focus on guiding the participant, and helping them open up, rather than telling them what you want to hear. Leading questions like ‘do you prefer A over B’ or ‘did you like A more’ can cause incorrect, and biased, results.
  • Don’t: Overwhelm with information
    This applies to both detail about the study, and what you want from participants, and information/guidance on how to do the test. Bear in mind that you want real insight from real users, which means they need to navigate the product organically and of their own accord. Likewise, if you provide too much context, it may overwhelm participants and change their instincts when completing the test.
  • Don’t: Take everything at face-value
    Bear in mind that even the most unbiased usability test will still be different to how users interact with your product in the real world, in their own environment, under natural circumstances. To counter any potential lab bias, take results and user feedback with a pinch of salt, and validate the results—and your conclusions—with additional testing methods.

Improve your user experience by asking better questions

Conducting usability tests can take considerable time and effort—and you only have a limited window to probe each user’s thoughts on your user experience design. Asking the right questions at the right time lets you squeeze as much juice out of each test as possible.

Get as much background info as you need, practice your wording to avoid leading questions, and think about the type of data you want to come away with at the end. Questions are one of the main techniques UX researchers have to interact with real users—so use them wisely.

Create your first usability test for free

Create usability tests, build sequences of tasks around them, and write questions for users to answer before, during, and after the test.

Frequently asked questions about usability testing questions

How do you write usability test questions?

Your usability testing questions should always be unbiased and open. Avoid adjectives whenever possible, and phrase your questions in a way that invites participants to share their thoughts openly. Opt for open-ended questions rather than yes/no questions and use simple and straightforward language—your usability testing results will be much clearer and more in-depth. Learn more about usability testing with Maze here.

What questions are asked in usability testing?

To get the most out of each usability test, you should ask demographic questions and background questions about people's product habits and preferences. This information allows you to make sure you find the right participants and gives usability testing results more context. Check out this usability testing template for some hints.