Helen Tsvirinkal on how user testing can improve every stage of your design process

Senior Product Designer Dan Ritzenthaler once said:

And he’s right to feel uncomfortable—user input is invaluable to the design process. It comes from the exact group of people your design is for. That’s why designers tend to conduct at least one round of usability testing before designs are sent off for development.

But is this enough?

Our data at Maze shows that design teams are likely to carry out usability testing only once or twice during the design process, usually towards the end. This is also known as summative testing. Testing a high-fidelity prototype is closer to the real thing, so it’s understandable that designers might opt to test late if there’s limited time and investment for testing.

While late-stage usability testing is incredibly important, carrying out early tests with less complete designs—aka formative testing—is also a fundamental part of the design process. Some designers might feel uncomfortable putting their work out in the wild when it’s clearly not finished, but it’s important to remember this design mantra:

And while an intuitive early design idea could be well-understood and praised by other designers in your team, an unbiased audience of users may see it differently. If there’s one group whose feedback is most important, it’s the people who use your designs every day.

By doing formative testing early and often in the design process, you can pinpoint real problems before they become baked into your design, prioritize design issues that are most important, and iterate on your design more rapidly and accurately.

To show how different kinds of user testing can guide your work as a designer every step of the way, we spoke to Helen Tsvirinkal—a product designer at Shopify—for an insider’s perspective on the design process.

But before we meet Helen and dive deeper into the value of user testing, let’s clarify what we mean by ‘formative’ and ‘summative’ testing.

Formative user testing aims to find out which aspects of a design work, and which don’t. Since it tests specific design flows or elements within a product, you can (and should) do formative testing at many stages of the design process. It doesn’t require designs to be in a state of near-completion—even a paper mock-up can be enough to get valuable insights.

There are many types of formative testing, but the most common is usability testing. A usability test asks participants to complete a set of tasks related to a specific part of a design, then rate how easy or difficult it was. Successful usability tests have less than 10 tasks, focusing on one design flow. Even testing just a few people will give you useful results—but as with all user tests, the more the merrier.

By conducting formative user tests frequently throughout the design process with low- and mid-fidelity prototypes, designers can quickly rectify problems as soon as they’re identified. The more you test, the more accurately you can iterate—and the better your design solutions become.

Looking for a usability testing tool to use? Browse our list of the top usability testing tools for better UX.

The aim of summative testing is to find out how usable your designs are overall, compared to a benchmark or your competitors. Because it tells you about the bigger picture, summative testing normally happens in the final stages of the design process or after a design goes live. Designs need to be almost complete to get an accurate idea of their usability overall, as opposed to formative testing where an early mockup is enough to get actionable insights.

When a product is about to be released and the design team needs statistical validation that the new design performs better than the old one, that’s where summative testing comes in.

Getting statistically relevant results means testing a large sample size, and asking people to grade their experience according to a pre-defined scale, e.g. the USE (Usefulness, Satisfaction, and Ease of use questionnaire). One easy way to get a big sample size to test your designs is to import your prototype into Maze and share a link to your user test to unlimited users.

If you want to see how your latest design stacks up against older versions of the product, summative testing can be very useful.

The best designers combine both formative and summative user testing to back their designs with data at every step of the process. For an inside take on how to approach design in this way—and the benefits of doing so—we spoke to Helen Tsvirinkal, Product Designer at Shopify, about how and when user testing should inform design.

Before you even start designing anything, Helen recommends devoting a decent amount of time to properly understanding the problem your design will solve.

Since there’s no design prototype to test yet, this mainly involves speaking to users to learn about their current needs and frustrations, getting advice from people in other teams who have worked on similar design projects, and reviewing any previous relevant user research or test results.

Helen stresses the importance of this stage, as it provides a strong foundation for your work further down the line—especially if tight deadlines end up preventing you from doing as much testing as you would like:

If you work in an agency and don’t have other teams who can advise you, ask the client for any customer interviews or user data they have. You can even do this before a project starts at the proposal stage. Including data from user testing in your design proposal won’t just impress your potential client—it’ll make it easier to get buy-in to do further user testing later in the project.

Once work on a design begins, you can conduct the next round of formative testing with a low-fidelity prototype or paper mock-up. At Shopify, the designers use a UI kit to drag and drop high-fidelity components when exploring designs, but a paper mock-up with boxes, arrows, and cards works just as well.

Helen says this stage is particularly important for bigger design projects with a lot of unknowns, like redesigning an app from the ground up:

Card sorting is one simple way to get an idea of how your users think. It involves asking users to categorize content items into labeled groups however they see fit. If you want, you can follow this up with tree testing to create a more realistic menu scenario.

While tests at this stage are less about specific aspects of usability—and more about whether your design is heading in the right general direction—Helen’s team still sets clear tasks to guide users through the test, so they focus on one problem at a time.

How much early formative testing should you do depends on the project:

For big projects like website re-designs, new features, or entire app overhauls, doing early formative tests with a quick prototype is vital to making sure your design ideas really address user needs. This’ll save you having to make major changes later in the process.

The next stage is crucial, as it’s where you’ll really put the usability of your designs to the test. If you’re working on a smaller design project or fixing a known UX problem, this might be the first time you do a proper user test. And while a mid-fidelity prototype may not be polished, it should at least resemble the design that will end up in the actual product—so your test results here will heavily influence how your final design looks and feels.

Mid-fidelity prototype tests ask participants to complete a routine set of tasks with a design that represents a realistic (but limited) part of a product. It could be a set of UX flows, or a single screen with menus. Users are asked to evaluate specific aspects of their experience, e.g., “Did you find this task easy or difficult to complete?”, “How could this process be improved?”

Helen finds speaking to users particularly interesting at this stage, as the closer resemblance of a mid-fidelity prototype to the real product can cause unexpected results:

And it’s not just UX flows that are useful to test at this stage. Up until this point, the terminology and UX copy you’ve been using in your design has probably been based on your assumptions—or maybe you just haven’t thought about it much yet. Mid-fidelity testing is the perfect time to find out if you and your users are speaking the same language.

Mid-fidelity user tests are your bread and butter when it comes to involving users in your design process. Running multiple rounds of short, focused tests—and using the results to inform each design iteration—is fundamental for creating truly user-centered designs. Helen recommends closely analyzing your results to identify the biggest, most common issues, and prioritizing your work based on user needs and resources.

If you’ve done a decent amount of formative testing and have built your designs based on the results you got, now’s the time to think about moving on to summative testing.

Remember: summative testing helps you measure or validate the usability of a product, and gives you usability metrics to compare against benchmarks. By the late stage of a project, you should already be pretty confident in the functionality of your design:

With your design almost complete, summative testing can reveal how usable the end result is compared to old designs or your competitors. Instead of asking participants questions, summative tests usually measure quantifiable results like how many tasks someone completed, how long it took them, how many clicks they needed, etc.

Since you need to collect a lot of these results to see a reliable pattern, you’ll need to test with at least twenty users. One easy way to collect unlimited quantitative results at this level is to use Maze to test your high-fidelity prototype before hand-off.

Our final piece of advice? Don’t miss a launch deadline by being too hard on yourself when the results come in:

Creating effective designs and seeing them used by thousands of people is an amazing feeling. But as every designer knows, there’s a lot of pressure that comes with it. Between tight deadlines, demanding clients, and constant feedback from your manager and colleagues, it’s easy to let your focus slip away from the people you’re designing for.

By building frequent user testing into your process from an early stage, you’re making sure that you discover and fix UX issues with each new design iteration, and that your final design solution doesn’t include major errors that place you back to square one, delay your launch, or even worse—are released live.

So before you even start designing, speak to your users to understand their problems. Carry out formative testing with low-fidelity prototypes to see if your earliest ideas work in practice. Create a feedback loop to support your design efforts with multiple short tests throughout the process. Then use summative testing at the final stage of a project to find out how well you did overall.

Because while time pressure and other obstacles can make it tempting to leave user testing until the end, remember: