Concept Testing: Moving Forward with the Right Ideas
Concept testing allows you to validate product, design, and marketing ideas early on and save time and resources by moving forward with the right concepts. Over the course of this guide, we’ll look at the benefits of concept testing, how to run a concept test, how to craft a survey, and real-life examples of concept tests. With advice from expert product managers, you’ll be ready to test, learn, and iterate with your customers—fast.
What is concept testing?
There are so many product ideas out there that sometimes you can be overwhelmed with choice. Your managers may feel strongly about your next project, you could have some ideas you’re excited about, and your team members may have ideas of their own, too. But how do you choose between all of these concepts and decide which to invest your company’s time and money in? That’s what concept testing is for.
The main difference between a good idea and a bad one is that a successful product concept appeals to the people it’s made for. The target audience is the focus from the start. After all, what is a product without its customers? As Silicon Valley entrepreneur Steve Blank says, "'Build it and they will come,' is not a strategy. It’s a prayer.”
To put the customer front and center in your ideation process, product leaders run concept testing to validate ideas with users before investing time into them.
When to use concept testing
Concept testing is the final stage of product discovery, a process whereby you introduce a product concept to your target audience to determine if it’s worth following through with the idea. What you’re testing could be as innovative as a new product or an entire ad campaign, a revamped web design, or new branding.
The same problem could be solved in many different ways and from many different angles, but not all would actually work or deliver significant value to your customer.
“The way I think of concept testing is the fine line between the phases of scoping the problem and the solution definitions as a part of the discovery,” says Netali Jakubovitz, Senior Product Manager at Maze. “Meaning, when doing discovery (before going into actual building and delivery) you've already selected an area to focus on, you learned a lot about it and set up boundaries to scope it.”
She continues, “Moving forward requires to figure out, within those boundaries, what kind of solution would work best for solving the problem in a way that provides actual value to your customers. The same problem could be solved in many different ways and from many different angles, but not all would actually work or deliver significant value to your customer.”
Concept testing can be applied at multiple stages of the product development process, such as during ideation, when prototyping, when defining marketing messaging, or just before the launch of the product. We’ll look into when to run a concept test in more detail below.
The benefits of concept testing
Concept testing proves valuable because it validates that your customers actually want the product. Idea validation is an essential part of UX research and a necessary step in creating successful products.
We could go on for days about the advantages of concept testing, but here are just a few of the reasons why this technique is an essential part of product success:
Save time and money
Approximately 95% of new products fail, according to research from Harvard Business School. A whopping amount of resources go down the drain when a product flops, from the money spent to create it, to the effort exerted by the product development team.
It's important to validate the concept we're going after before starting to build it due to the costly charge of engineering time, the complexity entailed in changing direction if development already started, or deploying a solution that solves a symptom but not the problem. This would send the team back to the concept drawing board and would be a waste of precious time, resources, and motivation.
While concept testing itself requires a slice of your budget, in the long term that investment is far less costly than launching an idea that fails. For example, once a software product is launched, correcting an error can cost 100 times as much as it would have if it were fixed in the development phase. Studies also reveal that development teams waste 40 to 50% of their time on avoidable reworks, a reality that can be avoided with concept testing.
It provides actual value for early validation, contributes to prioritization, and eventually increases the overall quality of our product at a low cost. There’s simply no reason not to do it!
Shelly Shmurack, Product Manager at Walmart Global Tech
Validate and launch with confidence
The data [from concept testing] allows product managers and decision-makers to gain confidence in favor or against an idea or a solution.
Andrea Ruggeri, Product Manager at Doodle
Concept testing also provides data to back up your idea and get buy-in from stakeholders or other members of your team. Bringing a new concept to life can be daunting, and management may understandably be cautious about taking that risk. However, once you have the research from concept testing on your side, even your most cautious team members will see the potential in your product concepts.
Andrea Ruggeri, Product Manager at Doodle, says that he finds concept testing useful for decreasing uncertainty. “The data allows product managers and decision-makers to gain confidence in favor or against an idea or a solution,” Andrea explains.
Discover new business opportunities
Concept testing can expose a new path for your company to take, whether that means optimizing the existing features of your product or pivoting entirely.
Concept testing helps explore various potential solutions to the problem and uncover the sweet spot of customer delight; so then you can know where to invest costly resources and energy. This translates to business revenue and growth.
Your assumptions about what makes your concept great are just that—assumptions. So testing shows what aspects of the idea actually appeal to your customers, which you can then capitalize on when developing or highlighting the concept's best attributes in your marketing campaigns.
Even a concept test that appears to fail still has value. "Early failures are not only desirable but also necessary since experimenters can quickly eliminate unfavorable options and refocus their efforts on more promising alternatives (often building on ideas that were initially unsuccessful)," writes Stefan H. Thomke, Author of “Experimentation Works”.
Martín Burgener, Product Director at eDreams, explains how failure is a key element of building products at eDreams: “If we think about the products eDreams have been working on, roughly 70-85% of all tests we do result in a failure. And that’s for a company that is product-led and has been making digital products for 12 years.”
Once you evaluate what matters to your target audience through concept testing, you can use those learning opportunities to tweak or completely change your product to maximize its success.
When to run a concept test
Concept testing is an iterative part of the product development process and can be done at several stages to test and validate key decisions. You can and should use concept testing multiple times in the process, from ideation up until launch.
Here are some specific points in the product development process when concept tests can be run:
At the discovery and ideation stage
Concept testing is handy during the product discovery stage because it narrows down the solution you should move forward with for a particular user problem.
Concept testing is an important part of the ideation phase when we’re testing alternative solutions to a problem, and can provide incredible insights into what customers actually want.
Grainne Conefrey, Product Manager at NewsWhip
Testing at this stage helps you check your team’s assumptions and validate product ideas with customers before moving into the development stage.
During the design stage
Once you’ve started designing your product, there are numerous opportunities for concept testing to be of use. You can test the particulars of design concepts, from low-fidelity (e.g. the layout of a web page) to high-fidelity prototypes (e.g. user flow, graphics, branding).
Early [design] concept testing allows for multiple iterations quickly and easily, making the final developed concept more precise and better optimized from the get-go.
Shelly Shmurack, Product Manager at Walmart Global Tech
Before the product launch
It’s never too late to conduct a concept test. Running a concept test before a product launch allows you to catch any last-minute changes that need to be made.
A pre-launch concept test can also help you figure out which marketing ideas, from messaging to graphics, to move forward with in your launch campaign.
Tips for effective concept testing
Here are a few tips on how to test and validate your idea with your target audience.
For an in-depth guide to conducting a concept test, check out chapter 2 of this guide.
Set a goal for your test
Just trying to gauge if customers “like” your product concept is a bit too vague. Come up with a precise aspect to measure like, Does my target market find my product useful enough to purchase it?
And if you want to take this a step further and be more specific, set up targets for your tests. For instance, if you’re concept testing a new feature idea, your goal could sound like this:
“We want to validate that the majority of our users (80% or more) find this feature useful.”
Being precise with test goals will allow you to quickly assess if a concept is worth moving forward with, or if you need to tweak it.
Ask your audience the right questions
You need to carefully consider the language used in the test and how questions are phrased so that customers can understand them and you'll receive rich feedback. The right research questions will reveal the insights necessary to validate your concept or evaluate what it’s lacking.
Poorly conceived questions like, Did you prefer Product A over Product B? will leave you still asking, Where did we go wrong? While the previous question may seem benign, it introduces an inherent bias towards Product A. Asking simply, Which product did you prefer? will get you an unbiased answer.
Tap into your target audience
There’s no point in conducting idea validation with participants who don't fit the demographics of your customer base. That’s why recruiting the right participants is key. With their feedback as guidance, you can develop and launch a successful product that meets customers’ needs.
Well-known examples of concept testing
To conclude this first chapter, let’s look at two famous examples of concept testing real-life products. In chapter four, we share three more concept testing examples focused on digital products Todoist, NewsWhip, and Better Proposals.
Back in 2015, Tesla conducted a concept test on a whole new scale. The company announced the upcoming Tesla Model 3, which still hadn’t been made at the time, and invited interested consumers to put down $1,000 deposits. Incredibly, about 400,000 customers invested in the Model 3 in the month after its announcement, as reported in HuffPost. Not only did Tesla determine whether their target customers would purchase the new car, but they also had a massive $400 million investment in the concept.
Concept testing is also vital for small-scale changes that nonetheless can be make-or-break moments for a product. Instrument manufacturing company Yamaha was trying to decide between a knob or a sliding fader for a new electric keyboard they were making. They enlisted their customers’ help in choosing the optimal feature to ensure that the keyboard met their target audience’s preferences.
The beauty of concept testing is that it's beneficial for everyone involved. Your customers get a product that's actually relevant to them, designers are reassured they're on the right track, and product teams save time and money moving forward with the right ideas.
In the following chapters, we’ll walk you through how to run a product concept test, the creation of a concept testing survey, and some real-life examples of concept tests. With these tools in your belt, you can move forward with confidence and help ensure your users find value in your products.
Here’s to making life easier for product owners and users 🎇