A/B testing designs to expand Braze’s product offering for SMS

A/B testing lets you compare variations of a design to find out which one performs better.

In this case study, we’ll show you how the design team at Braze used Maze to validate new designs of their SMS product offering through A/B testing. Apart from the intended insights, the tests also helped identify other frictions to improve, creating an even better product experience.

At Braze, we connect companies to their customers through our lifecycle engagement platform. To create even better interactive conversations, we wanted to add Multimedia Messaging Service (MMS) to our SMS capabilities. This would allow users to add media files—like images, videos and GIFs—to their SMS messages. But adding this new functionality came with new UI elements that we had to get right.

To ensure that everything worked as it should, we needed to test the new designs against our existing flow. This let us see how users navigated the product, where they got stuck, and what steps they took to get their jobs done. And it gave us the confidence that we were providing something that our customers both wanted and could easily use.

At Braze, we do a significant amount of research to understand and prioritize product development. We spend a lot of time speaking with customers and understanding their pain points, and have several formats for capturing input on product enhancements and the value they would provide, and we use ProductBoard to compile and organize most of the customer feedback we receive. The data compiled from research revealed that customers would like the ability to add multimedia content like images, GIFs, and videos to their SMS campaigns.

The question then was – how might we best enable marketers to easily add images and GIFs to their SMS campaigns?

I sat down with the engineering and product leads to talk about minimal functionalities and technical considerations. Together, we aligned on the high-level goals and defined what a successful outcome would look like. For this project success was clear: feature adoption.

Next, we developed an initial design that we could test with customers, keeping our key adoption metric in mind.

We decided to run an A/B test with two different mazes to:

We created two prototypes with the same high-level design components, but different layouts and ordering. For example, Prototype B put more emphasis on the zone for media attachments to bring more attention to this option, as compared to Prototype A . As you can see below, the team tested different visual treatments for some UI elements.

We set up each Maze test as follows:

To collect responses, we posted the test in our community called Braze Bonfire. As of January 31, 2022, over 5,000 community members use Braze Bonfire to chat, ask questions, and provide feedback on key platform features. For customers willing to participate, we created a private group for the test. We garnered nearly 20 responses per Maze test, most of which came in the first couple of days. Then we started observing behavior and assessing results.

First, we compared the prototype satisfaction rating, where Prototype A had a clear advantage.

From there, we took a deeper dive into the results of each maze. We were particularly interested in the bounced user paths, which helped us pinpoint where people were interacting before leaving the test. Were people misclicking? Were they spending more time than expected on a given screen?

There were plenty of unexpected but interesting results.

One was a dropdown selection where users had to choose MMS or SMS phone numbers. We saw people spending way too much time on this selection, along with a fair number of misclicks.

Even though the heatmap showed an overall higher success rate on this version, we knew we could do better. So we tweaked the design to make the selections even clearer.

Finally, we scheduled a 30-minute follow-up call to chat with participants about their experience. We also walked them through our updated designs for additional feedback.

This feedback validated our designs and gave us the confidence to roll out our new MMS functionality. And it’s been great to see the adoption of this highly requested feature, allowing our customers to add enhanced multimedia to their SMS messages in a clear and convenient way.