Common Mistakes in A/B Testing: Not Segmenting Your Data

Data analysis is the crux of A/B testing and optimization. But a lot, if not most, of us who do A/B testing and optimization aren’t data analysts by trade. We’re marketers, developers, product managers, merchandisers, and more. Some of us are lucky enough to work with experienced data analysts, but a lot of us are learning as we go. This brings me to one of the most common mistakes we see in A/B testing: not segmenting data. If you’re like me, when you first started to A/B test, you would log in to your performance matrix, check all your KPIs, focus on your most important KPIs, and call it a day. But that doesn’t give you the whole story. To really understand how your A/B tests are performing and how they impact user behavior, you need to get good at segmenting your data.

What is Data Segmentation?

Data segmentation is the process of taking your data and breaking it into smaller groups based on set parameters. For an example, let’s say we’re tracking the following metrics:

1. Clicks on featured item image
2. Adds to cart of featured item
3. Total adds to cart
4. Purchases
5. Revenue per visitor

Segmenting this data would mean looking at specific groups of users, for example, only users who completed a purchase, or only users on mobile devices, and looking at how your metrics perform just among those users.

You can segment according to any parameter. You can look just at users who completed a certain action, such as completing a purchase or clicking on a specific link. Or, you can look at users from a specific region, or on a certain type of device. You can even segment based on combinations of factors, like users in California who completed a purchase, or users who both completed a purchase and clicked on the featured item image.

Why is Data Segmentation So Important?

Segmentation is critical because it gives you a much fuller picture of what factors influence user behavior. Let’s return to our earlier example, and say that you see that in your variation with a new featured item image, purchases are higher than in the control group. You could easily call this a win and end it at that.

But there are still a number of unanswered questions: Did the users who saw the featured image actually click it? And vice versa, did the users who clicked the featured image purchase an item? Is revenue per visit higher among users who clicked the featured image?

You could come up with any number of questions to help clarify the experience. But you cannot get the answers without looking at segmented groups of users.

Quick Start Guide to Segmentation

If you’re new to data analysis, break your goals into smaller pieces and tackle them one at a time. For example, if you want to know what the purchase rate looked like for users who clicked the featured image, create a segment based on that behavior. If you want to know how many users who completed a purchase actually saw the featured image, create a segment just for users who completed a purchase. This way, you’ll start to gain some clarity not just on which variation performed better, but also on why it did.

To learn more about SiteSpect, visit our website.