A/B Testing Mistakes and How to Avoid Them Part 3 – Doing Too Much at One Time

By Kevin Plankey

May 18, 2023

Share

When you first start managing an A/B testing and optimization program, the possibilities can be both exciting and overwhelming. Chances are you’ve had a running list of improvements you want to make to your site and are eager to finally implement and test those. However, despite the temptation to jump into everything all at once, it’s important to take your optimization strategy step-by-step, as every change to your site can affect other parts of the shopping experience.

Miss my last post in this series? Read “A/B Testing Mistakes and How to Avoid Them Part 2 – Setting the Wrong Metrics“.

Pause, Prioritize, and Reassess

Problems typically occur when you simultaneously A/B test multiple parts of a conversion flow. Testing multiple aspects at once can obscure the root causes of the changes you’re seeing. For example, let’s say you have the following A/B tests all running at the same time:

  1. A homepage A/B test that aims to increase clicks on one product category page.
  2. An A/B test on your product category pages that highlights featured items and aims to increase adds-to-cart of those items.
  3. An A/B test in the cart for last-minute add-ons that aims to increase average order value.

At first glance, these are all great A/B tests, but now there are additional factors you need to consider. You know what happened in each of these A/B tests, but you’ll have new questions about “why”:

  • Are the increased adds-to-cart the result of the change on the product category pages, or because you’re seeing an influx of traffic to one category page highlighted in your homepage A/B test?
  • Has your average order value increased because of the new add-on promotion, or is it because more shoppers are seeing your product category pages?

The combination of factors can cloud your understanding of why the shopping journey is yielding different results. If you encounter this type of confusion, you can always pause your A/B tests and reprioritize them.

One way to do this is to map out your typical shopping journey. Are most shoppers entering the homepage? If that’s the case, then maybe focus just on that A/B test until you get more confidence in your results. Or do most of your shoppers enter directly on a product category page? If that’s the case, you’ll want to focus there. Either way, map out your shopping journey and restart your A/B testing sequence from the beginning. This will give you greater clarity about what causes the behavior changes you’re seeing on your site.

Of course, overlapping traffic is only a problem if shoppers are assigned to multiple campaigns. If your traffic levels allow it, another option is to run all three A/B tests at once, but split traffic between them. Each shopper will only be assigned to one campaign, rather than possibly seeing all three at once. Each campaign will likely take longer to reach a mature conclusion since the traffic will be lower for each, but you’ll be able to assess all three changes in isolation. The results of each A/B test can help you determine what areas to prioritize next.

Segmentation is Essential in Good Test Design

Segmentation is a critical part of any analysis for your A/B test. But especially when you find yourself with overlapping A/B tests that muddy your understanding of the shopping journey, it’s time to segment out your results to get some clarity. Put simply, if you’re newer to data analysis, segmentation is when you separate groups of shoppers based on common factors, for example, by device type, by action taken on the site, or by entrance page. This is simple to do and in SiteSpect you can easily create segments based on any number of factors.

For the above situation, where you have your three overlapping A/B tests, you can gain some clarity on your results by segmenting based on other actions the shopper has taken on the site. For example, for your A/B test on the product category page, try excluding shoppers who have clicked on the homepage feature you’re also A/B testing. This way, you can see the impact of the product category page A/B test without the additional weight of the homepage A/B test. Or, in the cart, try looking only at shoppers who clicked on a last-minute add-on item. This way, you can see how this feature directly affected the average order value. In general, it’s best practice to always segment your data to get the clearest picture possible.

Summary

Even though there may be a strong urge to tackle everything simultaneously, it is crucial to approach your optimization strategy gradually, considering that each modification to your website can have an impact on various aspects of the overall shopping experience.

Ready to redefine your optimization strategy? Speak with an experimentation expert to see a personalized product demo. It will be time well spent!

Share

Kevin Plankey

Kevin Plankey

Kevin Plankey is the Director of Demand Generation for SiteSpect and is responsible for marketing operations to include strategy and implementation of all demand generation efforts: social, email, website, organic/paid media, and event planning & promotion.

Suggested Posts

Subscribe to our blog: