Adding a Quantity Selector on Product List Pages

One of the hardest lessons to learn in A/B testing and site optimization is that sometimes really good ideas don’t actually improve conversions. While quick wins may be more gratifying, these learnings are actually the substance of (and number one reason for) A/B testing changes on your site. In this case, a large UK-based grocery and health supply retailer noticed in their existing data that many mobile users add multiples of the same item to their carts directly from product list page (PLP) and search results pages (SRP). In response, the optimization team introduced a new quantity selector to the “add to cart” call to action. This new feature allowed users to add multiples of the same item to their carts with less clicks and without having to leave the PLP or SRP.

Evaluating a Variety of Metrics

This A/B test focused on users who buy multiples of the same product, but it’s always important to look at both the smaller scale impact and the broader picture. In this case, the brand measured the number of mobile users who add multiples of an item to their carts from a PLP or SRP, and also looked at overall average order value (AOV) and conversions.

In the variation with the new quantity selector, the number of users who added multiples of an item to their carts increased by over 200%. However, AOV decreased by more 18%, and conversions decreased by more than 15%.

How to Interpret Results

The results on this A/B test are particularly interesting, since when viewed myopically users did in fact use the new quantity selector. But, the team still decided to revert to the original because of the impact on overall performance for AOV and conversions. In order to arrive at this decision, the team had to separate and prioritize their goals.

The first goal of the A/B test was to streamline the process for users to add multiples of the same item to their carts on mobile devices. However, this goal is part of a wider goal — to streamline the checkout process and increase orders overall. While it does seem that the quantity selector has some appeal, it also seemed to add more noise to the process, distracting users and possibly leading some away from their original goal. To round out the picture, since conversions and AOV were actually higher in the original version it suggests that editing item quantity within the cart wasn’t actually a deterrent or hindrance to purchase. For the subset of users purchasing multiples of the same item simply clicking the add to cart button multiple times on the PLP & SRP pages was sufficient for their needs. The brand took this learning and reverted to the original version of the experience, and now has a better understanding of how users shop on their mobile site.