Common Mistakes in A/B Testing: Testing Too Small

Often website optimization teams, especially new ones, A/B test aspects of sites that have been frequent subjects of debate, like button colors, calls to action, and hero images. Finally, we get to settle those debates once and for all: should the Buy Now button be green or blue? What’s the best phrase to convey our low-price strategy? Does matching the color scheme of the hero image to the logo increase sales? These tests can be important, not only to conversion rates and average order values, but to design team relationships as well.

These A/B tests are simple to set up, easy to explain, and winners are usually trivial to deploy. But in addition to relatively safe tweaks like these, we encourage you to also think big…even though thinking big can be difficult. 

You may have been working on your site for months or years, know it backwards and forwards, and have already improved conversion many times. Hundreds or thousands of customers make purchases on your site every day. It may bring in a substantial portion of your company’s revenue. It works! It’s good.

But never forget: the biggest enemy of a great website is a good website, and the largest impediment to improvement is risk of failure. Here’s how you can move past quick wins and start A/B testing big ideas. 

Start With Your Visitors

One of the best resources for A/B test ideas are your visitors. Who succeeds, who doesn’t, and why? Start with these questions to gain a better understanding of your visitors and what they might be missing on your site. 

• How many of your visitors scroll down three-quarters of your PDP, and what’s the correlation between PDP consumption and purchase? 
• How many PDPs does the average customer visit?
• Is there compelling PDP content below the fold? Should it remain there?
• How does the conversion rate between first time visitors and returning customers compare?
• Do your visitors search or browse more? Which path converts better?
• Why do X% of visitors begin checkout but never finish?

If you don’t know the answers to these questions you should find them through segment analytics and use them to ideate future A/B tests. 

Talk To Your Colleagues

Another excellent resource for A/B test ideas are your co-workers. All of them. Most of us shop online – for clothes, gadgets, cars, apartments, houses, vacations, and perhaps for whatever you’re selling too. We’ve all had both great and horrible experiences online, and when asked for honest opinions, we’re all more expert than we might think.

Make sure stakeholders and their teams know your A/B testing capabilities. Make sure your site designers, graphics artists, merchandisers, buyers, and marketers all know your A/B testing capabilities. Beg them to tell you how your site could be better. Make it fun. Make it a contest!

As you evaluate A/B test ideas, think about them in terms of your research to questions like those above. What makes a website visitor a customer? What are the behaviors of a successful visitor? Which aspects of the site help them succeed the most? How do you get more users to do what will make them successful?

Remember: A/B testing mitigates risk and allows the best ideas from the entire team to prevail. And though the change may be bold, the A/B test setup may be small (ask us!), and the risk to your site’s success will be small and manageable, because you can only A/B test a small number of visitors and can turn the A/B test off at any time! 

Take A Look At The Competition

A third resource: name several aspects of your competitor’s site that you think might be better than yours. Then think of how you can optimize your site to go above and beyond your competitors. 

The real risk to your business is not exploring the bold possibilities – and never finding next-level improvement. If your competitor is A/B testing big ideas and you’re not, they’re just a click away from stealing your next customer. 

To learn more about SiteSpect, visit our website.