By Kate Orchard
April 25, 2018
As an optimization consultant, I help users get the most of their A/B testing solutions every day. Often, the key to turning a good idea into an effective A/B test lies in the KPI. This week, I’m sharing some common KPI mistakes, as well as some tips for how to choose the most meaningful measures you can.
Imagine the scene: You’re set up with a brand-new A/B testing solution, excited to see what data you can gather and where you can take your brand from here. You even know where you’re going to start and have a first campaign designed and ready to A/B test. Now comes what is arguably the most important question: What are you going to measure? As an optimization consultant, I help users get the most of their A/B testing solutions every day. Often, the key to turning a good idea into an effective A/B test lies in the KPI. This week, I’m sharing some common KPI mistakes, as well as some tips for how to choose the most meaningful measures you can.
Mistake: Your KPI is too general
If you find your key metric is something like “conversions,” you’ll end up missing out on crucial data that will inform your growth. For example, say you’re A/B testing the phrasing of the text on your “Add to Cart” button. One version says “Add to Cart,” the other says “Buy Now.” Of course, you want to know which button leads to more end conversions, but you also probably want to know how many items make it into each cart, what the average order value of that cart is, and perhaps how many times the user clicks on that button.
Mistake: Your KPI is too specific
Now, let’s say you’re A/B testing variations of the navigation bar on your homepage. You may be focused on users getting from the navigation to the products page. But, if that’s all you measure you lose sight of other ways that users interact with that feature. Often, one change will have unintended results in seemingly unrelated areas, so be sure to include enough metrics to tell the whole story. For instance in this example, you may also want to measure what other pages users land on, how long they spend on the navigation bar, what the bounce rate is for pages they do land on, and how long they spend on their secondary page.
Solution: Think Backwards to Determine Important Metrics
The best strategy and most universal rule of thumb for determining good KPIs is to think backwards and look at the whole journey a customer would take with your feature. When you make a change on your digital channels, what is the specific goal of that change? Do you want more users to add items to their carts? Do you want users to spend longer looking at certain pages on your site? It’s crucial to zoom out and measure performance by actions beyond conversion or purchase. But, you also need to make sure that you measure equivalent actions between the control and variation. If you can only trigger your KPI in one experience and not the other, you won’t be able to compare the two variations. Keep stepping back until you find a KPI that directly measures your goal.
Brainstorm: Creative KPIs
Some A/B tests do call for KPIs such as purchase or form completion, but some interesting KPIs I’ve seen have measured out of the box metrics for game changing data. One brand ran an A/B test where the KPI was whether users clicked on a banner promotion. The banner was originally un-clickable, but the results of the A/B test showed that customers really wanted to click that banner. Another A/B test tracked what customers did after performing the measured action. For example, after clicking “Add to Cart,” were customers staying in their carts or going back to a product page? More common KPIs include how long visitors spend on a given page, scrolling depth, and page views per visit. Then there is always the KPI of purchase and average order value.
Go Forth and Measure
This overview should give you a starting place to conceptualizing the important metrics for your specific A/B tests. Still, work with your Optimization Consultant to determine the best metrics for you, so that you can get the most out of your A/B testing solution.
To learn more about SiteSpect, visit our website.
Subscribe to our blog: