Setting Intelligent Alerts for A/B Testing

Imagine you log into your A/B testing platform to check the results of a major A/B test that you’ve spent weeks planning, building, and releasing. You’re excited to see some results. Did you get the revenue impact you expected? How are customers responding to it? You’ve let it run for two weeks to gather enough data to reach significance, so you’re looking at results for the first time. But when you log in, you see that your KPI hasn’t been collecting any data, one of your variations has been hurting another metric at significance for 10 days, and a third variation has already outperformed the others across most of your metrics. In a moment, you’ve lost weeks of work, possibly lost significant revenue, and missed an opportunity to optimize your traffic.

When you’re running an A/B test or an experiment for several weeks, you may not be checking your campaign every day. It needs some time to see enough traffic before you can get actionable insights. But what happens if your campaign isn’t behaving the way you expect? You log into your A/B testing tool or analytics platform and rather than the data to drive decisions you’re left with either inaccurate reporting or just not enough information. Now you’ve lost time and either need to start over or you have to make a less certain decision.

The Right Alerts at the Right Time

Setting up alerts isn’t all about notifying you of the worst. It can also help you identify winning variations early so that you can optimize and direct your traffic towards experiences that have the best outcomes. Some of the most important alerts to consider are:

• Metrics without traffic: If your metrics aren’t firing appropriately (or at all), or aren’t seeing the traffic you expect, you can lose a lot of time and resources. You may expect to look at your data and be able to make clear decisions, but instead find that your data just isn’t there or it’s skewed. Knowing when those metrics aren’t seeing the levels you expect allows you to course-correct without wasting time.
• Campaigns without traffic: There are many reasons your campaign may not be getting the traffic you expect. There could be an error in the build, the audience definition, or some external issue that needs to be addressed. Find out right away when your campaigns aren’t getting traffic so you can fix the issue without losing more time.
• Hurting KPIs: If your campaign is measuring an important metric, such as form submissions, you’ll want to know right away if a variation is hurting that KPI. Letting a losing variation run for too long can cost you major revenue, and you don’t always know when you’re going to cross that threshold of significance. Alerts for hurting KPIs save you the time and money of letting a bad experience remain live on your site for too long.
• Winning variations: The inverse of hurting KPI alerts, if you don’t know what variations are already winning you can’t make the choice to direct more traffic to those variations, or even end the campaign and roll out a winning experience.

Intelligent Alerts to Optimize your Strategy

As helpful as alerts are, there are times when you may want to turn them off – your campaign is only released to a small subset of traffic, you strategically turned off a metric, or you need to hit a time frame rather than a metric threshold. This is why knowing your alert options and choosing them strategically based on the goals of your campaign can take your optimization efforts to the next level.

You can learn more about SiteSpect Alerts in our documentation, and learn more about SiteSpect at our website.