Running a successful website optimization program requires careful planning, execution, and a thorough analysis of results. As a practitioner, your schedule is already full with the day-to-day business of running your test campaigns. In addition to everything you do, it is important to understand how A/B and multivariate test results are derived. Math matters, and here is why.
Campaign results lead to critical business decisions. In order to make the best decision, it’s important to understand the complete picture. A recent industry post by a practitioner highlighted the need to understand test results. To help you meet this goal in your own organization, this post provides additional information about test methodology and details SiteSpect’s statistical methods.
We have all been taught to run tests to statistical significance to ensure that the effects being seen are not by chance. There are two common techniques to measure significance, a one-tailed test and a two-tailed test. Different vendors use different techniques.
A one-tailed test is defined as “a statistical test for which the critical region consists of all values of the test statistic greater than a given value or less than a given value but not both.” A two-tailed test is defined as “a statistical test for which the critical region consists of all values of the test statistic greater than a given value plus the values less than another given value.”
So what does this mean? Essentially a one-tailed test only considers the positive impact of changes. A two-tailed test, on the other hand, considers both, and therefore can measure not only positive impacts, but any negative impacts that may result. Using a two-tailed test in website optimization provides a more complete picture of the impact of a change.
SiteSpect uses two-tailed T-tests as the basis for our calculations. Since the beginning, SiteSpect has focused on delivering a powerful product true to the statistical nature of A/B and multivariate test practices. To retain this focus, our Chief Scientific Advisor, Dr. Michael Reed Sutherland, is responsible for the design and integrity of the statistical analysis components of the product. Our Professional Services teams work with Dr. Sutherland as needed on customer engagements.
Recognizing the importance of customer education, we will continue to deliver insightful content during our events and in upcoming blog posts. If you have any ideas on what you’d like to see covered, please comment on this post or email firstname.lastname@example.org.