Introduction to multivariate testing

Overview

PDF version

Common methods for running controlled experiments on websites range from simple A/B testing to sophisticated multivariate testing, also known as multivariable testing. In A/B testing, one or more new versions of a page or single site element compete against the original (control) version. For example, two new versions of a headline might compete against the original headline.

Multivariate testing, on the other hand, is like running many A/B tests concurrently, where there are multiple elements being tested at the same time. For example, two alternate product images, plus two alternate headlines, plus two alternate product copy text, for a total of 27 possible combinations (including the original control versions). What's important to understand about multivariate testing is that it not only shows you which combination of elements generate more sales or pull more leads, but also reveals which individual elements influence visitor behavior versus those that do not.

For example, did variations in the product image influence visitor behavior more, less, or the same as the copy? Understanding how each site element causes visitors to interact with your site is the essence of a "test-learn-repeat"process that marketers can use to synthesize new ideas and continually improve their site's ability to achieve — and exceed — their marketing goals.

Testing as a platform for continuous improvement

The process of testing reveals not only what works and should be implemented, but also what doesn't work and should be avoided. Every website idea, whether content, functionality, or campaign-related, should be put to the test to determine if it helps or hurts the visitor experience. While some new ideas lift conversions, others fail — sometimes significantly. But even with these failures, there is definable knowledge gained over what to avoid the next time. The ability to test a new idea and 'look before you leap' is an unmistakable advantage that breaks the constraints on marketing innovation. Only once a solid testing capability is in place, and the impact of any site change is able to be quantified, can marketers truly optimize their site's effectiveness.

How can multivariate testing optimize web marketing?

Multivariate testing can yield some spectacular results in enhancing online effectiveness. For example, SiteSpect worked with a well-known online auction site to perform a series of multivariate test campaigns to understand which elements were most influential in bidder conversions.

The team tested variations of elements such as page layout, messaging, calls-to-action, promotions, image sizes, and navigation. Using the SiteSpect optimization platform, all of this was possible without the need to make a single change to the underlying website.

The knowledge gleaned through these tests resulted in:

  • 429% increase in bidding activity
  • 83% increase in catalog browsing activity
  • 166% increase in individual item views
  • 590% increase in opt-in registrations

What to measure: defining test objectives

Before you start formulating a test hypothesis, or begin running tests, the first and most important step is to ensure that there are defined objectives for the website. You'll want to examine your marketing goals in order to determine the appropriate success factors that all of your organization's stakeholders can agree upon. Here are some typical measurable website goals:

  1. Make money: sell product, generate leads, and advertising or promotional click-throughs.
  2. Save money: enable users to adopt self-service features and/or answer product and service questions on their own (such as through online FAQs and documentation).
  3. Create brand awareness and industry visibility.

It's important for all stakeholders to agree on the goal(s) of the website, because when a decision is made to adjust or optimize something on the site, everyone's needs should be addressed. Make sure you are testing the things that truly matter for your organization and balancing performance across all stakeholders.

Once goals are determined, the next step is sifting through potential key performance indicators (KPIs) to decide upon those that will accurately measure progress towards specific marketing goals and benchmarks. For example:

  • If the goal is to make money, track those pages and areas of the site that users click on in the conversion funnel, such as the "Buy Now" button and resulting "Thank You" pages.
  • If the goal is to save money, track the interactions with both the self-service areas (like FAQs and help content) as well as non-self-service areas (such as contact and help ticket generation) of the site.
  • Brand awareness goals can be more difficult to track, but certain KPIs can be proxies for customer loyalty, such as recency and frequency of visits and time spent on site, and the percentage of return visitors. Other behaviors to track could include "send to a friend" or "print page" features that indicate a visitor's interest in sharing your site with others.

Beware of scenarios where an increase in one desirable KPI can cause a decrease in another (perhaps more valuable) KPI. This cannibalization can sometimes be a Catch-22, so the best practice is simply to track both KPIs to provide increased visibility of user behavior.

Once you've agreed upon your website goals and KPIs, now is the time to determine the goals of your multivariate testing strategy. These are often related to your website goals but can be much more granular.

Let's look at a typical retail/ecommerce website as an example. The owners of this site will obviously want their site to make money; here are some of the things they could test and measure in support of that goal:

  • Account or newsletter registration: Which navigation path successfully led to the most completions of a new account registration, newsletter signup, or other lead generation form?
  • E-commerce: What elements of the website led to the most "Add to Cart" clicks, followed by successful order completion pages (e.g. "Thank you for your order")? Which combination of product information such as graphics, descriptions, layout, and color increased average order value?
  • Promotions/offers: Did the visitor click on a promotion, and then did they take the additional step(s) necessary to complete the conversion process? Bear in mind that it is not enough to simply track clicks from the promotion; if you do this, you risk optimizing for what is most persuasive in getting users to "click" one step further, but not necessary all the way to the conversion. What you need to do is measure both clicks and conversions, and in doing so you will determine the best way to get people to convert. You may end up with fewer people clicking on the promotion, but a higher percentage of them will convert, leading to a higher absolute number of conversions.

Common errors to avoid

There are five types of mistakes that are easy to make when running multivariate tests:

  1. Improper factoring caused by poor or no isolation of individual test changes; for example, changing a headline's text, font color, and font size, all at the same time as an A/B test instead of a multivariate test.Why is this problematic? Because it's difficult or impossible to isolate the impact of each individual change — i.e., was it the font color and/or the text that caused the visitor to behave differently?
  2. Running a test too short/long. Stopping a test early because you think you have a winner increases the risk for statistically invalid data, and may increase time bias from events and/or conversion cycles. Running a test too long increases the risk of wasting time waiting for marginal results and consumes test samples that could be applied towards another test.
  3. Tracking or analyzing wrong Key Performance Indicators (KPIs). For example, measuring a KPI that is too far upstream (in a conversion funnel) from the ultimate goal, or measuring only one KPI when there are multiple indicators and/or goals that matter. There's also the risk that a measured KPI improves, but at the expense of another (untracked) KPI, or that the measured KPI is actually a bad predictor of the ultimate goal.
  4. Not targeting or segmenting visitors. This means optimizing your site or campaign for anyone and everyone by not targeting tests to include good visitors (and exclude bad visitors) and not segmenting the results. Why is this problematic? Because not all visitors are the same — they're at different stages of the buying/customer cycle, and some may be mistakenly on the wrong site altogether.
  5. Not taking action on results! This could range from not making the winning changes to your site or not taking what you've learned and running another test (the iterative test-learn repeat). The risk here is that there is no momentum gained, no ongoing strategy applied, no realization of test results, and worst of all — underwhelming ROI.
See SiteSpect In Action