When to A/B or Multivariate Test?

One of the questions that comes up all the time in web optimization — especially for bigger design changes — is whether you should A/B test or multivariate test. Each type of campaign structure serves a unique purpose and answers a different question. The simple answer is that multivariate testing is for situations where you have multiple elements, of which any combination (or multiple combinations) is a possibility. A/B testing is for comparing versions of the same element, or complete groups of elements. But, there are a few other considerations you’ll want to keep in mind before you decide which type to use.

Quick Definitions: What Is An A/B Test And A Multivariate Test?

A/B Test (Also A/B/n Test): An A/B test involves two or more variations of an element, or group of elements, compared against each other. A/B tests can have multiple changes making up a single experience, and you may test more than one version of the experience, but you are always testing full experiences against each other. For example, if you want to optimize your site header, you may tweak the logo size and header height for each version, and you may have five different versions, but each group of changes is considered one variation.

Multivariate (MVT) Test: A multivariate test compares different combinations of multiple elements. You could have as few as three variations, but those variations measure several separate elements in conjunction with each other. For example, a new hero image might contain a few options for a graphic, copy, and call to action, of which different combinations must be tested.

Considerations for Multivariate Testing

The terms for when to A/B test are pretty clear. If you are comparing complete versions of one experience, that’s an A/B test. However, multivariate tests have a few more factors to keep in mind.

1. Do you have enough traffic?

A multivariate test will split your users into more variations, and therefore any result will take longer to reach statistical significance. This may not be a problem. If your multivariate test is on a higher-traffic page, you may have enough users so that splitting them into multiple variations won’t slow down your results. However, if you are concerned about the amount of traffic or need results in a shorter amount of time, you may want to forgo multivariate testing in favor of A/B testing.

2. Is every combination of elements an actual contender?

Multivariate testing takes each element and combines it with every other element. When you do this with every combination in SiteSpect it’s called a full factorial. Now, there may be some number of combinations that just won’t work, perhaps for branding, style, or they just don’t make sense. You can exclude these combinations from your multivariate test. We call this a fractional factorial. However, if you reach a point where fewer combinations work together than don’t work together, it may not be worth testing this way. If you find yourself in this situation, you should evaluate if certain elements are really worth testing in combination. You may find that it’s more efficient to do an A/B test.

3. Are you at the beginning or end of your iteration process?

The previous two considerations are reasons to steer away from multivariate testing. Here, we have a reason you may opt for it. When you are at the beginning of a design or optimization process, you likely have relatively little information about what works and what doesn’t. Multivariate testing is a great way to get that initial data and narrow down your options. You can find out which elements are more likely to move the needle, so you can prioritize future tests accordingly. If you’re further along in your iteration process, maybe you want to focus on a few key elements. But multivariate testing is a great way to quickly reveal which elements are making the biggest impact for users.

What to do if you have multiple elements but decided not to multivariate test

If you’ve just read the above considerations and realize that perhaps multivariate testing isn’t the best option for this particular campaign, you can still get the information you need. Rather than throwing all of your elements into a pot and testing them all at the same time, you will simply A/B test individual elements against each other to determine winners there before throwing every option into the mix.

Let’s return to the example of a hero image with a graphic element, some copy, and a call to action. You will start by running three separate A/B tests – one to A/B test the image, one to A/B test the copy, and one to A/B test the button. If you start to see certain elements winning, you can then begin A/B testing different hero images that incorporate different elements. The difference is that you do this in stages rather than all at once.

You can see a brief example of an A/B test versus a multivariate test in this video. 

Check out more Best Practice videos here. To learn more about SiteSpect, visit our website.