Jeffrey Vocell

The holidays are fast approaching. That generally means, among other things, a bump in site traffic, sales, and offers. While being aware of your testing plan this summer, now is the time to start thinking about testing and experimentation plans for the holiday.

Whether your company is risk averse and likes to run only minor tests, or allows more freedom to run significant content changing experiments a plan going into the holiday is crucial to achieving your goals.

So, how do you get the most out of your holiday testing plan?

We suggest looking at these 5 main considerations:

1) Look at prior years metrics and testing plans – Before you do anything else this should be where you start. Consider what tests were run last year and also look into key metrics changed, or were affected, as a result of those tests.

Answer questions such as:

  • How much additional traffic did we receive during the holidays?
  • Were there any key segments that made up a large percentage of traffic (i.e. mobile)?
  • What tests did we run?
  • Which tests worked well, and which did not? For the tests that worked well, why?
  • How has traffic, or the traffic from specific segments changed this year? What is the projected traffic for holidays?
  • What site changes have been made since last year that could affect testing?

Based on your answers to some of these questions you can begin to create a plan. There may be a logical next test that can be run, or a test that did not work well due to poor implementation. This information will represent the groundwork for your holiday testing plan.

2) Re-read your testing charter and goals – If you have a charter (in other words, a mission statement) for testing and business goals tied to your testing program re-read them. Along with the previous consideration of looking at prior metrics and test plans your testing goals should give you a basis for coming up with a few hypothesis for testing.

3) How comfortable are you with risk? – As mentioned above, risk can be a primary component of holiday testing. Let’s define risk – given the assumption that any test ideas that come out of these considerations will be run during the holidays, at a time when sales, traffic, and user expectations are at their highest, how could a poorly performing variation affect these metrics?

Obviously, test-results can be watched in real-time and if they surpass a threshold that makes you feel uncomfortable the test can immediately be stopped. However, some may not even be comfortable with that approach.

Before coming up with any test hypothesis, no matter how good, think about the level of risk that your team of analysts and also management stakeholders are comfortable taking. If there is a threshold amongst the group, define what it is and get it written down so you can use it as a guideline for future holiday testing. It’s likely this guideline will shift as a team gains more experience, but it should still be a reference point for everyone to see.

Whether a button color change or a whole page-theme change are your levels of risk acceptance, ensure the entire team and testing plan are in alignment.

4) Come up with a process to incorporate winning variations and discarding losing variations – During the holiday time period when traffic and other key web metrics are up, it’s likely you will reach statistical significance much faster. Because of this, ensure that communication between key stakeholders is clear and there is a process for implementing winning variations quickly. Losing variations will also be apparent more quickly. When a losing variation is determined you can quickly kick it out and move on to the next test.

5) Incorporate Targeting – Tests that are targeted to a specific segment of your audience can often contain some of the most interesting results, and perhaps the biggest wins. Particularly, if you are an online retailer with brick and mortar stores this can be a good opportunity to show local store offers, sales, and promotional giveaways.

Consider some of the following customer or visitor segments when creating targeted test campaigns:

  • Device type (laptop vs. mobile, or iPhone vs. Android)
  • Location
  • Proximity to brick and mortar store, or to competitors store
  • Time (“5 days until Black Friday!”)
  • Referral source
  • Behavior

When targeting tests, also consider how different locations celebrate holidays. For example, you may want to exclude specific physical countries or locations from test campaigns if you customize the page with specific holiday themes that do not apply to a specific geographic location.

Start looking at and planning for your holiday testing now, so by the time the holidays come you will be prepared and ready to execute a testing plan that delivers on business results.

Tags: A/B & Multivariate Testing