The Yellow Brick Road to A/B Testing Greatness

Landing Page, UX, and Product! Oh my!

Dealing with competing A/B testing strategies can make you feel like Dorothy in the Wizard of Oz. We hear from marketing teams all the time about their struggles with diverging road maps. If your A/B testing program is caught in a tornado with no yellow brick road in sight, here are some strategies for finding focus and creating a road map to A/B testing success.

Use Your Brain – Plan Ahead

All businesses have short, medium, and long term plans to guide their actions and help steer them toward their vision. These plans develop over time and so should your optimization strategy.

Don’t set anything in stone: Most successful organizations don’t become successful by chance; they plan, but they also remain open and able to shift at a moment’s notice. Apply the same idea to your optimization program. Have short, medium, and long term milestones on your road map. The further out on your road map the more flexible your A/B testing schedule should be. Your road map will remain more relevant if you focus your scheduling efforts to only 2 – 3 months out.
Keep it flexible: Business priorities and initiatives evolve over time; your road map needs to evolve, too. Consider the optimization road map a living document rather than a static list of ideas. Use multiple A/B testing tracks that complement each other and consider the various device types, areas of the site and decision metrics available to support these tracks. Staying flexible helps you maintain momentum and account for factors like seasonality, site releases, and new initiatives.

Use Your Heart: Aggregate & Set Priorities

It is not unusual for organizations to have multiple stakeholders involved with optimization. This can make it hard to focus. Each team has different goals and motivations for A/B testing. It’s your job to balance them. The best way to achieve balance is to aggregate these roadmaps, provide the teams with a shared focus and agree on a set of criteria to prioritize A/B tests fairly.

Don’t focus on the loudest voice in the room: It can be easy to hear only the ideas from the loudest contributors. It’s easy to go with the HiPPO (highest paid person’s opinion), but you’ve got to ask yourself if it’s a great idea to set priorities based on one person’s power. The end result is often an unbalanced A/B testing plan with disengaged stakeholders. Remember, ideas from all areas of your business have merit and the best concepts can come from surprising places.
Define criteria for setting priorities: Define criteria that are important to the business and score A/B test ideas based on them. Consider how closely they align with the company’s goals and initiatives. This determines how important each idea is to the business. For example:
- What is the potential impact? You can use quantitative and qualitative data to justify priorities and focus on high traffic areas of the site with high bounce or low conversion rather than unqualified locations.
- Consider ease of setup. You want a balance of complex and simple A/B test builds to match your resource availability.
- How re-usable is the A/B test? If it is relevant for multiple locations on your site or across multiple domains then its impact and priority are likely to multiply.
These criteria help you base decisions on facts over emotion.

Have Courage

The most successful organizations involve and engage all teams effectively, which requires relevant communication to all stakeholders. The same can be said for companies with the most productive optimization programs who often see a cultural change in decision making and their digital strategy.

Don’t operate in isolation: Without healthy two way communication, your optimization program risks isolation. It is tricky to find the right level of communication, but without it, building a culture of data-driven decision making and A/B testing across the organization isn’t possible.
Communicate your plans: Let the entire business know the immediate schedule for your road map. People must see the optimization plan and A/B testing status. They have to feel involved with the process and most importantly that their ideas are making it into the schedule. It is especially important to foster ideas and avoid inefficiency and duplication of effort when you have multiple teams and departments involved. The same is true of communicating A/B test results, good, bad, or ugly. If you can celebrate the new intelligence gained on an A/B test rather than purely good results, you are more likely to foster ideas and greater engagement across all departments.