Building a Culture of Optimization: Planning (Part 1)

Whether you have been A/B testing for years or you are just getting started, building a successful optimization program depends on careful planning, implementation, and measurement. This is the first in a three-part series of articles that looks at the steps involved in creating a successful optimization program.

In this article, we’ll look at three critical elements in optimization planning:

Forming a Great A/B Testing Team
Getting Your Stakeholders on Board
Writing a Formal A/B Test Plan

Forming a Great A/B Testing Team

John Donne said that “No man is an island” and in A/B testing this is more true than not. Unlike in web analytics where one good analyst can provide great insights, the A/B testing process is typically too political and requires too many resources to be executed alone. Consider what can be involved in making a change to a web site:

Someone decides that a change is warranted.
Management needs to review the change request, and if approved, initiate the change management process.
Depending on the change, developers, designers, marketers, and copywriters may be required.
Information Technology needs to deploy the change.
Quality Assurance needs to A/B test the change.
Analytics needs to validate the change.

Now, compare that to situations where an A/B testing program is in place. Instead of someone deciding a change needs to be made, marketers ask themselves, “what would happen if we changed X, Y, or Z, or even all of them?” While the best A/B testing solutions help to minimize the need for some of the resources required—most commonly developers and IT—there is still a need for a cohesive group to create and run A/B tests, and analyze their results.

Address this need by forming an A/B Testing Team of appropriate resources within your organization. Then give this team a mandate for improvement and allow them to incrementally prove their value and earn the right to A/B test increasingly large projects. Assign your internal super-stars to this team to couple their insights with technology and process capable of measurably demonstrating their brilliance. Give them access to other internal resources, at least as long as they continue to produce results.

The two most important roles on your A/B Testing Team are the project manager and the executive sponsor. The project manager needs to be someone who combines incredible organizational skills with a shocking enthusiasm for change. Their role is to ensure that defined A/B testing processes are followed to the letter, thereby increasingly the likelihood of successful A/B tests. This person does not have to be a jack-of-all-trades, but they have to know Jack (or Jill) to get their job done:

The skills to produce a sound statistical design are not required, but the ability to understand basic statistical principles is.
The skills to create brilliant design are not required, but the ability to think from the perspective of an end user is, because visitors do not interact with your site the way you think they do.
The technical skills to write JavaScript, HTML, or ActionScript are not required, but the ability to keep all resources—technical or otherwise—on task and on time is. Seamlessly juggling people, process, and technology are the hallmarks of an effective project manager.

The executive sponsor’s role is obvious: without an internal champion for change on the management team, most A/B testing projects are dead in the water. Fortunately there is a new class of digitally minded executives who are actively signing up to manage A/B testing projects. These soon-to-be “rock-stars” clearly see the opportunity available and want their names associated with the financial gains that A/B testing so commonly produces. They also recognize the value of using A/B testing to prevent mistakes from happening and are just as happy to report saving the company millions as they are reporting incremental revenue.

Finally, make sure to socialize the A/B Testing Team’s efforts throughout the company. Doing this—essentially announcing to the world your commitment to optimization and demonstrating this commitment by assigning some of your most valuable resources—has a two-fold effect. First, it clarifies to other internal resources why their content is changing in a seemingly random way.

Second, it creates an expectation for the Team to produce and gives them a platform to talk about their successes and failures. Obviously, building and socializing a team like this requires senior management’s support.

Getting Your Stakeholders on Board

Management’s support for A/B testing projects is absolutely critical. Having buy-in from members of the senior management team will make or break A/B testing efforts. Work with these stakeholders from the beginning—in concert with the executive sponsor—and directly solicit their feedback, suggestions, and ideas that can be A/B tested by the newly formed A/B Testing Team.

You might consider establishing a “Multivariate Testing Steering Committee” made up of senior people who are helping to decide what will be A/B tested, when, and how. I recommend socializing the A/B testing program with senior management early on. You will undoubtedly need their support to assemble the A/B Testing Team and will often need budget, approval, or assistance getting A/B testing technology deployed. By approaching management with a clear plan for success, you are far more likely to gain their critical support and validation for your work.

Writing a (Formal) A/B Testing Plan

Companies with highly successful A/B testing and optimization programs have a very formal process for requesting and planning A/B tests. Despite the fact that some in our industry love to proclaim, “A/B Test early, A/B test often, A/B test aggressively,” the reality is that good A/B testing can be quite involved and results are usually commensurate with effort. Absent a structured plan for A/B testing, it is incredibly easy to end up with meaningless data, wasted time, and frustrated internal stakeholders.

Fortunately, developing an A/B test plan is quite simple. Make sure yours answers the following questions:

What is being A/B tested?
Why is it being A/B tested?
What are the expectations for the A/B test?
What are the measures of success for the A/B test?
What are the risks associated with running the A/B test?
What internal resources are required to run the A/B test?
Who is requesting the A/B test?
By when are results needed?

Individually, each of these questions is relatively easy to answer. Some are technical (“what risks are there” and “what resources are required”), some are theoretical (“what are the expectations”), and some are political (“who is requesting” and“by when are results needed?”). The best answers are not page-long explanations; rather, concise explanations designed to help the A/B Testing Team best plan for the deployment of the A/B test.

Most people initially get stuck answering questions three, four, and five. Measures of success and risks associated with A/B testing are important enough issues that they merit their own best practices. Expectations are tough, at least until you start to get the hang of A/B testing, because it is impossible to know ahead of time whether a change will result in a substantial improvement, a small improvement, or a net decline.

Your A/B Testing Team should plan to have a formal A/B testing-planning document ready to go when you start to socialize the group with senior stakeholders. The presence of this document and a few examples of the kind of information you’re looking for will go a long way towards demonstrating that you are serious about A/B testing. Most senior executives have seen enough ad hoc exercises designed to drive incremental improvement during their tenure to appreciate the level of consideration a plan conveys and understand the likelihood of failure in the absence of an A/B testing plan.

Most importantly, requiring a formal A/B test-planning document from anyone in the organization wanting to leverage the A/B Testing Team, including members of the A/B Testing Team themselves, will allow the team to insert A/B tests into a long-term schedule prioritized by opportunity, risk, and political considerations. While I don’t recommend the creation of a timeline so structured that real opportunities will be lost—A/B testing is frequently an opportunistic endeavor, especially when there is a high-level of awareness about A/B testing efforts—having this “roadmap” for A/B testing projects dramatically improves each A/B test’s likelihood for being successfully executed.

Ultimately, the goal for requiring a formal A/B test plan is to drive home an appropriate level of seriousness and rigor about A/B testing in your organization. Especially if your results are similar to the companies interviewed for this research, your successes will breed the desire to create more successes. If any product manager who walks through the door can have his or her A/B test jump the queue with little more than waving of the hands and saying “make the button more blue,” then you are destined to struggle to get your A/B testing program off the ground.

Conversely, if you provide clear guidance about what is required and how the requirements will be evaluated and slotted, at least in our experience, you will soon exceed your expectations and be well on your way to success!

(For more information, download the whitepaper titled “Successful Web Site Testing Practices: Ten Best Practices for Building a World-Class Testing and Optimization Program.”)

learn more about SiteSpect, visit our website.