Building a Successful Optimization Program

If you are an ecommerce or digital business with high traffic volume, you might be considering or re-evaluating your optimization program. Building a successful optimization program depends on careful planning, implementation, and measurement. In this article, we’ll look at three critical elements for implementing a profitable optimization program:

1. Forming a Great Experimentation Team
2. Creating an A/B Testing Plan
3. Getting Your Stakeholders on Board

Forming a Great Experimentation Team

Experimentation involves a lot of different skills coming together in unison. This can’t happen without some kind of project management and central process to hold it all together and make everything run on time. Most companies start out with a conversion manager/specialist or something similarly titled, whose background is likely to be in marketing, analytics, UX, or maybe a mix of each. If your company wishes to have an in-house Experimentation Team and not outsource to an agency, you need to consider all the skills required for this team:

• Analytics & Research – the ability to listen to and understand customers using multiple different sources of data, translate the data into insight and ideas and generate hypotheses for experimentation.
• Design & Creative – the ability to both visualize and realize the application of a test hypothesis on the front-end user experience.
• Development and Technology – the ability to actually build and produce the designed experience, as well as support the implementation and maintenance of the various technologies required for the whole operation. [source]

Can your company afford to hire someone for each of these roles? Will you need to outsource? Once your company has determined how these skills will be satisfied, you can build this team based on the resources allocated. The last two considerations are leadership support and socializing team efforts.

Leadership support includes digitally-minded executives who champion A/B testing projects and see the opportunity available and want their names associated with the financial gains that A/B testing so commonly produces. They also recognize the value of using A/B testing to prevent mistakes from happening and are just as happy to report saving the company millions as they are reporting incremental revenue. Having leadership support the Experimentation Team will help sustain your optimization program.

Finally, make sure to socialize the Experimentation Team’s efforts throughout the company. Doing this will provide clarity to other internal resources why their content is changing in a seemingly random way. Second, it creates accountability for the Team to produce and gives them a platform to talk about their successes and failures.

Creating an A/B Testing Plan

Companies with highly successful A/B testing and optimization programs have a very formal process for requesting and planning A/B tests. Despite the fact that some in our industry love to proclaim, “A/B test early, A/B test often, A/B test aggressively,” the reality is that good A/B testing can be quite involved and results are usually commensurate with effort. Without a structured plan for A/B testing, it is incredibly easy to end up with meaningless data, wasted time, and frustrated internal stakeholders.

Developing an A/B testing plan is quite simple. The following questions are a good place to start.

1. What is being A/B tested?
2. Why is it being A/B tested?
3. What are the expectations for the A/B test?
4. What are the measures of success for the A/B test?
5. What are the risks associated with running the A/B test?
6. What internal resources are required to run the A/B test?
7. Who is requesting the A/B test?
8. By when are results needed?

Individually, each of these questions is relatively easy to answer. Some are technical (#5 & #6), some are theoretical (#3), and some are political (#7 & #8). The best answers are not page-long explanations; rather, concise explanations designed to help the Experimentation Team best plan for the deployment of the A/B test.

Most people initially get stuck answering questions three, four, and five. Measures of success and risks associated with A/B testing are important enough issues that they merit their own best practices. Expectations are tough, at least until you start to get the hang of A/B testing because it is impossible to predict whether a change will result in a substantial improvement, a small improvement, or a net decline.

Your Experimentation Team should plan to have a formal A/B testing plan documented and ready to go when you start to socialize the group with senior stakeholders. The presence of this document and a few examples of the kind of information you’re looking for will go a long way towards demonstrating that you are serious about A/B testing. Most senior executives have seen enough ad hoc exercises designed to drive incremental improvement during their tenure to appreciate the level of consideration a plan conveys and understand the likelihood of failure in the absence of an A/B testing plan.

Requiring a formal A/B Test Planning document from anyone in the organization wanting to leverage the Experimentation Team, will allow the team to insert A/B tests into a long-term schedule prioritized by opportunity, risk, and political considerations. While I don’t recommend the creation of a timeline so structured that real opportunities will be lost—A/B testing is frequently an opportunistic endeavor, especially when there is a high level of awareness about A/B testing efforts—having this “roadmap” for A/B testing projects dramatically improves each A/B test’s likelihood for being successfully executed.

Ultimately, the goal for requiring a formal A/B test plan is to drive home an appropriate level of seriousness and rigor about A/B testing in your organization. Especially if your results are similar to the companies interviewed for this research, your successes will breed the desire to create more successes. If any product manager who walks through the door can have his or her A/B test jump the queue with little more than waving of the hands and saying “make the button more blue,” then you are destined to struggle to get your A/B testing program off the ground.

Conversely, if you provide clear guidance about what is required and how the requirements will be evaluated and slotted, at least in our experience, you will soon exceed your expectations and be well on your way to success.

Getting Your Stakeholders on Board

Management’s support for A/B testing projects is absolutely critical. Having buy-in from members of the senior management team will make or break A/B testing efforts. Work with these stakeholders from the beginning—in concert with the executive sponsor—and directly solicit their feedback, suggestions, and ideas that can be A/B tested by the newly formed Experimentation Team.

Another consideration is to establish a “Multivariate Testing Steering Committee” made up of senior members who are helping to decide what will be A/B tested, when, and how. I recommend socializing the A/B testing program with senior management early on. You will undoubtedly need their support to assemble the Experimentation Team and will often need budget, approval, or assistance getting A/B testing technology implemented. By approaching management with a clear plan for success, you are far more likely to gain their critical support and validation for your work.

Summary

By creating and implementing the three critical elements outlined, your company will be set up for a higher rate of success when it involves optimizing the customer experience. With a team, a plan and stakeholder support in place, the next phase is making sure your tech stack can support your goals and objectives for experimentation.

If you are considering an optimization platform, we encourage you to download our Ebook: Choosing an Optimization Platform. If you would like to see our platform in action, schedule a demo: Click Here.