How to Ramp Up from a Few A/B Tests to Hundreds of Tests in 9 Months

If you want to run more A/B tests, it is critical to understand how to scale your optimization organization. We chatted with Bobby Patterson, Founder and CEO of Optimetrum and former head of the digital analytics and optimization group at Target, a SiteSpect client. He spoke about how companies can benefit from the best practices recommended in his recent post “Building a Testing and Optimization Program: Ingredients for Success.”

Bobby Patterson, Founder and CEO,
Optimetrum

What are the top 3 things companies should be doing in order to ramp up their A/B testing?

I recommend these 3 strategies:

1. Create the right environment. It’s important to create a safe environment for fostering A/B test ideas and experimentation at the beginning of your program. A safe environment involves transparency around how you determine A/B test results and a positive focus on what can be learned from A/B tests that end in a statistical tie or failure.
2. Develop a repeatable process. Start your A/B testing program out small and build an operating model that is well-defined for business stakeholders. The process needs to clearly define each step in the optimization lifecycle, along with roles and responsibilities. Start with the steps that involve participation outside the A/B testing team, such as the process for submitting A/B test nominations and how results will be communicated. Try out your new process on a small scale and incorporate feedback from key partners along the way. Buy-in is important, but you will achieve greater operating efficiency by avoiding an approach that requires approval from a steering committee. Once you have the external interfaces to your operating model working smoothly, take some time to streamline internal team activities to help increase your throughput.
3. Extend your team with other resources. It can take years to staff up your team for continual A/B testing. A good place to begin is by negotiating help in building campaigns from your web development team. If your company has analytics or development resources in another part of the world, use the time zone differences to your advantage. And you’ll be surprised at how receptive other teams will be to getting involved. People love the idea of experimenting and find it rewarding to put value on their work and get customer feedback on design decisions. It’s also helpful to have professional service hours to manage demand during peak periods of time. It takes the pressure off by having a “flex team” ready when you need them.

How do you set up your team in order to achieve high  A/B test volume?

There are a few basic roles in the A/B testing team including a Program Manager, an Analytics expert with a Statistics background, Developers, and a Quality Assurance expert.  Your Program Manager should have project management and leadership experience and will be responsible for putting the business process in place. Developers must be proficient in HTML, CSS, and JavaScript, and should understand technical aspects of campaigns. And as I noted previously, it’s helpful to have professional service assistance in case you need it.

What are best practices for receiving A/B test ideas from different parts of the organization?

A good optimization program will cast a wide net for A/B test ideas. Your A/B testing Program Manager should meet individually with leaders and talk about how A/B testing and optimization can help improve their area of the business. They may end up wanting more than you can deliver, so you’ll need to educate them on the operating model for A/B testing.  Always remember to ask if they have people who can help with content, development, or analytics.

What does it mean to A/B test different aspects of your site to leverage all your traffic?

If you want to do a lot of A/B testing, plan to use 100% of traffic from your site in order to run as many A/B tests in parallel as possible. Work with business owners to develop an optimization roadmap that identifies which parts of the site you want to focus on for the upcoming period. Aligning with the broader digital product roadmap is critical to ensuring that you are able to use A/B testing as a way to inform design decisions at the right time.

A common challenge for optimization leaders lies in dealing with many competing priorities. Aligning A/B tests with a roadmap and business priorities will help, but it’s also critical to weigh the potential value of each A/B test and prioritize accordingly. As an example, optimizing entry pages or checkout (for an e-commerce site) may yield significant gains compared to A/B testing secondary areas of your site.

So what is your organization doing to become an optimization organization? We’d love to know! Tell us by leaving a comment below.

To learn more about SiteSpect, visit our website.