A/B Testing Solutions: Should you Build or Buy? (Part 2 of a series)

By Paul Bernier

December 1, 2017


Updated 10/20/2020

You can catch Part 1 of these series here: https://www.sitespect.com/blog-aligning-single-page-apps-optimized-customer-experience-6/.

Should you build it or buy it? That’s a question that comes up all the time when I talk to my colleagues in the A/B testing and optimization world. After many discussions and much thought on the topic, I think I’ve found the answer.

Why Reinvent the Wheel?

Building something new, even though ready-made solutions may already exist, can be a compelling proposition – if you build your own A/B testing platform you’ll have an immediate cost savings, you can customize it to exactly what you want (provided you have the resources to develop it), and your organization has ultimate control. And after all, organizations like Amazon and Google have done it, so why can’t you? But the answer to that simple question isn’t an easy one. There are two factors that need to be accounted for:

  1. There is a critical need to empower business and technical users alike;
  2. There’s quite a bit of complexity, skills, and resources involved in building an A/B testing platform.

In-house tools tend to focus solely on server side A/B testing

Running the entire optimization program on in-house server-side A/B testing tools shifts the A/B testing paradigm to the development side of the organization. This solution can impact your velocity by relying solely on developers and being limited by code release cycles. Server-side A/B testing should certainly have a place in your A/B testing program. It helps answer critical business decisions such as price sensitivity, user impacts of re-platforming, complex flow optimization and so on. While it excels at executing on complex user experience changes, it falls short when it comes to empowering non-technical users to run simpler experiments. To cope with the shortcomings of an in-house tool that sits only on the server side, organizations often start looking into hybrid models. They leverage in-house solutions for complex backend A/B tests and use a third party tool for simpler ones. This solution comes with a high cost, since managing, prioritizing and coordinating an A/B testing program across two different platforms adds a lot of overhead. Lastly, building an A/B testing platform from the ground up is resource intensive. Marketers, web analysts, developers, and IT resources need to be involved in this undertaking. Just some of the factors that need developing include:

  • Traffic control, A/B test priorities, sampling, random A/B test allocation, collision prevention and avoidance
  • Session management, consistent experience enforcement, omnichannel
  • Reporting and analytics, advanced segmentation and statistical significance
  • Targeting and personalization for user attributes such as Geolocation, Mobile Devices, customer profile
  • Preview, quality assurance and troubleshooting
  • Bug fixes and maintenance

What’s the Focus of Your Organization

A critical question needs to be answered at this point — are you a customer centric organization or a development firm? While this may sound like a trivial question, it’s key to avoid going down a path where resources are spent building an A/B testing platform, instead of actually building A/B tests that improve your customer’s experience and your bottom line. Building in-house might be cost effective for a developer-driven organization. But for most, a unified A/B testing platform that empowers your team to optimize your customer experience from front end to back end, with support for single page apps out of the box and proven ROI, is a sound business decision.

To learn more about SiteSpect, visit our website.


Paul Bernier

Paul Bernier

Paul Bernier is Vice President of Product Management at SiteSpect. He has a background in website optimization, recommendations, and development, as well as web analytics. He is based in Boston.

Suggested Posts

Subscribe to our blog: