Paul Bernier

Should you build it or buy it? That’s a question that keeps percolating up when I talk to my colleagues in the testing and optimization world. After many discussions and much thought on the topic, I think I’ve found the answer.

Why Reinvent the Wheel?

Building something new, even though ready-made solutions may already exist, can be a compelling proposition – if you build your own testing platform you’ll have an immediate cost savings. After all, organizations like Amazon and Google have done it, so why can’t you? But the answer to that simple question isn’t an easy one. There are two factors that need to be accounted for:

  1. There is a critical need to empower business and technical users alike;
  2. There’s quite a bit of complexity, skills and resources involved in building a testing platform.

In-house tools tend to focus solely on server side testing.

Running the entire optimization program on in-house server side or backend tools shifts the testing paradigm to the development side of the organization. This solution can impact your velocity by relying solely on developers and being limited by code release cycles. Server side testing should certainly have a place in your testing program. It helps answer critical business decisions such as price sensitivity, user impacts of re-platforming, complex flow optimization and so on. While it excels at executing on complex user experience changes, it falls short when it comes to empowering non-technical users to run simpler experiments. To cope with the shortcomings of an in-house tool that sits only on the server side, organizations often start looking into hybrid models. They leverage in-house solutions for complex backend tests and use a third party tool for simpler ones. This solution comes with a high cost, since managing, prioritizing and coordinating a testing program across two different platforms adds a lot of overhead. Lastly, building a testing platform from the ground up is resource intensive. Marketers, web analysts, developers and IT resources need to be involved in this undertaking:

  • Traffic control, test priorities, sampling, random test allocation, collision prevention and avoidance;
  • Session management, consistent experience enforcement, omnichannel; 
  • Reporting and analytics, advanced segmentation and statistical significance;
  • Targeting and personalization for user attributes such as Geolocation, Mobile Devices, customer profile;
  • Preview, quality assurance and troubleshooting; 
  • Bug fixes and maintenance.

A critical question needs to be answered at this point – are you a customer centric organization or a development firm? While this may sound like a trivial question, it’s key to avoid going down a path where resources are spent building a test platform, instead of actually building tests that improve your customer’s experience and your bottom line. Building in-house might be cost effective for a developer-driven organization. But for most, a unified testing platform that empowers your team to optimize your customer experience from front end to back end, with support for single page apps out of the box and proven ROI, is a sound business decision.

Tags: Testing Optimization

Add new comment

This question is for testing whether you are a human visitor and to prevent automated spam submissions.
4 + 2 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.