What does my experimentation maturity audit score mean?
The overall experimentation score corresponds to different levels of maturity, set using our 15+ years of experience in testing and optimization. The score corresponds to the following levels of maturity.
What are the four pillars of an experimentation program?
Strategy & Culture
Strategy & culture is a hugely important area for businesses, but an area that is often not addressed until later on in the development of your experimentation program.
We see this a lot: individuals often start out experimenting in small ways, with staff members (often in marketing or product roles) using some of their working week to allocate to this practice. It’s usually not until later on, once fundamental experimentation processes and tools are in place, and results are beginning to be seen that the wider business gets on board, and experimentation is more widely considered across all functions.
The problem with this approach is that it often leads to failure. Without senior buy-in which provides:
- Funding for tools and research
- A dedicated team and the right team structure to assist the rest of the business with testings.
- The authority to test strategic elements of the business and a culture that accepts failure and learnings, leading to bigger wins and more insights and learnings.
Without the above, many programs fall by the wayside due to competing work commitments or a lack of specialist expertise needed to run a program that delivers impactful results.
People & Skills
This area of our assessment relates not only to the expertise needed to run effective experimentation programs but the structure of your teams.
While non-specialist individuals often start testing programs, it soon becomes apparent that specialist skills are needed to push beyond basic testing. This is one of the harder areas to nail down because:
- It can be hard to find people with the right skill sets.
- It’s hard to hire some of the needed roles if you don’t fully understand what skills are needed or how to test candidates for such skills.
- You need budget and buy-in for managers and leaders.
Even once you have conquered the skills gap, how you structure your team also plays an important part in how well your experimentation program will work. There are many approaches to this such as creating centers of excellence that can act as a central skills hub and resource for teams across businesses to run experiments - one of the ultimate goals for high-maturity businesses.
Process & Governance
The foundation of experimentation programs, this area relates to how you:
- Conduct research to come up with hypotheses.
- Prioritize your ideas and conduct ideation sessions to create variants.
- Your testing workflow–how you manage ideas through to development, QA, calling tests, and post-test analysis.
- The metrics you set to evaluate the program itself.
It doesn’t matter how much money, skills, buy-in, data, tools, or strategy you throw at your experimentation program, without the right processes you’ll end up doing nothing more than spaghetti testing (throwing anything at the wall and hoping something sticks).
This tends to be how most businesses start out, running ad hoc tests based on little more than hunches. But after running your first few tests it often becomes apparent that this approach doesn’t yield results, especially not consistently and more formal processes are needed.
Data & Tools
This is where most businesses start. Before they get a strategy in place, or processes, before they hire dedicated staff, they usually think about what A/B testing tool they need and sign up. But there’s a lot more to this than just getting an A/B testing tool, you need to:
- Select tools with the right level of functionality for your needs now and in the future, as well as understand their limitations.
- Think about your tool stack as a whole, and the interoperability of data and integration between them.
- Think about how you’ll combat siloed data around your business.
- Consider the level of self-service vs. specialist expertise needed to use the tools.
- Plan how you will maintain the tools and keep them up to date.
- Set out how you will regularly check the validity of tracking and data from tools.
- Think about what and how will you make data accessible and usable for different people. Whether that’s giving data analysts access to raw data and BI tools, to dashboards for senior managers.
Having tools, and data you can use and trust impacts what you test, the validity of your findings, and whether your experimentation program is working. Without it, you’re flying blind.