Communicating Experiments Up and Down the Chain of Command

When you’re running an optimization program, your job is to design and conduct experiments that increase conversions and engagement on your digital channels. But that’s not where it ends — just as important as improving metrics across your digital channels is communicating those results across your organization. It can be a challenge to do this effectively, both when communicating up the chain of command and down it. Both are important, and there are some strategies you can employ to make your communications as effective as possible and generate support and enthusiasm across the organization. 

I spoke with John Ostrowski, Principal — Growth Experiments at iTech Media, about how to champion your experimentation program throughout your organization. When it comes to experimentation, Ostrowski says, “I think about 5 “Ps” when advocating experiments: Purpose, Participation, Policies, Persistence, and Perception. A communication plan should support the entire spectrum. If you have to prioritise, focus on communicating the purpose of your program and getting more minds to participate. Those generate strong momentum to keep the experimentation flywheel flying.”

In addition, you’ll want to consider: Your audience — who the stakeholders are; The stages of reporting — roadmap, results, and process; And finally, the frequency of communication — typically weekly or monthly.

Who Are Your Stakeholders?

The first question to ask when putting together a report is who are your stakeholders? This is where you consider participation and policies. Who needs to be involved, and what is the company culture around reporting? With digital experimentation you will likely be reporting to three audience categories: senior level executives across departments; analysts, strategists, or marketers; product managers, and technical or IT managers. While the facts of your reports won’t change, the level of background, context, and details will need to be tailored to your audience. 

When communicating to senior level executives, focus on results and tailor them to the executive’s department. These reports should be streamlined, and you should present your data in the most straightforward form. You can also consider creating extra slides that cover more details and hiding them in your presentation in case questions arise. As Ostrowski emphasizes, this is also your opportunity to ask questions, understand business goals, and keep everyone involved in the optimization process. The combination of results oriented reporting and collaborative discussion builds confidence and investment, and helps you keep your optimization program moving in the right direction.

For analysts, strategists, or marketers, you want to dive more deeply into your process. You’ll likely be collaborating with these teams often, so this group will already understand the day to day work of experimentation. Here’s where you can dive into what you’ve learned from failed experiments or more ambiguous metrics. For technical and IT managers, dig into how your experiments are built and how they fit into the architecture of your digital channels. Look at the performance impact of your experiments, and how the site functions overall. Focus on where your work intersects and the goals you have in common.

Reporting Stages

There are several steps throughout the process where you’ll want to report to your colleagues. Consider which milestones you’ll report to each group of stakeholders. This is where you can be very persistent, Ostrowski says. You can break these stages of communication into roadmap, results, and process.

Roadmap communication should cover both where you’ve come from and where you’re going. No experiment happens in isolation, so anticipate the questions about how this impacts the next set of A/B tests, the next quarter’s goals, and other concurrent work. Communicate your long term goals and how each experiment fits into them. 

Results communication should cater to the audience and focus on their interests. Think about the relevant metrics and outcomes. Strive to make this communication as clear and simple as possible so as not to leave room for misinterpretation.

Process communication should focus on resource allocation, time spent, and how the experimentation strategy fits into larger goals and initiatives. This helps you gather the support you need, and fosters communication and collaboration between departments.

Reporting Frequency and Style

How often you communicate and how you do it will vary based on your organization’s culture and your specific needs. But there are some great ideas to help keep your colleagues engaged for every frequency of communication — and each style of communication can influence the wider perception of your experimentation program. Ostrowski shares, “At iTech, we consider three layers for communicating A/B test results. 

1. Product Team: Usually through an “Experiment Meeting Review” where we run a retrospective from hypothesis to learnings and make a decision on rollout, iteration, or dump.
2. Experimentation chapter:  We share intriguing results and best performers with optimizers from other teams so others can leverage findings.
3. Organization: At the end of each month, we will share a newsletter with performance across products, specific stats such as the total number of A/B tests per product, and avg. lift. We also use the same piece to communicate news on larger initiatives the chapter is leading. Andrea Corvi, Senior Experimentation Manager at iTech Media, has an excellent template and a unique way to communicate ‘rescued conversions’ from A/B tests that fail that reminds us also to celebrate when we don’t roll out an experience that’s harmful to the business. The best part is the closing CRO Meme at the end of each issue!

You can also think about your communication levels in terms of frequency and break this into weekly and monthly communications. Weekly, consider the following ideas: 

• 1:1 meetings or team syncs
• Sharing calendars or planning docs
• Email status updates to your immediate team and manager
• Share highlights on a company-wide dashboard or wiki

Weekly communications should be quick updates and status reports, rather than in depth presentations. 

Monthly, you can show off your polished slides or presentations, and organize meetings with larger groups from your organization. Here’s where you can really advocate and evangelize your work. Consider your audience, but this is your chance to dive deep into what worked, what didn’t, and why.

Conclusion

Communicating your work is an important part of experimentation — it’s the key to generating interest, support, and collaboration. Our final bit of advice, no matter who your audience or what the medium, is to highlight “why.” Why are you sharing this information? Why did you design this experiment? Why did visitors perform the way they did? Keep these guidelines in mind to make your communication as effective as possible.

****

“Positive” John Ostrowski helps leaders to understand uncertainty for better decision-making with a unique Growth approach. He’s confident sharing his angle after coaching multiple teams on both sides of the force, marketing and product.

After navigating the hyper-growth of education technology during the pandemic leading Conversions at Brainly, he’s now working as a Principal for Growth Experiments at iTech Media, guiding Product Managers navigating end-to-end the process of experimentation.

Connect with him on LinkedIn. He’s actively writing about Growth and Experimentation with a stock market twist sprinkled with positivity.