Designing a Slower Site
The team structured the campaign as an A/B split test to be shown to all users. This meant that 50% of visitors to the site, regardless of digital channel (computer, smartphone, or tablet), would experience the A/B test variation of a one-second added delay in load speed for all pages.
To do this, the brand worked with SiteSpect’s optimization team to build out this A/B test experience using its native product capabilities. Rather than actually slowing down the site load time, SiteSpect manipulated the CSS to hide all page content until a one-second timer triggered, after which the full page content was presented. This gives the appearance of a slower loading site, and ensures a consistent delay experience across all users.
The team hypothesized that this change would worsen the user experience and thus negatively impact conversion rates.
It turned out that slowing down the site did in fact hurt conversions, as expected — but by how much?
The variation with a one-second page delay saw a -10.83% (p < 0.01, ~99% significance) decline to purchase conversation rates. When extrapolated to all traffic and all customers, the brand would see an approximate £10,000 reduction in revenue per day. Interestingly, when the data was segmented specifically for mobile devices, the effect was less pronounced and the conversion rate decreased by -4.38%; this was likely because smartphone users experience more delay for other reasons, such as data network connection, and thus are “trained” to expect lower performance. Another discrepancy surfaced between new and returning users. For new users, purchase rate fell by -16.56% at significance 99%, while for returning users the difference was only -6.97% at significance 82%. While these lower numbers are still impactful, this suggests that returning users with some amount of brand loyalty were more tolerant of hiccups in site performance.
“This A/B test was one of our most fruitful and received attention across the company, including from our most senior executives. While we weren’t surprised that page load times directly impact conversion behavior, the specific data and insights that this SiteSpect A/B test revealed gives us tremendous power to predict impact of any changes going forward. Elements like creative assets, functionality, and third-party modules could all add some latency, and this A/B test enables us to predict and mitigate risk, while maintaining an optimal user experience,” says the team’s Senior Digital UX & Product Manager.
Of course, this campaign was not run for a long period of time due to its statistically negative impact. The team had to strike a balance between the benefit of receiving data and generating the insight while on the flip side acknowledging that the campaign was negatively impacting site performance while live. The UX department took a pragmatic approach to this, and maintained the campaign for just long enough to reach statistical significance with their primary KPIs before ending the campaign to minimize the negative impact to users.
It’s uncommon to run a A/B test where you deliberately worsen user experience, especially a A/B test designed and implemented by a UX department. But the new knowledge generated by this A/B test is leading initiatives across the company.
The insights provided will socialize the importance of page speed on user experience to the broader business, and ensure that it is at the forefront of decision making. The brand will now prioritize projects on their development roadmap based on their potential impact on page speed, and on the other side, help to deprioritize projects whose negative page speed impact may overshadow projected benefits.
To learn more about SiteSpect, visit our website.