Case Study

Mozilla Optimizes Web Site Content and Performance through Multivariate Testing

Firefox Mozilla Logo

 

Founded
1998

Industry
Software Development

Headquarters
San Francisco, CA



Mozilla is a free software community founded in 1998 by members of Netscape. The Mozilla community uses, develops, spreads and supports Mozilla products, thereby promoting exclusively free software and open standards, with only minor exceptions. The community is supported institutionally by the non-profit Mozilla Foundation and its tax-paying subsidiary, the Mozilla Corporation.

Mozilla’s current products include the Firefox web browser, Thunderbird e-mail client (now through a subsidiary), Bugzilla bug tracking system, Gecko layout engine, Pocket “read-it-later-online” service, and others.

Challenge

Make website testing and optimization an ongoing priority, focusing not only on creative content
elements, but also on page load speed.

Solution

Mozilla turned to SiteSpect for its non-intrusive approach to website optimization that does not require JavaScript page tagging or ongoing IT involvement.

Results

  • Mozilla’s first two A/B tests resulted in double-digit engagement improvements of 33% on the First Run page and more than 50% increase in engagement on the What’s New page.
  • A 32-level multivariate test revealed that the simplest creative presentation increased conversion rates by 2.3%. This improvement translated to an additional 1.7 million Firefox downloads per year.
  • An A/B test that applied page load time optimizations found that a streamlined version of the site reduced page load time by 2.2 seconds, resulting in an increased conversions rate of 15.4%, ultimately yielding an additional 10 million Firefox downloads per year.

Early Results

With the help of SiteSpect’s non-intrusive platform, Mozilla found an easy way to start testing and optimizing its site. As opposed to previous attempts with other products (including a homegrown solution), Mozilla is now able to run tests on an ongoing basis — at least two concurrent experiments at any given time. Mozilla’s overall results with SiteSpect have been compelling. The first two A/B tests saw double-digit improvements to engagement of 33% on the First Run page and more than 50% on the What’s New page.

Mozilla’s first two A/B tests
saw double-digit engagement
improvements — 33% on the First
Run page and more than 50% on
the What’s New page.

Beyond Engagement

In addition to enhancing visitor engagement, Mozilla wanted to increase its conversion rate on Firefox downloads. The conversion rate on the main Firefox product page (http://www.mozilla.com/firefox) was already high, but Mozilla wondered what it would take to boost that number even higher. To find out, a 32-level multivariate test was run on the following page factors:

  • Features copy
  • Footer detail
  • Download button
  • Tips copy
  • Download stats box (for “social proof”)
Factors that Mozilla tested
3 out of 32 page variations Mozilla tested

Simple = Better

The test concluded that only one element positively impacted the download conversion rate: the download stats box. All other elements had either no effect or actually drove conversions down. In fact, the simplest overall page performed the best, boosting the conversion rate by 2.3% (99% confidence), increasing Firefox downloads by 1.7 million per year.

Faster = Better?

But why did the simpler variation perform better? One hypothesis was that more users converted because the page loaded faster. To test this hypothesis, Mozilla analyzed the distribution of page load times for visitors who downloaded Firefox and visitors who didn’t.

The results showed that half of the ‘downloaders’ loaded the landing page in less than 2.10 seconds while half of the ‘non-downloaders’ loaded the landing page in less than 3.67 seconds, which is 75% slower.

Testing Speed Improvements

Given the page load speed data, Mozilla utilized SiteSpect to run an A/B test to evaluate its hypothesis that page load speed is a factor that influences conversion rate. It was a simple A/B test in which SiteSpect delivered streamlined page variations using a variety of optimization techniques on the site’s JavaScript and CSS content.

The Results Are In: Speed – or Rather Lack Thereof – Kills Conversion

Reducing the page load time by 2.2 seconds increased conversions by 15.4% (99.99% confidence), which translated into an additional 10 million downloads per year. Interestingly, when segmenting the results by browser, the increase in speed had an even more dramatic effect on users of Microsoft Internet Explorer (Firefox’s target audience for conversion), who experience an increased conversion rate of 27.6%. Clearly, speed influences end-user behavior. By testing performance optimizations with SiteSpect, Mozilla was able to produce a winning combination of content that reduced page load time and boosted conversions dramatically.

Want to learn more? Let's talk.