Case Study

ShopNBC Finds Hidden Treasures Through Multivariate Testing

Founded
1990

Industry
Retail

Headquarters
Eden Prairie, Minnesota

ShopNBC.com is operated by ValueVision Media, a multi-channel, direct-to-consumer retailer that sells general merchandise to consumers through television, the Internet, and direct mail. Founded in 1990, the Company operates the third largest television home shopping network in the United States — ShopNBC — and is a leader in upscale television home shopping. ShopNBC.com offers customers a rich shopping experience filled with Web-Exclusive offers, customized search features, interactive buying guides, videos and auctions. ShopNBC also offers a top quality and rich TV-like experience thru their newly launched initiative, ShopNBC.TV — the excitement of live demonstration selling combined with a rich internet experience.

Challenge

With a reputation for providing visitors with an exciting and efficient shopping experience, ShopNBC needed a way to quickly test site changes without disrupting their online store and without burdening valuable IT and marketing resources.

Solution

ShopNBC’s Internet marketing team turned to SiteSpect, both for its non-intrusive testing technology and professional services. SiteSpect enabled ShopNBC to design and implement A/B and multivariate tests for validating new creative ideas and optimizing promotional offers, product copy, and user interface elements.

Results

Using SiteSpect, ShopNBC’s Internet marketing team tested the impact of variations on copy, content, and layout changes without any involvement from their IT department.
A recent test designed to optimize the product detail pages on ShopNBC.com resulted in a 16% increase in the average order value across all departments and product lines. The optimizations from this test have the potential to yield many millions of dollars in incremental revenue.
Simultaneously testing multiple promotional offers enabled ShopNBC to quickly react to changes in customer demand and inventory availability.
In addition to increased revenue through the optimization of ShopNBC.com, SiteSpect helped ShopNBC to avoid lost revenue from potential changes that, had they been implemented blindly, would have hurt sales.

It Starts with a Hypothesis

When ShopNBC first started testing, their goal was to answer a question whose answer they thought they already knew. The Internet marketing team’s initial goal with testing was to determine whether their instincts were correct. As their hypotheses became more complex, and they progressed from A/B testing to multivariate testing with SiteSpect, ShopNBC learned that combinations of variations could sometimes produce unexpected positive results.

The Power of Interaction Effects

What ShopNBC experienced was one of the benefits of Multivariate Testing: Interaction Effects. By testing multiple Web-site elements (“test factors”) and content variations simultaneously, the Interaction Effects reflected the impact that variations had on each other when presented in combination with each other. In many cases, these winning combinations far-outperformed each of the individual variations — the effect of interactions that occurred beyond the additive effects that ShopNBC had anticipated.

And The Winner Is . . . . .

ShopNBC recently designed a test to optimize the layout and content on the product detail pages of ShopNBC.com. Their hypothesis was that by drawing attention to key elements of the page, they could encourage more customers to initiate and complete purchases that they were considering. ShopNBC suspected that several key features were getting lost on the page and that they were losing sales as a result.

Working with SiteSpect’s Professional Services Team, ShopNBC identified five key factors that they hypothesized would influence the actions of users who were contemplating a purchase.

Variations that were tested on the product detail pages:

Location and size of the Add-to-Cart Button
Headlines identifying cross-sell items
Styles of tabs that link users to product details
Color, size, and style of Clearance and Limited Time pricing offers
Highlighting of payment options for qualified buyers

Case Study ShopNBC graphic

The multivariate test was initially implemented as a screening test with the goal of identifying which variations had a positive impact on the performance of the product detail pages. Subsequent tests were run in order to zero-in on the optimal combination of variations. The results of the test were surprising; not only was ShopNBC able to achieve their primary objective by getting 4% more shoppers to begin the checkout process, the average value of the items in those users’ shopping carts increased by 16%! Looking closer at the results, they noticed that, individually, each of the winning variations of the five factors that were tested played a relatively small role in the results that were achieved. When tested in combination with variations of the other factors, however, significant interaction effects occurred where the whole benefit was truly greater than the sum of the parts. This multivariate test, one of many run by ShopNBC with SiteSpect’s guidance, revealed far more than originally anticipated. In this case, a collection of small changes added-up to a significant improvement for ShopNBC.com and its customers.