With a reputation for providing visitors with an exciting and efficient shopping experience, ShopNBC needed a way to quickly test site changes without disrupting their online store and without burdening valuable IT and marketing resources.
ShopNBC's Internet marketing team turned to SiteSpect, both for its non-intrusive testing technology and professional services. SiteSpect enabled ShopNBC to design and implement A/B and multivariate tests for validating new creative ideas and optimizing promotional offers, product copy, and user interface elements.
When ShopNBC first started testing, their goal was to answer a question whose answer they thought they already knew. The Internet marketing team's initial goal with testing was to determine whether their instincts were correct. As their hypotheses became more complex, and they progressed from A/B testing to multivariate testing with SiteSpect, ShopNBC learned that combinations of variations could sometimes produce unexpected positive results.
What ShopNBC experienced was one of the benefits of Multivariate Testing: Interaction Effects. By testing multiple Web-site elements ("test factors") and content variations simultaneously, the Interaction Effects reflected the impact that variations had on each other when presented in combination with each other. In many cases, these winning combinations far-outperformed each of the individual variations — the effect of interactions that occurred beyond the additive effects that ShopNBC had anticipated.
ShopNBC recently designed a test to optimize the layout and content on the product detail pages of ShopNBC.com. Their hypothesis was that by drawing attention to key elements of the page, they could encourage more customers to initiate and complete purchases that they were considering. ShopNBC suspected that several key features were getting lost on the page and that they were losing sales as a result.
Working with SiteSpect's Professional Services Team, ShopNBC identified five key factors that they hypothesized would influence the actions of users who were contemplating a purchase.
Variations that were tested on the product detail pages:
The multivariate test was initially implemented as a screening test with the goal of identifying which variations had a positive impact on the performance of the product detail pages. Subsequent tests were run in order to zero-in on the optimal combination of variations.
The results of the test were surprising; not only was ShopNBC able to achieve their primary objective by getting 4% more shoppers to begin the checkout process, the average value of the items in those users' shopping carts increased by 16%! Looking closer at the results, they noticed that, individually, each of the winning variations of the five factors that were tested played a relatively small role in the results that were achieved. When tested in combination with variations of the other factors, however, significant interaction effects occurred where the whole benefit was truly greater than the sum of the parts.
This multivariate test, one of many run by ShopNBC with SiteSpect's guidance, revealed far more than originally anticipated. In this case, a collection of small changes added-up to a significant improvement for ShopNBC.com and its customers.
ShopNBC.com is operated by ValueVision Media, a multi-channel, direct-to-consumer retailer that sells general merchandise to consumers through television, the Internet, and direct mail. Founded in 1990, the Company operates the third largest television home shopping network in the United States — ShopNBC — and is a leader in upscale television home shopping. ShopNBC.com offers customers a rich shopping experience filled with Web-Exclusive offers, customized search features, interactive buying guides, videos and auctions. ShopNBC also offers a top quality and rich TV like experience thru their newly launched initiative, ShopNBC.TV — the excitement of live demonstration selling combined with a rich internet experience.