Phone Calls Are So 2002: Great Examples for A/B Testing

Optimization is all about providing the best possible customer experience. How do we reduce friction, get customers the information they need, and give them a great experience online? For this brand, a big part of meeting this goal is requiring customers to login in order to purchase. This is a major retail chain selling grocery, home goods, and electronics, and so having that level of personalization to encourage self-service online is critical for their business. For this A/B test, they focused not on the shopping experience, but on the login experience.

The Organization

Major retail chain selling grocery, home goods, and electronics online.

The Project

In order to buy goods from this online store, customers must login to their accounts. Of course, sometimes people forget their passwords. When this happens, the store provided several reset options:

1) Call Member Services
2) Online password reset

However, the phone number for Member Services was much more prominent in the window than the self-service password reset. As a result, their customer service team was fielding a lot of phone calls to reset customer passwords. This caused a lot of unnecessary friction for the customer, who has to spend extra time and make an extra effort, and also cost the company resources better spent on other customer service efforts. Too many customers who forgot their passwords ultimately abandoned their cart, and the team needed to optimize this process.

For this A/B test, the team sought a way to encourage customers who had forgotten their passwords to pursue the self-service reset option by A/B testing variations of the design.

The Tests

The optimization team began by A/B testing a variation that increased the prominence of the online password reset button, and decreased the prominence of the Member Services phone number. They ultimately ran three iterations of this, each time taking their learnings and refining the design.

It turned out that the most successful variation brought forward and emphasized the self-service password reset option, and moved the Member Services phone number further back in the self-support process.

Results and Next Steps

As a result of the three campaigns related to “Forgot Password,” the brand measured a 3.7% lift in successful password resets, a 5% lift in checkouts, and an in-visit gross merchandise volume lift of 4.4% which equates to a $10 million annual incremental revenue. Using SiteSpect’s QuickChange feature, the brand was able to release the winning variation to 100% of traffic, and then permanently apply the change as part of its next site release.

To learn more about SiteSpect, visit our website.