On-Site Search: A/B Test the Functional

Are you trying to figure out how important it is to A/B test on-site search? If so, consider this example from Walmart. The team conducted a series of functional A/B tests to optimize how their search engine works. The goal of the project was to “translate your searches into the stuff you really want to buy.” During development, they continuously A/B tested and refined their search engine. Within a year, they found that “20 percent more searches turned into sales than before.”

As this example shows, it’s important to focus on what the search engine returns. When you A/B test the visual aspects of search, you A/B test the look and feel. If you’re ready to focus on the functional aspects of your site, here are some ideas to get you started:

1. Relevance: Find out if your search delivers results that are relevant to your visitors. If the results don’t matter, you may frustrate potential customers. Examine your search’s inclusion and exclusion criteria. Keep in mind that meaningful results improve conversion.
2. Default Sort Order: Look at the default sort order for search results. Keep in mind that default sort is not one-size-fits-all. For example, consider that a travel site offers hotel and rental car bookings. When a visitor searches for rental cars, the results are sorted by price. You may want to A/B test whether price is the best sort order for hotels or if another option, such as distance, works better. Try using filter options to identify alternate default sort orders.
3. Customer Ratings: Optimize customer ratings and how they are used in search results. Find the ideal number of ratings and the right balance of positive and negative reviews. You may want to A/B test whether to include customer ratings in search results.
4. Error Handling: Be helpful when search errors, such as a misspelled word or misplaced hyphen, occur on your site. To show how a well-known site handles errors, I entered ‘tabelt’ into Amazon’s search tool. The site returned results for ‘tablet’ and also included a link to view any results for my original search term. You should A/B test how your site search handles errors to learn whether your error handling helps or hurts conversions.
5. Features: Experiment with the unused features of your search tool. You may not have looked at these features since you started using the tool. Enable new features and measure the results. You can also A/B test turning off certain features to gauge the effect.

At SiteSpect we’ve seen several of our customers benefit from focusing on these aspects of their site. Often times, these A/B tests involve a cross-functional team that includes marketers, product managers, and IT teams.

So far, this series includes ten A/B test ideas, both visual and functional, to optimize on-site search. Our next post will include two more advanced A/B test ideas and advice on how to bring it all together.

To learn more about SiteSpect, visit our website.