Experimentation as a Mindset: The Key to a Culture of Optimization

April 9, 2015

What would happen if you approached your beliefs as assumptions that could be tested? You could create hypotheses for which you could design experiments to find out where the real truth lies. 

What would happen if you approached your beliefs as assumptions that could be tested? You could create hypotheses for which you could design experiments to find out where the real truth lies.

To build a culture of optimization in your company, that’s exactly what you need to do.

Adopting experimentation as a mindset will enable you to test more ideas, brainstorm more alternatives, and maximize what’s possible

The best part is that experimentation is not limited to your marketing initiatives; it includes other parts of your organization and can therefore encompass the entire customer experience.

Some companies are way ahead of the game in creating and nurturing an organization-wide culture of optimization. The highly competitive online marketplace has driven companies to pioneer the experimentation mindset beyond the digital marketing, analytics, and testing teams to include product marketing as well as the user experience and development teams.

Although those teams may have very different goals, this new mindset allows them to unify behind one common goal of experimentation, which leads to a better customer experience and increased engagement, conversions—and revenue.

So, how might companies actually put this new mindset into action? First, they can apply experimentation and optimization to their entire Web and mobile presence. By carefully examining what they believe to be true and then testing alternatives, they can discover where the real truth lies.

Website Optimization Ideas

Consider the content, features, and functionality of your site to discover what works and what doesn’t. For example, you could experiment with the following:


  • A lot versus a little
  • Style and tone, such as chatty versus formal
  • Positioning: in other words, which value proposition, features, and benefits work best?
  • Call-to-action text

Font, color, and size

  • Think about your target audience (e.g., 10% of men are colorblind).
  • If your audience is older, test larger fonts.
  • Consider readability; for example, serif fonts are often easier to read than sans serif fonts.


  • Color
  • Size
  • Design
  • Location
  • Copy


  • Sequence of items
  • Labels (e.g., “My profile” vs. “Profile” vs. “Settings”)
  • Inline text links
  • Link style: underline, bold, color, etc.


  • Content of image (e.g., man vs. woman vs. group)
  • Size
  • Location on page
  • Design treatment (e.g., shadow background, outlined, etc.)


  • Optimal number of required fields versus optional fields
  • Design
  • Location
  • Size


  • Location and size of areas/boxes on the page
  • Attention focus: what layouts help focus vs. create distraction?
  • Where should you place buttons and links?

Features and Functionality

  • New feature releases. Soft-launch new features on a subset of users and learn how they respond, in real time. The additional customer insight can help you iterate on features more quickly and deliver enhancements that actually move the needle.
  • Feature segmentation. Present different features to different audiences—based on customer value, device, geography, new vs. existing customers, etc.—to optimize KPIs.
  • Search algorithm optimization. Make head-to-head comparisons of the effectiveness of different versions of your site-search algorithms or even multiple search vendor software solutions.
  • Pricing experiments. Test your “free shipping” threshold or offer different price promotions to distinct sets of targeted users; for example, test the impact of changing the threshold to obtain free shipping from $50 to $35.
  • Complete site redesigns. Test radical changes to your site experience targeted to specific users.
  • Checkout flow and functionality. Optimize your checkout flow through experimentation even on secure https pages. For example, you could test making your five-step checkout process only three steps for logged in users and see what a difference that makes.

Mobile Optimization

Customers are no longer tethered to their desktops, yet they expect an optimized browsing experience on their mobile device; so, you’ll want to test all aspects of your mobile site, including load times, search, and checkout, to deliver an optimal experience.

Experiment with different mobile experiences such as…

  • Traditional websites
  • Responsive design sites
  • M.dot, t.dot, and other dedicated mobile sites
  • Angular, single-page applications, and hybrid apps
  • Native applications

Test how different types of mobile content influence user behavior, and ultimately, site profitability. Test, measure, and deliver the content, layout, and promotional offers that are most effective for each mobile device category.

Also, consider targeting your tested content using a combination of static and dynamic criteria, including the following, for example:

  • Device type, including tablets and smartphones
  • Mobile operating system
  • Screen dimensions and rotation support
  • Browser capabilities
  • Cellular network data speeds
  • Preferred markup language
  • Keyboard type

Experimentation does not end with testing; it also extends to segmentation and targeting strategies to enable you to target your tests. Target visitors based on criteria including the following:

  • User behavior, such as adding an item to a cart, bookmarking a favorite, or viewing a product category page
  • Geo-targeting
  • History, such as recency, frequency, and monetary value of previous purchases and transactions
  • Mobile device and platform
  • Browser
  • Referral source of visit, such as Google or PPC
  • Day-parting and time-parting
  • Language
  • Externally defined criteria from third-party CRM or SFA databases

Even as the CMO of an optimization solutions provider, I am often presented with the opportunity to test something I hadn’t thought of before. I was recently wrangling with a couple of new taglines when my CEO (the founder of SiteSpect, Eric J. Hansen) said, “That would make a great A/B test.” And he’s right: Rather than an executive deciding on what the best tagline is, let’s test it and have our audience use their clicks to tell us what they like.

That is experimentation as a mindset in action!

This article originally appeared in the April 2015 issue of MarketingProfs. You can read the original version here.