SiteSpect will dedicate resources to the equivalent of a full-time employee to work with your team at your offices.SiteSpect will dedicate resources to the equivalent of a half-time employee to work exclusively with your team across any of the professional services.SiteSpect will dedicate resources to the equivalent of a full-time employee to work exclusively with your team across any of the professional services.SiteSpect provides on-site training sessions custom designed especially for your business and IT teams to help you achieve your business goals.SiteSpect has a curriculum of training modules and best practices that you can take advantage of to increase your team's adoption and mastery of the user interface.We offer training to get you up and running quickly, including test building courses & analytics interpretation.Take advantage of SiteSpect's premium help and knowledge portal where you will find FAQs, how-tos, and detailed step-by-step instructions for successful use of the SiteSpect platform.We offer monthly webinars on SiteSpect featured functionalities and best practices for increased value and success.Identify additional testing and personalization opportunities from the results of past campaigns.Our professional services team will work with you to create visualizations to demonstrate how SiteSpect affects your revenue.Optimization Consultants will provide full analytical reports on campaign performance, highlighting KPI impact, segment analysis, and key insights.Optimization Consultants will show you the best way to get the most out of your traffic across a large number of tests and personalization campaigns.Learn how to track and monitor running tests and personalization campaigns.Learn best practices for efficiently creating and executing tests and personalization campaigns.Design, develop, and optimize all of the components for your program.Learn how to structure experiments to maximize learning and efficiency. A/B vs. MVT, Controlled Experiments vs. Personalization.Add a workflow tool to integrate your prioritization and team workflow directly into SiteSpect.Your Optimization Consultant provides best practices and process recommendations for running a high functioning optimization program.SiteSpect will provide guidance and support for making decisions regarding scheduling tests in order to maximize efficiency and generate as much learning as possible as efficiently as possible.Working with key stakeholders in your organization, SiteSpect will provide test ideas that improve the customer experience and move the needle on your key performance metrics.Your Optimization Consultant will provide best practices for creating data-driven hypotheses that answer key business questions.Based on our experience and your analytics, your Optimization Consultant will help you identify optimization opportunities and establish a workflow for collecting and managing input from across your organization.Provided by your SiteSpect Optimization Consultant who understands your business objectives and works with you to achieve your testing goals and increase ROI.Number of hours SiteSpect professional services representatives will work on strategy, creation, or analysis for your optimization program.Enforce an optimization workflow with campaign administrators, builders and read-only roles.Custom integration with 3rd party tools such as Adobe Analytics, Google Analytics, CrazyEgg, Hotjar and more.Build and optimize the customer experience on iOS and Android apps. Build experiences visually, directly on your mobile device, leverage our SPAs or use API only solution.Leverage Custom Variables to capture dimensions about users, such as products purchased, categories viewed and items searched.Isolate users in single Campaigns or allow them in multiple overlapping experiences.Create a unified customer experience by connecting experiences for recognized users across devices.Automate your optimization program through our API: http://developers.sitespect.com/Test and deploy server-side functionality with on/off switches or progressive rollouts. Leverage cookies, headers or parameters for feature flagging. Test and optimize your CMS, recommendation engine, and other vendor tools.Leverage our SPA SDK to test and optimize your Single Page Application. Support for major frameworks such as Angular, React, and others.Modify the source code of pages on the fly with Regular Expressions, with zero impact to performance. Replace, remove, add in new functionality before pages are rendered in the browser.SiteSpect offers a number of capabilities for effective front-end testing. Use Visual Editor or Regular Expressions to modify source code, support SPA frameworks, define pages for reusability in tests, and minimize performance hits for users and servers.Use point-and-click actions to modify text, images, buttons, promotional banners, and styling. Create, move, rearrange, hide, and track behaviors across channels. Works with SPAs out of the box, such as React and Angular.Push winning Variations to all traffic to benefit early from wins. Push hot fixes your website to quickly resolve issues, to all traffic or segments of users.Respond to real-time alerts and in-product messages to proactively manage your campaigns. Known when a campaign has a winning variation, is hurting conversion, does not have any recent visits, or is disabled.Quickly analyze and understand relevant campaigns and outcomes. Measure conversion, KPIs, user actions, events, conversion funnels, and other key user interactions in our comprehensive dashboard.Personalize the experience for mobile devices, geo location, user types, and user behaviors. Build your own Audiences to align with your key segments of users and leverage your Data Layer.Experience our proxy architecture built from the ground up to avoid flicker and ensure fast experience delivery across desktop, tablet, and mobile devices.Training to get you up and running quickly, includes test building courses & analytics interpretation.Dedicated optimization consultant for strategic and tactical guidance. Your GoLive Report is a success plan to get your team up and running, including website audit, test ideas and optimization roadmap. You will also get ROI (return on investment) modeling backed by Forrester.Depending on which package there are countless tests, segmentations and optimizations within your control across almost any digital touchpoint.Run as many Campaigns as you want.Our optimization experts work with you to recommend 10 impactful test ideas to improve customer experience and impact your bottom line, and build 10 Campaigns so you can quickly benefit from our platform.Directly ingest SiteSpect traffic logs into your monitoring infrastructure to diagnose underperforming applications/servers and identify site errors in near real time.SiteSpect sends a sample of log data to Logz.io, a log visualization service. This service offers full flexibility to view and analyze logs relating to traffic health and routing. Your network operations center can also use this service to build your own visualization and dashboards or integrate into existing systems.SiteSpect is GDPR Compliant. The General Data Protection Regulation (GDPR) is a legal framework that sets guidelines for the collecting and processing personal information of individuals within or who are citizens of the European Union. More information about SiteSpect’s GDPR support is located on our Customer Notice On EU GDPR Support page.SiteSpect is HIPAA ready, which saves you time when you use SiteSpect in your HIPAA-compliant environment. SiteSpect obfuscates and encrypts visitor behavior and does not store any personal data. SiteSpect also offers on-premise and other deployment models to guarantee personal data never leaves your datacenter.SiteSpect supports Single Sign-On (SSO) authentication through Active Directory and SAML. SSO allows you to centrally manage SiteSpect permissions and login credentials. It also makes it easier for SiteSpect users to quickly log in and switch between their applications saving time and improving productivity.Restrict user access to SiteSpect by IP address.Restrict access to your websites by IP address. Create lists or ranges of IP addresses that allow trusted (whitelisted) users to access your websites or prevent access from disallowed (blacklisted) users.Two-Factor Authentication, also known as 2FA, requires not only a password and username but also something that only a user has access to, such as a token. Using a username and password together with a token makes it harder for potential intruders to gain access to SiteSpect.SiteSpect complies with both the EU-US and the Swiss-US Privacy Shield Frameworks. SiteSpect certifies that it adheres to the Privacy Shield Principles with respect to personal data. More information about SiteSpect’s Privacy Shield support is available on our Business Services Privacy Policy - GDPR and Privacy Shield page.SiteSpect includes a complete history of every user action. Quickly undo changes in the audit history list and audit any change as part of your change management process.SiteSpect maintains the privacy of all customer data by ensuring no test data is exposed and that data is not grouped with or shared with others in any way.SiteSpect leverages Alert Logic to monitor and track security vulnerabilities. This provides you with an additional line of defense as we will notify you of any detected threats or attacks on your website.SiteSpect supports nine different permission levels that provide control and configurable access. Set view and edit permissions on a per user basis for building, managing, and analyzing campaigns.SiteSpect fully supports encryption for all types of in-transit data such as end-user content, SDK and API transmissions, and business user data. We also support testing on HTTP and HTTPS websites.

Why Your Experimentation Program May Need a Statistics Refresher

November 5, 2019

Share

The digital world needs more experimentation. But it also needs better experimentation. Testing without strong measurement discipline is a lot like the food pictures people post on Instagram: impressions of a reality that doesn’t exist. This happens a lot, often within experienced teams, for several reasons.

At the start, it helps to acknowledge that we, the experimenters, are not unbiased observers of data. Yes, we love data. Yes, we glory in data-driven decision making. But we also love to win, to find the gold nuggets that will make our companies successful, our egos pleasantly plump, and our resumes impressively decorated with numbers. But this preference, this bias, however slight, compromises our judgment and interpretation of data from the get-go.

Compound this phenomenon with the limitless options and methods at our disposal to misrepresent and abuse statistics and what you emerge with is something that resembles data-driven decision making, but in reality is little more than a tangle of assertions that are extremely difficult to dispute without full visibility into the data. In short, we can make powerful arguments that suit our agenda, but peel back enough layers of the onion and you may have an empty core.

I think it’s probably secondary reasoning, but our intuitions around probability and significance can sit far off the mark. Those small biases then lead the way to data-influenced but intuitive decision making. Thus, statistics.

An Example of Statistics vs. Intuition

I love to reflect on the so-called “Monty Hall” problem made famous by the old game show Let’s Make a Deal.

Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Is it to your advantage to switch your choice?

Most of us will reason that there’s no advantage to switching. That it’s a 50/50 chance of winning either way. But here we’d be dead wrong. Switching would give you a 66% chance of winning while staying would give you only 33% chance. A quick illustration will help you “see” the fallacy pretty quickly.

You start with 3 possible scenarios:

Now, in the first step, you choose Door 1.

In step 2, the host reveals a goat behind one of the other doors.

Now, in how many of the 3 Scenarios would switching get you a car?

Seen at this level, it’s a simple problem with a clear solution. But our intuitions, even when scrutinized, have a really hard time picking up on that initially. Likewise, when we look at data comparing 2 groups and observe a difference between them, it’s hard not to immediately judge one as being superior to the other even while scenarios exist in which what you’re observing is completely random or your data set does not reflect the true difference in the groups. Thus, statistics.

The Place for Experimentation

So what of our intuitions? Should we simply ignore them? Should we feel so uncertain about our judgment that we do nothing without testing? Should we discount years of experience doing UX, design, user research, analytics, ecommerce, or other digital marketing functions? What is experience worth anyway?

I’m of 2 minds here. On the one hand, our intuitions and judgments should become better over time as we run our own experiments and witness the results of many others’ experiments. This should lead to better, more impactful ideas making their way into our pipeline of hypotheses.

On the other hand, we operate in an environment and era of rapid change. The competition changes. Our customers needs and expectations change. Design sensibilities change. Technology changes. Economies change. And all of these changes threaten to undermine the ongoing advantage of any given winning treatment and our refined judgment along with it. We see this in action in the phenomenon known as Regression to Mean where performance gains see diminishing returns over time. Such unexpected and unwelcome trends have been perplexing A/B testers for years. It seems that even after running a controlled experiment, results are not guaranteed and we should anticipate the benefit to lead a short but joyous life. The uncertainty produced is akin to that described brilliantly in Dr. Seuss’ whimsical yet insightful message to youth:

“You will come to a place where the streets are not marked. Some windows are lighted. But mostly they’re darked. A place you could sprain both your elbow and chin! Do you dare to stay out? Do you dare to go in? How much can you lose? How much can you win?
And if you go in, should you turn left or right…or right-and-three-quarters? Or, maybe, not quite? Or go around back and sneak in from behind? Simple it’s not, I’m afraid you will find, for a mind-maker-upper to make up his mind.”
–Dr. Seuss Oh the Places You’ll Go!

Until such a time as design innovation ceases, competitors stop striving, and consumers become reliably predictable, controlled experimentation remains the best way to go about de-risking experience changes for consumers at scale.

But even though experimentation is often a highly affordable option for reducing uncertainty, most organizations have many more decisions to make than they can fund tests for. Indeed, many decisions are more trivial than a test can justify. There are also more ways to reduce uncertainty. Experience, your own and that of others, does count for something. Data analysis counts. Usability testing counts. Surveys count. Customer support history counts. Records from sales interactions count. Heuristics and “best practices” count. Shoot, even your mom’s opinion counts (a lot more than one might think!). All of these sources provide the kind of evidence that we should demand for even the most trivial of risks. We should also scrutinize all sources of evidence and acknowledge, even, expect them to produce errors at some cadence.

When the chances for being wrong multiplied by the cost of being wrong is greater than the tolerance for loss, then it’s time to either abandon ship or gather more evidence. And while A/B testing is not the panacea of de-risking business decisions, it might be the closest we can get.

Merritt Aho is an Optimization Director at Search Discover. Learn more about him and his work here.

Share

Suggested Posts

package at door_0.jpg

A/B Testing When to Advertise Free Shipping

Tis the season shrug.png

Seasonal Conversion Rate Optimization: Don’t Make These Five Mistakes

han shot first.png

Why it Matters That Han Shot First: From Star Wars to Marketing

Leave a Comment

Your email address will not be published. Required fields are marked *