Control your environment without installing any third-party code.As an add-on package, SiteSpect Enhanced Analytics combines two powerful tools to fortify and finely tune campaign data:
  1. Outlier Data: To identify and mitigate anomalous data that can skew data-driven decision making.
  2. Real User Monitoring (RUM): Capture & analyze actual user interactions and layer performance metrics into any campaign report.
For more information visit sitespect.com/analytics.
SiteSpect is CCPA Compliant. The California Consumer Privacy Act (CCPA) is a state statute intended to enhance privacy rights and consumer protection for US residents of California. More information about SiteSpect’s CCPA support is located on our Customer Notice On EU GDPR and CCPA Support page.SiteSpect will dedicate resources to the equivalent of a full-time employee to work with your team at your offices.SiteSpect will dedicate resources to the equivalent of a half-time employee to work exclusively with your team across any of the professional services.SiteSpect will dedicate resources to the equivalent of a full-time employee to work exclusively with your team across any of the professional services.SiteSpect provides on-site training sessions custom designed especially for your business and IT teams to help you achieve your business goals.SiteSpect has a curriculum of training modules and best practices that you can take advantage of to increase your team's adoption and mastery of the user interface.We offer training to get you up and running quickly, including test building courses & analytics interpretation.Take advantage of SiteSpect's premium help and knowledge portal where you will find FAQs, how-tos, and detailed step-by-step instructions for successful use of the SiteSpect platform.We offer monthly webinars on SiteSpect featured functionalities and best practices for increased value and success.Identify additional testing and personalization opportunities from the results of past campaigns.Our professional services team will work with you to create visualizations to demonstrate how SiteSpect affects your revenue.Optimization Consultants will provide full analytical reports on campaign performance, highlighting KPI impact, segment analysis, and key insights.Optimization Consultants will show you the best way to get the most out of your traffic across a large number of tests and personalization campaigns.Learn how to track and monitor running tests and personalization campaigns.Learn best practices for efficiently creating and executing tests and personalization campaigns.Design, develop, and optimize all of the components for your program.Learn how to structure experiments to maximize learning and efficiency. A/B vs. MVT, Controlled Experiments vs. Personalization.Add a workflow tool to integrate your prioritization and team workflow directly into SiteSpect.Your Optimization Consultant provides best practices and process recommendations for running a high functioning optimization program.SiteSpect will provide guidance and support for making decisions regarding scheduling tests in order to maximize efficiency and generate as much learning as possible as efficiently as possible.Working with key stakeholders in your organization, SiteSpect will provide test ideas that improve the customer experience and move the needle on your key performance metrics.Your Optimization Consultant will provide best practices for creating data-driven hypotheses that answer key business questions.Based on our experience and your analytics, your Optimization Consultant will help you identify optimization opportunities and establish a workflow for collecting and managing input from across your organization.Provided by your SiteSpect Optimization Consultant who understands your business objectives and works with you to achieve your testing goals and increase ROI.Number of hours SiteSpect professional services representatives will work on strategy, creation, or analysis for your optimization program.Enforce an optimization workflow with campaign administrators, builders and read-only roles.Custom integration with 3rd party tools such as Adobe Analytics, Google Analytics, CrazyEgg, Hotjar and more.Build and optimize the customer experience on iOS and Android apps. Build experiences visually, directly on your mobile device, leverage our SPAs or use API only solution.Leverage Custom Variables to capture dimensions about users, such as products purchased, categories viewed and items searched.Isolate users in single Campaigns or allow them in multiple overlapping experiences.Create a unified customer experience by connecting experiences for recognized users across devices.Automate your optimization program through our API: http://developers.sitespect.com/Test and deploy server-side functionality with on/off switches or progressive rollouts. Leverage cookies, headers or parameters for feature flagging. Test and optimize your CMS, recommendation engine, and other vendor tools.Leverage our SPA SDK to test and optimize your Single Page Application. Support for major frameworks such as Angular, React, and others.Modify the source code of pages on the fly with Regular Expressions, with zero impact to performance. Replace, remove, add in new functionality before pages are rendered in the browser.SiteSpect offers a number of capabilities for effective front-end testing. Use Visual Editor or Regular Expressions to modify source code, support SPA frameworks, define pages for reusability in tests, and minimize performance hits for users and servers.Use point-and-click actions to modify text, images, buttons, promotional banners, and styling. Create, move, rearrange, hide, and track behaviors across channels. Works with SPAs out of the box, such as React and Angular.Push winning Variations to all traffic to benefit early from wins. Push hot fixes your website to quickly resolve issues, to all traffic or segments of users.Respond to real-time alerts and in-product messages to proactively manage your campaigns. Known when a campaign has a winning variation, is hurting conversion, does not have any recent visits, or is disabled.Quickly analyze and understand relevant campaigns and outcomes. Measure conversion, KPIs, user actions, events, conversion funnels, and other key user interactions in our comprehensive dashboard.Personalize the experience for mobile devices, geo location, user types, and user behaviors. Build your own Audiences to align with your key segments of users and leverage your Data Layer.Experience our proxy architecture built from the ground up to avoid flicker and ensure fast experience delivery across desktop, tablet, and mobile devices.Training to get you up and running quickly, includes test building courses & analytics interpretation.Dedicated optimization consultant for strategic and tactical guidance. Your GoLive Report is a success plan to get your team up and running, including website audit, test ideas and optimization roadmap. You will also get ROI (return on investment) modeling backed by Forrester.Depending on which package there are countless tests, segmentations and optimizations within your control across almost any digital touchpoint.Run as many Campaigns as you want.Our optimization experts work with you to recommend 10 impactful test ideas to improve customer experience and impact your bottom line, and build 10 Campaigns so you can quickly benefit from our platform.Directly ingest SiteSpect traffic logs into your monitoring infrastructure to diagnose underperforming applications/servers and identify site errors in near real time.SiteSpect sends a sample of log data to Logz.io, a log visualization service. This service offers full flexibility to view and analyze logs relating to traffic health and routing. Your network operations center can also use this service to build your own visualization and dashboards or integrate into existing systems.SiteSpect is GDPR Compliant. The General Data Protection Regulation (GDPR) is a legal framework that sets guidelines for the collecting and processing personal information of individuals within or who are citizens of the European Union. More information about SiteSpect’s GDPR support is located on our Customer Notice On EU GDPR Support page.SiteSpect is HIPAA ready, which saves you time when you use SiteSpect in your HIPAA-compliant environment. SiteSpect obfuscates and encrypts visitor behavior and does not store any personal data. SiteSpect also offers on-premise and other deployment models to guarantee personal data never leaves your datacenter.SiteSpect supports Single Sign-On (SSO) authentication through Active Directory and SAML. SSO allows you to centrally manage SiteSpect permissions and login credentials. It also makes it easier for SiteSpect users to quickly log in and switch between their applications saving time and improving productivity.Restrict user access to SiteSpect by IP address.Restrict access to your websites by IP address. Create lists or ranges of IP addresses that allow trusted (whitelisted) users to access your websites or prevent access from disallowed (blacklisted) users.Two-Factor Authentication, also known as 2FA, requires not only a password and username but also something that only a user has access to, such as a token. Using a username and password together with a token makes it harder for potential intruders to gain access to SiteSpect.SiteSpect complies with both the EU-US and the Swiss-US Privacy Shield Frameworks. SiteSpect certifies that it adheres to the Privacy Shield Principles with respect to personal data. More information about SiteSpect’s Privacy Shield support is available on our Business Services Privacy Policy - GDPR and Privacy Shield page.SiteSpect includes a complete history of every user action. Quickly undo changes in the audit history list and audit any change as part of your change management process.SiteSpect maintains the privacy of all customer data by ensuring no test data is exposed and that data is not grouped with or shared with others in any way.SiteSpect leverages Alert Logic to monitor and track security vulnerabilities. This provides you with an additional line of defense as we will notify you of any detected threats or attacks on your website.SiteSpect supports nine different permission levels that provide control and configurable access. Set view and edit permissions on a per user basis for building, managing, and analyzing campaigns.SiteSpect fully supports encryption for all types of in-transit data such as end-user content, SDK and API transmissions, and business user data. We also support testing on HTTP and HTTPS websites.

What Do You Do With Flat A/B Test Results? How To Iterate for Optimization

March 27, 2019

Share

We’ve talked about how to choose your first A/B test, but what about your second, third, fourth, or fifth A/B test? Where do you go from there? In this blog I’ll walk through how to iterate in a way that’s productive, instructive, and gets you the most value out of your A/B testing program.

Optimization Pro interpreting data

Interpreting Your Analytics

If you’re new to A/B testing, it can feel like a let down if your first A/B test doesn’t show a huge impact to your KPI. But, ConversionXL nicely lays out the numbers in this article, showing that anywhere from 50% – 80% of A/B test results are inconclusive [read: still valid]. Other CRO experts estimate something like 1 out of 7 A/B tests has a clear winner. Despite this information, it can be easy to focus on big wins or big risk mitigations with statistical significance. But it’s crucial to remember that flat results are still valid and relevant.

Once you get into the rhythm of A/B testing, these flat results can actually build the groundwork for your more impactful A/B tests. The learnings you get will allow you to narrow in on the parts of your site that affect user behavior the most. But in order to achieve this, you need to have planned out an effective campaign structure with a strong hypothesis, and considered a wide variety of metrics and user segments. Let’s look at an example:

Example A/B Test:

 

A/B Test: Reduce your checkout flow from three pages to one page

Metrics: Add to cart, purchase, time on page

Segments: Browser, device type, location, new users, return users

In the above example, the big hope is that a shorter check out flow will reduce friction and increase purchases. But, when the A/B test reaches maturity you find that your KPI numbers haven’t moved significantly. This looks at first like the checkout flow made no difference, but now let’s look at the other metrics and segments:

Across All Segments:

Purchase: flat

Add to Cart: flat.

Time on Page (first page of checkout flow): increased

By Device Type:

Purchase: flat across Chrome, Firefox, and Safari; decreased on desktop, increased on tablet and mobile

Add to Cart: flat across devices and browsers

Time on page: flat across browsers, increased on desktop, decreased on mobile and tablet

By User Type:

Purchase: flat for return users but increased for new users

Add to Cart: flat

Time on page: flat for return users but decreased for new users

Now, while our A/B test proved flat in terms of our overall metrics, we actually learned some really valuable information about specific segments of users, and the “flat” result was deceptive. From the data, we can see that the variation actually improved the experience for mobile users, tablet users, new users, and made their checkout flow more successful. However, desktop and return users kept getting stuck and abandoning that new checkout page. In this example A/B test, it’s important to note that each subset of data had a large enough sample size to reach significance.

How to Iterate

Now you can iterate on your original hypothesis. Common wisdom suggests that shorter checkout flows are more effective, but now you know that’s not true in this instance. Since users are getting stuck here, what if you try breaking the checkout flow into two pages (rather than the original three), redesigning the look of the checkout, or even lengthening the checkout flow to take more pages. Alternatively, could you get away with requiring fewer steps for checkout? Maybe the key isn’t the number of pages but the amount of information required? Are there optional fields you could remove? You may have some fields that cause more friction than others. Metrics tracking which fields throw validation errors most in the first version of your campaign would give you further areas to explore in the next iteration.

Now, just because you lifted conversions on mobile doesn’t mean you’re done. Fewer pages in checkout succeeded here, but the findings would suggest you should consider taking a different approach for desktop. Now you’re ready to dive into your next A/B test.

Communicating Your “Flat” Learnings

Even in companies with the most mature and robust A/B testing programs, optimization specialists struggle to communicate the value of flat learnings across departments. When you hear the word “optimize” you expect a string of big wins. To wrap up this blog, I’ll give some tips for how to communicate the value of these learnings outside of the optimization team.

One of the most valuable things you can do for your team is to make sure you do not put all of your eggs into one basket with a primary variant and a primary KPI. These are experiments and A/B tests, and you never know for sure what metrics will see the biggest impact or which variations will move the needle. If you knew that, you wouldn’t need to A/B test. So, when your report looks flat there are some important pieces to highlight in your communications.

First off, a flat result is still a valid result. You may not have proved your hypothesis but the business has the choice to revert to control or choose to rollout the variant with the expectation it is not going to shift the conversion needle but is a conversion neutral option.

When you present your results, make sure your analytics show the full picture — don’t present just one metric, or one segment. While you don’t want to overcomplicate your findings, you also don’t want to reduce the value of your experiment. If the total conversion metrics didn’t move significantly what about for specific segments of users? What did some of the secondary metrics show? Bounce rate, time on page, or errors are likely to point to how to evolve your hypothesis and iterate on your campaign giving you actionable next steps to reach your goal.

To support your optimization efforts, you might also consider each A/B test as a step in a broader goal. For example, if you set out to optimize your checkout flow, this will entail a series of A/B tests, variations, and analyses. You’ll end up with a huge impact and impressive ROI, but it can take time to really understand how users behave on your site and how to improve the experience to reach that desired result.

To learn more about SiteSpect, visit our website.

Share

Luke Hardwick is a Manager of Customer Success at SiteSpect, consulting for SiteSpect users on their optimization and personalization road maps and projects. Luke is based in London and has experience as an conversion rate optimization specialist across many softwares before landing at SiteSpect.

Email Luke Hardwick

Suggested Posts

online shopping.jpg

How to Test for the Conversion Stage of the Customer Journey

Designing Websites for Multivariate Testing:  Four Key Considerations to Ensure Your Success

Designing Websites for Multivariate Testing: Four Key Considerations to Ensure Your Success

path leads to decision which changes the path in two directions on colorful background

Part 4/6: 5 Questions You Need to Answer to Optimize an Experience

Leave a Comment

Your email address will not be published. Required fields are marked *