As an add-on package, SiteSpect Enhanced Analytics combines two powerful tools to fortify and finely tune campaign data:
  1. Outlier Data: To identify and mitigate anomalous data that can skew data-driven decision making.
  2. Real User Monitoring (RUM): Capture & analyze actual user interactions and layer performance metrics into any campaign report.
For more information visit
SiteSpect is CCPA Compliant. The California Consumer Privacy Act (CCPA) is a state statute intended to enhance privacy rights and consumer protection for US residents of California. More information about SiteSpect’s CCPA support is located on our Customer Notice On EU GDPR and CCPA Support page.SiteSpect will dedicate resources to the equivalent of a full-time employee to work with your team at your offices.SiteSpect will dedicate resources to the equivalent of a half-time employee to work exclusively with your team across any of the professional services.SiteSpect will dedicate resources to the equivalent of a full-time employee to work exclusively with your team across any of the professional services.SiteSpect provides on-site training sessions custom designed especially for your business and IT teams to help you achieve your business goals.SiteSpect has a curriculum of training modules and best practices that you can take advantage of to increase your team's adoption and mastery of the user interface.We offer training to get you up and running quickly, including test building courses & analytics interpretation.Take advantage of SiteSpect's premium help and knowledge portal where you will find FAQs, how-tos, and detailed step-by-step instructions for successful use of the SiteSpect platform.We offer monthly webinars on SiteSpect featured functionalities and best practices for increased value and success.Identify additional testing and personalization opportunities from the results of past campaigns.Our professional services team will work with you to create visualizations to demonstrate how SiteSpect affects your revenue.Optimization Consultants will provide full analytical reports on campaign performance, highlighting KPI impact, segment analysis, and key insights.Optimization Consultants will show you the best way to get the most out of your traffic across a large number of tests and personalization campaigns.Learn how to track and monitor running tests and personalization campaigns.Learn best practices for efficiently creating and executing tests and personalization campaigns.Design, develop, and optimize all of the components for your program.Learn how to structure experiments to maximize learning and efficiency. A/B vs. MVT, Controlled Experiments vs. Personalization.Add a workflow tool to integrate your prioritization and team workflow directly into SiteSpect.Your Optimization Consultant provides best practices and process recommendations for running a high functioning optimization program.SiteSpect will provide guidance and support for making decisions regarding scheduling tests in order to maximize efficiency and generate as much learning as possible as efficiently as possible.Working with key stakeholders in your organization, SiteSpect will provide test ideas that improve the customer experience and move the needle on your key performance metrics.Your Optimization Consultant will provide best practices for creating data-driven hypotheses that answer key business questions.Based on our experience and your analytics, your Optimization Consultant will help you identify optimization opportunities and establish a workflow for collecting and managing input from across your organization.Provided by your SiteSpect Optimization Consultant who understands your business objectives and works with you to achieve your testing goals and increase ROI.Number of hours SiteSpect professional services representatives will work on strategy, creation, or analysis for your optimization program.Enforce an optimization workflow with campaign administrators, builders and read-only roles.Custom integration with 3rd party tools such as Adobe Analytics, Google Analytics, CrazyEgg, Hotjar and more.Build and optimize the customer experience on iOS and Android apps. Build experiences visually, directly on your mobile device, leverage our SPAs or use API only solution.Leverage Custom Variables to capture dimensions about users, such as products purchased, categories viewed and items searched.Isolate users in single Campaigns or allow them in multiple overlapping experiences.Create a unified customer experience by connecting experiences for recognized users across devices.Automate your optimization program through our API: and deploy server-side functionality with on/off switches or progressive rollouts. Leverage cookies, headers or parameters for feature flagging. Test and optimize your CMS, recommendation engine, and other vendor tools.Leverage our SPA SDK to test and optimize your Single Page Application. Support for major frameworks such as Angular, React, and others.Modify the source code of pages on the fly with Regular Expressions, with zero impact to performance. Replace, remove, add in new functionality before pages are rendered in the browser.SiteSpect offers a number of capabilities for effective front-end testing. Use Visual Editor or Regular Expressions to modify source code, support SPA frameworks, define pages for reusability in tests, and minimize performance hits for users and servers.Use point-and-click actions to modify text, images, buttons, promotional banners, and styling. Create, move, rearrange, hide, and track behaviors across channels. Works with SPAs out of the box, such as React and Angular.Push winning Variations to all traffic to benefit early from wins. Push hot fixes your website to quickly resolve issues, to all traffic or segments of users.Respond to real-time alerts and in-product messages to proactively manage your campaigns. Known when a campaign has a winning variation, is hurting conversion, does not have any recent visits, or is disabled.Quickly analyze and understand relevant campaigns and outcomes. Measure conversion, KPIs, user actions, events, conversion funnels, and other key user interactions in our comprehensive dashboard.Personalize the experience for mobile devices, geo location, user types, and user behaviors. Build your own Audiences to align with your key segments of users and leverage your Data Layer.Experience our proxy architecture built from the ground up to avoid flicker and ensure fast experience delivery across desktop, tablet, and mobile devices.Training to get you up and running quickly, includes test building courses & analytics interpretation.Dedicated optimization consultant for strategic and tactical guidance. Your GoLive Report is a success plan to get your team up and running, including website audit, test ideas and optimization roadmap. You will also get ROI (return on investment) modeling backed by Forrester.Depending on which package there are countless tests, segmentations and optimizations within your control across almost any digital touchpoint.Run as many Campaigns as you want.Our optimization experts work with you to recommend 10 impactful test ideas to improve customer experience and impact your bottom line, and build 10 Campaigns so you can quickly benefit from our platform.Directly ingest SiteSpect traffic logs into your monitoring infrastructure to diagnose underperforming applications/servers and identify site errors in near real time.SiteSpect sends a sample of log data to, a log visualization service. This service offers full flexibility to view and analyze logs relating to traffic health and routing. Your network operations center can also use this service to build your own visualization and dashboards or integrate into existing systems.SiteSpect is GDPR Compliant. The General Data Protection Regulation (GDPR) is a legal framework that sets guidelines for the collecting and processing personal information of individuals within or who are citizens of the European Union. More information about SiteSpect’s GDPR support is located on our Customer Notice On EU GDPR Support page.SiteSpect is HIPAA ready, which saves you time when you use SiteSpect in your HIPAA-compliant environment. SiteSpect obfuscates and encrypts visitor behavior and does not store any personal data. SiteSpect also offers on-premise and other deployment models to guarantee personal data never leaves your datacenter.SiteSpect supports Single Sign-On (SSO) authentication through Active Directory and SAML. SSO allows you to centrally manage SiteSpect permissions and login credentials. It also makes it easier for SiteSpect users to quickly log in and switch between their applications saving time and improving productivity.Restrict user access to SiteSpect by IP address.Restrict access to your websites by IP address. Create lists or ranges of IP addresses that allow trusted (whitelisted) users to access your websites or prevent access from disallowed (blacklisted) users.Two-Factor Authentication, also known as 2FA, requires not only a password and username but also something that only a user has access to, such as a token. Using a username and password together with a token makes it harder for potential intruders to gain access to SiteSpect.SiteSpect complies with both the EU-US and the Swiss-US Privacy Shield Frameworks. SiteSpect certifies that it adheres to the Privacy Shield Principles with respect to personal data. More information about SiteSpect’s Privacy Shield support is available on our Business Services Privacy Policy - GDPR and Privacy Shield page.SiteSpect includes a complete history of every user action. Quickly undo changes in the audit history list and audit any change as part of your change management process.SiteSpect maintains the privacy of all customer data by ensuring no test data is exposed and that data is not grouped with or shared with others in any way.SiteSpect leverages Alert Logic to monitor and track security vulnerabilities. This provides you with an additional line of defense as we will notify you of any detected threats or attacks on your website.SiteSpect supports nine different permission levels that provide control and configurable access. Set view and edit permissions on a per user basis for building, managing, and analyzing campaigns.SiteSpect fully supports encryption for all types of in-transit data such as end-user content, SDK and API transmissions, and business user data. We also support testing on HTTP and HTTPS websites.

Building a Website Testing and Optimization Program: Implementation (Part 2)

January 23, 2013


A common disappointment among companies deploying testing and optimization technology stems from tests that fail to produce the type of gains expected. Seemingly without rhyme or reason, even the most dramatic design changes yield “no significant differences” based on simple measures such as click-through and even less for more involved down-stream metrics such as conversion rate. While this is the reality of testing, I believe that much of the disappointment stems from a lack of attention to the definition of “success” and“failure” as the design or changes are implemented.

In this post, we’ll look at four key steps in implementing your optimization program:

  1. Clearly Define Success and Failure
  2. Ensure Good Test Design
  3. Clarify Your Testing Timeline
  4. Test Different Audience Segments

Clearly Define Success and Failure

A common disappointment among companies deploying testing and optimization technology stems from tests that fail to produce the type of gains expected. Seemingly without rhyme or reason, even the most dramatic design changes yield “no significant differences” based on simple measures such as click-through and even less for more involved down-stream metrics such as conversion rate. While this is the reality of testing, I believe that much of the disappointment stems from a lack of attention to the definition of “success” and“failure” as the design or changes are implemented.

Success in testing can be measured many different ways:

  • For some, “success” is a dramatic increase in a revenue-based metric, knowing that most senior stakeholders will respond to incremental revenue.
  • For others, “success” is a small increase in key visitor engagement metrics, knowing that a series of small gains eventually adds up.
  • For still others, “success” is a reduction in the number of problems present throughout the site, knowing that reducing barriers improves usability.
  • For some, especially those with an increasingly dated site, “success” is simply being able to deploy a new look without negatively impacting existing key performance indicators.

A lack of “success” in testing is often viewed as a failure on someone’s part, but that is actually rarely the case. In reality, testing powers a continual learning process about your visitors and customers. If a particular image fails to increase conversion rates, you have learned that your audience does not respond to that particular image. If subsequent testing reveals that variations of the same image yield similar results, then you learn something about your audience’s reaction to the image’s content. In this context, there is no such thing as “failure” in testing, only a failure to achieve the specific defined objective.

Keep in mind that not every test can yield incremental millions in revenue for your business. Some tests will fail to produce the change desired; others will yield results but not across the key performance indicators; and still others will simply fail to produce statistically relevant differences. But it is our firm opinion that there are no “failures” in testing other than a failure to carefully design your tests and a failure to carefully consider what you’ve learned.

Ensure Good Test Design

Success with testing depends heavily on the quality of your test design. One of the reasons we recommend requiring a formal test plan is so that the Testing Team has as much information as possible to determine how the test should be run. Especially when you start to aggressively test, good test design helps ensure that any effects from participation in multiple tests can be taken into account, either by identification and isolation or outright removal from the result set.

To this end it is reasonable to consult with someone experienced in experimental design in the online world—either from your vendor or a third-party. There are several elements that constitute a good test design and it is important to pay attention to them. For example, you should:

  • know whether you need an A/B or multivariate test.
  • pick the test array that works best for your needs, either a full or fractional factorial array.
  • make sure you are running the test long enough based on traffic and conversions in order to get a statistically valid sample size.
  • make sure you are properly testing variations of factors. Improper factoring is caused by poor (or no) isolation of individual changes; for example, changing a headline’s text, font, color, and size all at the same time.

Another mistake new testers often make is always running tests against anyone and everyone; a good test design means you are targeting your tests to a relevant audience, and then performing additional segmentation on the results.

Clarify Your Testing Timeline

One of the most unfortunate mistakes that companies make when getting started with testing is to only test for statistical significance. A great deal has been written about test design and full factorial versus fractional factorial versus A/B testing. While these are all important considerations, none are nearly as important as having a test sample that takes day-part and day-of-week variation into account.

Consider that even on the highest volume sites, there are typical peaks and valleys in traffic caused by target audience geography, marketing efforts, and the particular interaction model promoted by the site. Within each of these peaks and valleys, your site is attracting a particular type of visitor—late night visitors, early risers, lunch-timers across different time zones, etc.

Assuming you’re not trying to target a specific audience segment, a truly random sample of visitors will account for this variation and sample across these visitor variants. In order to reduce test bias as much as possible, a general rule-of-thumb for test planning is the “7+1” testing model.

In this model, you will be testing over an entire week (seven days) and building in a little extra time to make sure that you have a clean break in the data for analysis. Thus, “7+1”means running your test for a full week with an extra day on the front end. By giving the test a day before you start actively tracking results, you allow for slippage and the need for last-minute changes, plus it gives the analysis team the ability to gather data starting at midnight at the end of the “+1 ”day.

And by running the test over an entire week, you will account for all of the potential day-part and day-of-week variation, at least as much as is possible. If you have the luxury of time, you may want to consider extending the test to a “14+1”model, doubling the amount of time you run the test. With two weeks, you will be better able to account for additional variation in the data arising from tactical marketing efforts, a sudden increase in referral from social media, holidays, and current events, etc.

One of the advantages of the “7+1”model is that you can adjust your sample size to still only gather as much data as you need; you’ll just gather that data more slowly. Rather than taking a 20% sample over four days to get statistical significance, the “7+1”model may guide you to take a 5% sample over seven days. The smaller sample lessens your risk associated with testing since if the tests fair poorly, fewer visitors will be exposed to them and you’re still able to get to statistical significance in a relatively short period of time. Further, it allows non-test traffic to be eligible for assignment to other tests that you may be running concurrently.

The major complaint about the “7+1”model is that it takes time and if you just open the spigot on the test, you can achieve statistical significance in a matter of hours in some cases. While this sounds good, opening the spigot on testing is exactly how not to achieve success through testing. Unless you have a very sophisticated understanding of your audience and the sampling technique employed, the “fire hose” model will likely leave you with more questions than helpful insights. Anyone who doesn’t like the results can simply argue that your sample does not represent the diversity of user types coming to the site and refuse to accept your analysis. Whatever kind of test you are running (A/B or multivariate), you want to make sure you’ve run your test long enough to obtain a statistically valid sample size—the number of participants assigned to the test.

Your sample size will be determined by a combination of traffic volume, your baseline (control) conversion rate, and the conversion rate observed by test participants. You’ll want to make sure you obtain an appropriate sample without bias in time of day, day or week, holiday/event, etc.

For example, you might run a test with a huge sample size and obtain statistically significant results in one day, but this only reflects how visitors behaved on that particular day. So take care that the test is run across a longer period of time (at least 7+1 or 14+1), and perhaps longer depending on the situation) to insure against bias.

Remember: the best testers work thoughtfully and carefully, and they are willing to spend a little extra time on process or testing to make sure they deliver accurate, reliable, and believable results to socialize through the rest of the organization.

Test Different Audience Segments

Advanced testers are testing against key visitor and customer segments. The logic behind this is clear: why optimize your site for everyone when you can focus your optimization efforts on those visitors who have already demonstrated value to your business?

There are two ways to conduct a segmented test: ad hoc and post hoc. The former method requires that you are able to identify segment members in real time so that the testing engine can assign people appropriately. For example, you may be targeting “first-time visitors” or “visitors referred from Google organic search results, which, depending on the testing platform you use, can be easily done.

The latter method for segmenting is post hoc—after the fact—which is more an analysis technique than a testing strategy. In this case, you will mine test results for segment members and compare these results across control and test groups. This strategy also involves some work between testing and analytics vendors but is often more forgiving, especially if your testing vendor supports full data export and is able to provide the analytics vendor’s ID.

Regardless of how you produce the data, focus on your key segments when communicating your test results. If you have the data and the time, it is definitely better to be able to tell management, “The test produced a 5% lift in click-through rate across all visitors and a 15% increase in click-through rate across our most valuable customer segment.” This message should resonate loud and clear, especially if your measurement team has done a good job at leveraging visitor segmentation.

(For more information, download the whitepaper titled “Successful Web Site Testing Practices: Ten Best Practices for Building a World-Class Testing and Optimization Program.”)


Suggested Posts


An Analytics Framework for Optimization


12 Strategies for Improving Online Conversions – Webinar Now Available On-Demand

solutions developer.jpg

Get to Know a Solutions Developer

Leave a Comment

Your email address will not be published. Required fields are marked *