Bad Bots and How to Stop Them

Everyone knows that bots can be really bad news. Did you know that today automated bots outnumber humans on the internet? As a result, they are often mistaken for real users. The marketing team wants their activity filtered in order to get quality metrics, which means IT gets the headache of resolving these issues as effectively, efficiently, and quickly as possible. It’s a tough challenge: Filter too much and valuable human user data gets removed, filter too little and the bots get through, skewing your data… which is even worse.

SiteSpect takes a proven approach to solving this difficult and ever-evolving problem.

Our Bot Filtering Method

Our solution places JavaScript at the top of a page that injects a request for an image object. The object we use is aptly named ard.png, which stands for Automated Bot Detection (ARD). ARD identifies users and bots that do not render JavaScript. We then exclude these from your A/B testing data.

ARD checks are only applied to new users to the site. SiteSpect automatically adds that JavaScript at the top of the page along with certain parameters. SiteSpect is waiting for that user to send a subsequent request with that object and the correct parameters and cookies set. Users are given three minutes to send that information to us. If not received the system marks that user as a bot and delete their data from the campaign.

SiteSpect does not block the request. Instead, SiteSpect does not include the bot data in the campaign reports. Once a user has passed the ARD check, subsequent requests and site visits are not re-evaluated.

The Good and the Bad Bots

You may ask why not simply block the bots? The answer to that is that there are good bots out there. Think Google or Bing for instance. You want to them to be able to crawl your site. You just don’t want them being added to a campaign.

Can you get metrics about users flagged as bots? The answer is yes. High-level bot detection information is available in the SiteSpect Control Panel. Click on Manage and then navigate to Monitoring->Site Metrics. The last graph lists users that failed all of our bot detection mechanisms.

So, to all of you who have ever wondered why SiteSpect injects that blocking JavaScript at the top of your page, now you know. And you also know why it improves the quality of your data, makes your marketers happy, and lets you move onto more important things.

“Tal Talks” is an article series for IT professionals from SiteSpect’s resident IT guru, Tal Cohen. For over 12 years Tal has been assisting SiteSpect customers with digital infrastructure issues and challenges in improving customer experience.

For more information on how SiteSpect handles robots, bots, and crawlers, check out our knowledge base article, click here.

Read More Tal Talks

To learn more about SiteSpect, visit our website.