Data is only good as the methods you use to collect it. With Google Analytics you track everything including search bots which isn’t helpful. Search bots crawl every page of your site and could mess up your data. This is especially true if you have low traffic. Luckily, within Google Analytics there's a setting you can check to filter out known search bots.
How Much Traffic Do Search Bots Generate
According to the 2015 edition of Imperva Incapsula’s Bot Traffic Report:
- Humans make up 51% of all traffic online
- Good bots (bots that follow the rules) make up 19% of traffic
- Bad bots (bots that ignore rules & best practices) make up 29% of traffic
That means that 49% of all traffic is coming from bots. If you're tracking even a small fraction of this it will destroy any meaningful data.
Most bots don't run with JavaScript enabled. That means if you use the JavaScript code for Google Analytics that you already ignore most search bots. Having said that – JavaScript frameworks are on the rise and search bots need to see generated content and they'll likely start enabling JavaScript.
And of course if you use a server side (API) method of sending data to Google Analytics you might already be tracking a ton of search bots. This implementation isn't as common but you do tend to see it more with e-commerce & membership sites so they can send sensitive information to GA without anyone snooping.
Since bots are such a large percentage of traffic even if only a few of them use JavaScript it can mess up your data. So it's better to be safe than sorry.
Filter Bots & Spiders
Let's get on to actually filtering out those bots. Log into your Google Analytics account. You should see something like this:
A list of your accounts within Google Analytics. I named my three views: master, test, and unfiltered. Yours may be called something different.
Go ahead and click on one of your views (either Master, Test, or unfiltered, for the image above). And then click on Admin.
Click the Admin tab
And then View Settings.
Click View Settings for your main view
And we finally find the setting. Check Exclude all hits from known bots and spiders and click Save. And that's it. You've filtered out the bots.
Filter bots!
It's a tiny setting but it can filter out a lot of misleading data.
What Exactly is a Known Search Bot?
If you're curious like me you'll want to know what a known search bot is. Does it just mean googlebot? Are there hundreds, thousands, or maybe even millions?
Google Analytics uses the IAB/ABC International Spiders and Bots List. This list is updated monthly by people at Microsoft, Google, Yahoo, AOL, and a whole bunch of other companies. You'd have to be a member to get the full list (for the low low price of $14,000). But even without the full list I'm pretty confident in these companies and their desire to report bots.
Happy filtering!