Bot filtering

This page explains which hits Clear Saga leaves out as automated traffic, so you can trust that the numbers are people, and so you know why a test you ran with a script or a headless browser never appeared.

Why bots are left out #

A large share of the requests any website receives does not come from people. Search engines read your pages, monitoring services check that the site is up, and scripts fetch pages for every reason there is. Counted as visits, they would inflate your page views and fill the list of sources with noise.

The filter is always on. It has no settings and there is nothing to configure.

What is filtered #

Every hit goes through a fixed list of checks before it is stored. The first check that matches decides, and its reason is what the dashboard shows:

• No user agent
• Verified crawler
• Script or HTTP client
• Declared bot
• Automated browser
• Referrer spam
• Datacenter network

In the order they run, the checks are:

1. No user agent. A browser always says what it is. A request that does not is not a browser.
2. Verified crawler. The network that delivers the request vouches that it comes from a known crawler, such as the one of a search engine.
3. Script or HTTP client. The user agent names a command line tool, a programming library or an automation framework. This is what catches curl, most scripts, and headless browsers that announce themselves.
4. Declared bot. The user agent matches a maintained list of known bot signatures.
5. Automated browser. The script on your page saw that the browser is driven by automation and marked the hit. This catches a real browser that is steered by a testing tool, which the user agent alone would not give away.
6. Referrer spam. The hit claims to come from a domain on a list of known spam domains. Such hits exist only to plant a link in your list of sources. The list that ships today is a short one of well-known domains, so it will not catch every spam domain.
7. Datacenter network. The request comes from the network of a cloud hosting provider, where servers live and people do not. Networks that carry consumer VPN and private relay traffic are left off this list on purpose, so real people on unusual networks still count.

The lists behind these checks ship with the product. Nothing is fetched while a hit is handled.

What happens to a filtered hit #

• It is not stored. It appears in no report, no visit and no funnel, and it cannot be brought back later.
• It does not count toward your allowance. Only stored events use the monthly allowance of your plan. See plans, usage and limits.
• Only a count is kept. For each hour, the number of filtered requests per reason is saved, and nothing about the requests themselves.
• The sender cannot tell. A filtered hit gets the same answer as a stored one, so a bot learns nothing from it.

Where to see it #

On the main dashboard, the Sources card has a Bots tab. It lists the reasons with the number of requests each one filtered in the chosen range, and the share that was filtered out of all recorded and filtered requests. These are requests, not visitors: a filtered hit is never turned into a visit, so there is nothing to count people by. Filters do not apply to this tab for the same reason, and the counts are kept per whole hour.

Settings, Installation also has one line on the last day: how many hits were recorded, filtered as bots, left out by exclusion rules or dropped.

Why your own test does not show up #

If you check your installation with a script, the visit is filtered. This is the filter working, not a broken tag.

• A request sent with curl or from a program is filtered as a script or HTTP client.
• A browser driven by a testing or automation tool reports that it is automated. The hit is filtered as a script or HTTP client when the browser names the tool, and as an automated browser otherwise.
• A test that runs on a cloud server or in a hosted build pipeline comes from a datacenter network.

To test, open your site yourself in a normal browser window, on your usual connection. Also make sure Global Privacy Control and Do Not Track are off in that browser, and that you have not opted that browser out, because all three stop the script before it sends anything. Troubleshooting has the full list of things to check.

What cannot be caught #

No filter of this kind is complete, and it is better to know the gaps:

• A bot built to look like a person. A real browser on a home connection, run by software that hides the automation mark, passes every check. Catching it would mean watching how a visitor behaves, and that is not done.
• Bots that never run the script. Many crawlers fetch your pages and do not run JavaScript. They send no hit at all, so they are neither counted nor listed as filtered. The Bots tab therefore shows only automated traffic that ran the script or called the collection address directly.
• New spam domains and new tools. The lists are fixed files. They change only when the product is updated.

The filter can also be wrong in the other direction. A person who browses from a cloud server, for example through a remote desktop at a hosting provider, is filtered as datacenter traffic.

If one source of unwanted traffic keeps getting through, you can leave it out yourself by path, hostname, country or address. See excluding visits.

Next: Excluding visits, Troubleshooting, Privacy: cookieless by default