Log Hero allows users to access their log file data in Google Analytics. In short, this innovative solution makes the once painstaking task of log file analysis super easy. In this post, we will explain how to use and filter by user agent in Log Hero. Ultimately, user agent is useful to see which search engines are crawling your website. In addition, user agent provides valuable information such as browser version and operating system data.
Let’s get started.
What Is a User Agent?
First things first: what is a user agent? A user agent is one particular piece of information stored in your log files. (By the way, if you are new to log files, check out this article for a comprehensive overview.)
Log files are records kept by a server of who accesses your website and what assets are accessed.
A log file entry can look like this, for instance:
66.249.66.1 – – [30/Sep/2017:16:09:05 -0400] “GET /dashboards/ HTTP/1.1” 200 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
Today, we are only interested in the user agent which is highlighted in bold below.
66.249.66.1 – – [30/Sep/2017:16:09:05 -0400] “GET /dashboards/ HTTP/1.1” 200 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
When a search engine crawls your website, usually we can see this in the user agent string. Notice how in the example above we can see “Googlebot/2.1” in the user agent.
Search engines often use different crawlers for different purposes such as desktop, mobile or API traffic. Each crawler is identifiable by a specific user agent string. In our example above, we know that this user agent string identifies Google’s desktop crawler.
List of Common User Agents for Google and Bing
Here are some of the most common Google and Bing user agents that you will come across. Note that even some specific crawlers can be identified by multiple user agent strings.
Google User Agent Strings
| Crawler | User agent tokens (used in robots.txt) | Full user agent string (as seen in website log files) |
| APIs-Google | APIs-Google | APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html) |
| AdSense | Mediapartners-Google | Mediapartners-Google |
| AdsBot Mobile Web Android
(Checks Android web page ad quality) |
AdsBot-Google-Mobile | Mozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) |
| AdsBot Mobile Web
(Checks iPhone web page ad quality) |
AdsBot-Google-Mobile | Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) |
| AdsBot
(Checks desktop web page ad quality) |
AdsBot-Google | AdsBot-Google (+http://www.google.com/adsbot.html) |
| Googlebot Images | Googlebot-Image
Googlebot |
Googlebot-Image/1.0 |
| Googlebot News | Googlebot-News
Googlebot |
Googlebot-News |
| Googlebot Video | Googlebot-Video
Googlebot |
Googlebot-Video/1.0 |
| Googlebot
(Desktop) |
Googlebot | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
or (rarely used): Googlebot/2.1 (+http://www.google.com/bot.html) |
| Googlebot
(Smartphone) |
Googlebot | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| Mobile AdSense | Mediapartners-Google | (Various mobile device types) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) |
| Mobile Apps Android
(Checks Android app page ad quality. Obeys AdsBot-Google robots rules.) |
AdsBot-Google-Mobile-Apps | AdsBot-Google-Mobile-Apps |
(Source: Google)
Bing User Agent Strings
| Crawler | Role | User Agent String |
| Bingbot | Standard crawler | Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) |
| MSNBot | Previous default crawler.. | msnbot/2.0b (+http://search.msn.com/msnbot.htm) |
| MSNBot-Media | crawler for images and video. | msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) |
| AdIdxBot | Crawls ads for quality control purposes. Same as Bingbot, AdIdxBot has both desktop and mobile variants. | Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
|
| BingPreview | BingPreview is used to generate page snapshots.Note that BingPreview also has “desktop” and “mobile” variants. | Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko BingPreview/1.0b |
(Source: Bing)
How to Filter By User Agent in Log Hero
Log Hero imports this data to Google Analytics from your server log files. This data is then made available by using custom dimensions to filter. Different custom dimensions are available depending on the plan you have selected. Please check this page for further details.
Firstly, please ensure you have correctly set up Log Hero – if you are unsure check our useful setup guide.
From within your Log Hero standard view, type User Agent as a custom dimension to filter your data.
Hey presto! You can now see the user agent strings associated with your website, ordered by default by landing page. You can change this primary dimension to any of the standard secondary dimensions associated with Acquisition, Behaviour, Technology Users categories.
To see the user agents for a particular landing page, simply click on the page you are interested in and type in user agent as a custom dimension. Below is an example where we filter the user agents associated with the homepage:
From here, as a Log Hero free user, you can filter the user agents within Google Analytics by setting up an advanced filter. Just click on the little magnifying glass on the right and use those settings:
Alternatively, you can export this data into Excel or Google Sheets and apply filters to find the associated crawler.
However, this approach has a major drawback: around 25% of the traffic is actually fake, so it’s difficult to ensure your data is accurate. It’s also pretty time consuming to have to filter this manually.
Filtering with the Bot Name dimension
For this reason, Log Hero has developed the Bot Name custom dimension. This dimension is only available to users on paid plans (Big Hero and above).
Let’s filter again, this time by Bot Name as our custom dimension.
Log Hero automatically displays the name of each crawler. Not only this but it filters out the fake traffic, which is then listed as Unknown Bot.
Make sure to check back regularly for more product updates, tips and tricks. Find out more about Log Hero plans here.