Googlebot Hits in Google Analytics after Using URL Inspection Tool

Choose the Service You Need

The other day I was playing with the new GSC URL inspection tool and found one interesting thing. I decided to shoot a quick post about that. This is not something super new or extraordinary, but still good to understand.

We all know that Google Analytics reports many hits from bots: this system is too easy to manipulate. But normally, GA doesn’t have any info on Googlebot visits unless you specifically configure it to show this data. Here is what Google Help says about that:

search-engine-bots-in-google-analytics

But here is what I see in my Google Analytics:

googlebot-in-google-analytics

The thing is that when you’re using the URL inspect tool for testing a live URL, Google sends a hit (2 hits in most cases) to your Google Analytics view. You can see it in the real-time reports:

google-search-console-hits-in-google-analytics

The country is the USA, the source, browser and many other dimensions are (not set).

Here is how these hits are recorded in the server logs:

66.249.75.68 – – [11/Sep/2018:23:25:51 -0400] “GET gtm-variable-check-if-google-analytics-code-is-found-in-page-source/ HTTP/1.1” 200 10192 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

66.249.75.68 - - [11/Sep/2018:23:25:51 -0400] "GET gtm-variable-check-if-google-analytics-code-is-found-in-page-source/ HTTP/1.1" 200 10192 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

66.249.75.68 - - [11/Sep/2018:23:25:51 -0400] "GET /robots.txt HTTP/1.1" 200 123 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Note that there are 2 hits from Googlebot, the second one being from Nexus 5X which Google uses to check mobile-friendliness. But in Google Analytics, Device Category still shows “desktop” for both hits.

As you can see from the logs, Google visits the robots.txt file on each live URL inspection. This makes sense as the bot needs to check if indexation is allowed.

Just want to highlight it with this beautiful blue colour: the hits are recorded by Google Analytics only when you are testing live URLs as opposed to retrieving data from Google index.

What’s about the old Google Search Console?

The old GSC also has a similar tool – Fetch as Google. It also sends 2 hits when you choose ‘Fetch and Render’ (and does not send them if you are only fetching the page).

old-google-search-console-fetch-as-google

By default, both hits sent by Googlebot are reported in Google Analytics as “desktop”. But if you choose “Mobile: Smartphone”, you will see that one of the hits is “desktop” and the other one is “mobile”.

google bot-hits-google-analytics

What’s in it for me?

Nothing terribly serious. But if you often use the URL inspection tool to test live pages, I would exclude googlebot traffic from your filtered GA view and add a custom segment to filter it out from the past reports. The steps are the same as I described in this post about amazon keyword in Google Analytics, just use “googlebot.com” in the ISP filter.

That’s it! Hope you’ve found this quick note useful.