"Not Set" on Google Analytics "Queries" Report

I first noticed at the end of September that the keyword/query data that I can normally see when I am checking client data in Google Analytics didn’t look quite the same. The data pulled in from Search Console for the “Queries” Report was showing large percentages of keywords as “Not Set” in Analytics.

As Jennifer Slegg at The SEM Post reported all access to Search Queries in Google Search Console was removed on 16^th September and replaced with the Search Analytics report so I was not surprised to see something different – what I wasn’t prepared for was a seeming inconsistency in the data I saw pulled in from Search Console to my Google Analytics “Queries” report.

All of my Analytics views are linked to Google Search Console so that I can see which keywords are bringing visitors to the various websites I work on.

After the re-branding of Webmaster Tools to Search Console happened back in May all of this linkage with Google Analytics worked as it always had done, but after the replacement of the Search Queries report with the Search Analytics report I started to see large percentages of keywords displayed as “Not Set” – very reminiscent of the old “Not Provided” that caused us SEOs so much angst way back (when?)

On the Search Console Help Forum there have been several postings of people having the same problem and a reply by AnalyticsEdge suggested that these “not set” clicks were simply additional clicks that had not previously been reported with the old Search Queries report.

But I wanted to investigate further to better understand my data…

For most of my clients it was clearly true that the “not set” queries were indeed additional visits (so, many thanks AnalyticsEdge) which had not appeared at all when I ran the reports before 16^th September. I know this because the total number of clicks was higher by approximately the same number as the “not set” click volumes.

Search Console, and it’s previous incarnation, has never provided all query data so I don’t rely on it for absolute figures of visitor numbers because that is reliably available elsewhere. But, still, what I am seeing now makes me question my earlier stats and reports. For some clients I am now seeing 2,000+ additional clicks on the Queries report but have no idea which keywords they relate to.

According to Google these data might not be reported for a number of reasons – one being:

“To protect user privacy, Search Analytics doesn’t show all data. For example, we might not track some queries that are made a very small number of times or those that contain personal or sensitive information. “

So maybe these additional visits are for search phrases rarely used but they could also be ones containing confidential information. Actually this is quite unlikely for the websites I look after so I’m hoping it’s the former. In which case they would have little to add to my knowledge of keywords bringing visitors to my clients’ sites.

Typically, weekly or monthly, I will export Analytics reports to a PDF file to use for compiling month-end client reports so I always have a record of my reports outside Google Analytics. So I can compare the data Analytics gave me last month (stored in a PDF file) and the data it gives me now for that date range a month ago – and the 2 datasets are different.

Let’s take a look at an example from 18th August – 17^th September – there were 1,555 clicks all with a given keyword on my PDF report produced on 17^th September. When I viewed that same date range today in Analytics I see something completely different – 3,500 clicks with approximately 2,000 of them showing the keyword as “Not Set”.

So, yes, these “not set” keywords are additional data not previously reported in this example ( but see what I have to say about that for other clients below) and I simply have to assume that these are data Google has decided to filter out based on criteria such as limited search volume or containing personal information.

Am I being unreasonable to expect a reporting tool to consistently produce the same data for the same date range? Or should we just move on with this new dataset?

Even though there are explanations that these are additional clicks for which Google has decided to not provide the actual query, this makes me feel uncomfortable. I would like an explanation of why these additional visits are suddenly being shown because I rely on these data for month-to-month comparisons.

I know that Barry Schwartz (@rustybrick) emailed Google about this “not set” issue but so far he has not had a reply, although they are apparently looking into it.

Why Am I Using Search Console?

Of course, the reason why I am using Search Console’s Search Analytics data (whether I pull it into GA or not) is to just check up on what search terms are bringing visitors to a site and also check individual click-through rates – looking for new opportunities and opportunities to better target certain search phrases. I can also use these data to monitor visits from mobile devices. I don’t necessarily need the data to be 100% accurate and transparent to do that but it would be nice if it was.

Problem Filtering Search Analytics data in Google Analytics

What is also making me concerned is that I have other clients for which I cannot confirm that the numbers of “not set” keywords match up with additional numbers of clicks when I compare PDF report files produced prior to 16^th September with the same date range viewed after 16th September in Google Analytics.

This is because my PDF files filtered the data by country but this advanced filter option is no longer working in Analytics so for these websites it is not a clear-cut case of the “not set” queries simply being additional queries not reported before.

Let me explain more fully which reports I use and the exact problem I am seeing now. Feel free to get in touch and let me know if you are having similar issues – I know I’m not alone.

First I’m sure you all have your Search Console property linked to your Analytics account but if not you can check out how to do that by looking at Google’s own help here.

Now, in Analytics scroll down to the Acquisition>>Search Engine Optimization>> Queries report

What I often do next is take a look at which countries the clicks are mostly coming from so I add Country as a secondary dimension to the report…

And then I can see the country column…

So having added the country column I could simply do an Advanced Filter to only show the queries for my chosen country – in this case the UK.

But not anymore…..

When I try and include only specific countries (or indeed any of the advanced filters available) Analytics returns NO DATA AT ALL.

So annoying – this was working fine until 16th September. I will raise the problem with Google and keep you posted…

In the meantime let’s have a look at what Google tells us about discrepancies in Search Console data

Reasons Search Console data differs from data in other Google tools:

To protect user privacy, Search Analytics doesn’t show all data. For example, we might not track some queries that are made a very small number of times or those that contain personal or sensitive information.
Some processing of our source data might cause these stats to differ from stats listed in other sources (for example, to eliminate duplicates and visits from robots). However, these changes should not be significant.
Some tools define “keywords” differently. For example, the Keywords tool in Google AdWords displays the total number of Google searches for that keyword across the web. Search Analytics shows only those keyword searches that returned your pages in Google search results.
There can be a lag between when the numbers are calculated and when they are visible to webmasters. Although data gets published in intervals, we continually collect it. Normally, however, collected data should be available in 2-3 days.
Time zones matter. Search Analytics tracks daily data according to Pacific Daylight Time (PDT). If your other systems use different time zones, your daily views may not match exactly. For example, Google Analytics shows time in the webmaster’s local time zone.
Some tools, such as Google Analytics, track traffic only from users who have enabled JavaScript in their browser.

What Google tells us about discrepancies between the old Search Queries data and the new Search Analytics data

Firstly, the Search Analytics data is much more accurate so we can only assume that any discrepancies are not due to the loss of any data but rather additional data being provided, albeit data (such as the “not set” queries) of limited use to us. It is also calculated differently.

Individual page impression counts merged. The old Search Queries report counts every single page in the search results as an impression; the new Search Analytics report counts all links to the same site as a single impression (unless you group, filter, or compare by page).
Search properties and devices separated. The old Search Queries report has an option to filter by Web, Image, Mobile and Video, where Mobile means web searches from a mobile device. Mobile and desktop search metrics were combined for both images and videos. In the new Search Analytics report, the device type and the search type are separate.
Coverage and partial counts differ. The old Search Queries report and the Search Analytics report have different thresholds for how much data to store. This means that data coverage might not be the same for both reports, and summed values might be different between the two reports.
Image click count reduced. The new Search Analytics report counts only clicks on an expanded image (or on the “visit page” link) in an Image Search result that points to your page. The older Search Queries report counted any click on an image, expanded or not, in both Web Search and Image Search. This means that the click count will be lower, but more meaningful, in this new report.
Data consolidated by full domain. If you own multiple hosts in a domain (for example, www.example.com and example.com), you might see your click and impression counts drop for each host. Why? The old Search Queries report often assigned click, impression, or other data by domain name, where a domain might span multiple hosts. So, for example, a click on a link to www.example.com might be counted for both www.example.com and example.com accounts. In order to avoid this double counting, the new Search Analytics report now assigns all click, impression, and other search data to a single, complete host name. So, for example, a click or impression on www.example.com will only be counted toward www.example.com, and not to m.example.com, example.com, or any other variations. As a result, you might see lower totals for a specific account, but this does not reflect changed search traffic or user behavior, only an accounting change that ensures more accurate information for each account.