Data retention fails to improve search results: study

A new study into changes in search engine’s data retention policy has concluded that search results don’t suffer from less data retention.

The study, conducted by Lesley Chiou and Catherine Tucker at the US National Bureau of Economic Research, used changes in EU law and the resultant shifts in search engine data retention policy to analyse the impact said changes had on search engine result quality.

Three changes were made, in 2008 and 2011 by Yahoo and in 2010 by Bing. Google, which accounts for 90% of European search traffic, did not make changes to its data retention policy so was not included in the study. There is cause for criticism in a methodology which focuses on some of the most avoided internet functionality around, but the report is not without interest so onward we go.

The study looked at the number of websites visited off the back of a search query from each of the providers both before and after policy changes. The number of websites visited off the back of a search query is taken as a proxy for the success of the search query. The results showed that after policies were changed to anonymise or remove user data there was no change in search query success.

The findings in the study speak to some basic assumptions of the internet’s current working paradigm: that in return for free services your data is used to enhance said service both for the users and for advertisers. While the results only speak to the highest metric for search engine quality – not looking into the number clicks to second results pages, for example – the findings do still undermine the idea that more data means a better service.

Secondly, and perhaps more pertinently, the study does not look at the success of search query advertising; such data would be difficult to collect. The study does not claim to speak to this topic, however, this is perhaps the crux of the issue, internet companies want our data in order to advertise to us, which is where they get their revenue.

While changes in data retention policy may not have harboured a detrimental effect on user functionality, they may still have done for the search engine’s advertising efficacy.

It is curious that the thing consumers provide in order to use “free” internet services, our data, does not necessarily contribute to an improvement in that service for consumers. Off the back of this study one could conceive of circumstances where a service which stored data in exchange for free use was capable of achieving profitability, and then increasing profitability, without change to the user’s service, by virtue of data retention.

In essence, user data contains inherent and potentially expansive value over time, and thus can be used to drive profits while there is little or no change to the user experience. Whether users are comfortable with this is for each to decide, but the findings in this report do provide some nuance to the understanding of the relationship between consumer data and service quality.