In a recent piece on the New York Times, Harvard’s Dr. Mullainathan talks about investigating claims of planned obsolescence in iPhone models – slowing down older models as updates appear. Looking at Google trends data, Laura Trucco, a Ph.D. student in Economics at Harvard, revealed that the worldwide searches for “iPhone slow” had an incredible spike at approximately the same time an update was released. The implication was clear, especially when compared to search results for “Samsung Galaxy slow” where, surprisingly, no correlation was found with release dates. Whilst not settling on a conclusion, it was enough to leave an unpleasant taste in the mouths of Apple users.
The article has received incredible attention since it broke earlier today, with everyone from the Guardian, Huffington Post and The Independent reporting the story. It’s made quite a big splash, which has been surprising to many given it is based on a totally spurious connection that ignores many technical factors. The major one, of course, is that each version of the iPhone is accompanied by an operating system update which is optimised for the newer hardware.
Google Trends – the service which looks at search results on a weekly basis, and is at the backbone behind the article by Dr. Mullainathan — can often be an interesting source of information that points to interesting correlations. It’s important, though, to remain critical when such news stories break. Just like Google Flu trends was applauded for its ability to predict and prevent flu outbreaks, it later became apparent that the initial reaction was overhyped.
The lessons from these examples are important to remember when stories like the one today are released. As John Ayers, one of the authors of a recent study that criticises Google Flu trends for being inaccurate, advises,
“Big data is no substitute for good methods, and consumers need to better discern good from bad methods…Big data has big value…But to realise these gains we need better science,” Ayers said.
In blunter terms: the research posted by Harvard is just half a story and relies on neither a good method, nor good science. If big data has taught us anything, it’s that multiple data sources must be analysed in conjunction with one another in order to gain any worthwhile insights. What the aforementioned case study has done, however, is the complete opposite, and the widespread coverage of the story is nothing short of irresponsible journalism.
Interested in more content like this? Sign up to our newsletter, and you wont miss a thing!
[mc4wp_form]
(Image Credit: Yutaka Tsutano)
Have we read the same article? Dr Mullainathan makes the same point about the limitations of big data, and about operating systems:
1) “In this way, the whole exercise perfectly encapsulates the advantages and limitations of “big data.”” […] “Finally, we see a big limitation: This data reveals only correlations, not conclusions.”
2) “Every major iPhone release coincides with a major new operating system release.” […]”a new operating system, optimized for new phones, would slow down older phones.” “We are left with at least two different interpretations of the sudden spike in “iPhone slow” queries, one conspiratorial and one benign.”
The objection here, Marco, is that Dr Mullainathan built his headline around a total non-story. He touches on counterpoints throughout the article, but never with any real conviction. Either of those two points you mentioned should have been reasons to seriously reconsider his approach to the story, rather than sidenotes in the text.