One company correctly predicted both Brexit and Trump’s victory, using data insights that most pollsters missed. This is how to make your own.
The US presidential election results sent shock waves around the world and led us to question the validity of political polls, one of the longest-standing and most prestigious data products. Those of us who watched the polls were told that Hillary Clinton was clearly in the lead. In 2016, The Huffington Post tracked 377 polls from 43 pollsters in the US through Nov. 7th, as visualized here. The model predicted that Clinton was very likely leading.
This chart, as it appeared in the Huffington Post, combines the latest opinion polls into trendlines using a poll-tracking model and was updated whenever a new poll was released.
One South African company, BrandsEye, correctly predicted both the outcome of the Brexit vote and Trump’s victory. BrandsEye delivered a Data-as-a-Service (DaaS) product, amalgamating social media analysis, geolocation data and other inputs to create impactful insight that most pollsters missed.
What is Data-as-a-Service and why is this important?
In my blog Creating your Data-as-a-Service Customer, I explained that Data-as-a-Service (DaaS) can be described as productized data-driven insight on demand. Going beyond the technology stack and into core business decisions, DaaS distinguishes itself from other ‘-as-a-Service’ products, because it enables data to become an active partner in human decision making.
Creating the a product that delivers actionable insights using the latest technologies can be very profitable, as I explained in my blog Turning Big Data Disillusionment into an Opportunity with Data-as-a-Service Products.
Seizing the opportunity
The triumph of social media analysis taught us many valuable lessons about how to improve our existing data products by harnessing the promise of the new technologies. As data analysis becomes more sophisticated, many companies who do not see this as core to their business or do not have the means to build the requisite skillset in-house will look to providers of DaaS products.
Those who can proactively productize and scale technology at the peak of inflated expectations in Gartner’s Hype Cycle for Business Intelligence and Analytics, 2016, such as Real-Time Analytics, Geo Spacial and Location Data, Self Service Data Preparation, Hadoop-Based Data Discovery and more can realize huge monetary and strategic opportunities.
BrandsEye’s method
BrandsEye started collecting social media data in July 2016. In total, over 37.6 million conversations from over 4 million authors were collected for analysis. BrandsEye’s geolocation algorithms identified in which state the author was located, and finally the conversations from the battleground states were sent to the BrandsEye Crowd – a crowdsourcing platform trained to understand sentiment. As visualized on BrandsEye’s website, this method successfully predicted both the Brexit vote and Trump’s victory.
How did traditional polling fall short?
The Pew Research Center offered these explanations:
- Non-response bias. This refers to the people who participate in the polls – the sample population. People who have lower educations, lower incomes and are less civically engaged are less likely to take a survey for an election pollster. For that reason, the subset of the population polled was not representative of the people who vote. When the segments who are less likely to be included in traditional polls showed up to vote in greater quantities than they had done previously, the accuracy of results were negatively impacted.
- Shy Trumper. Voting for Trump was understood by many voters as socially undesirable. Thus, many poll participants did not admit their true intentions to vote for Trump to pollsters.
- More people plan to vote than actually vote. Actions speak louder than words. Voting is socially desirable; therefore, some people tell pollsters that they will vote, and might actually intend to do so at the moment they are getting polled, but then don’t show up on election day. The pollsters are aware of this behavior and try to model a likely electorate (people who vote) based on a number of factors. This time the electorate displayed different behavior than had previously been observed. Models based on flawed assumptions will output flawed results.
Why is social media analysis such a powerful tool for predicting election results?
While polls give a numeric prediction of what an expected result could look like based on survey results, social media enables analysts to understand the how and why from listening to social media activities.
A few benefits of social media analysis:
- Actual activities of a large group of people: Social media data is data created by our actual activities, not responses to survey questions about a future event.
- Participation: Social media analysis listens to individuals who express their views on a platform rather than relying on sampling techniques based assumptions from historic observations.
- Ability to monitor real-time: Social media platforms are being used all the time by masses of people. A correctly configured social listening activity can keep analysts posted on the latest trends and alert decision makers to critical arisings.
Note: I am not saying that social media analysis is always superior to surveys, but highlighting that when predicting election outcomes, actual activity and participation in opinion-forming can be seen by listening to social media. There are some caveats with social media analysis:
- Social media data is unstructured and huge. Finding results when listening to social media can be like boiling the Pacific Ocean. Comprehensive planning and data discovery is necessary.
- Detailed social media analysis can require advanced tools, specialized skills that go beyond big data (i.e. linguistics).
How can you apply these lessons to creating your next innovative Data-as-a-Service product?
Hacks
Here are some hacks that will help you action the learnings from the analysis of the US presidential elections, along with an exercise you can use to jump start your team’s next innovative Data-as-a-Service product. Those who download these hacks will receive our New Year’s Report for Innovative Data-as-a-Service Products, where we will provide you with insight and tips for monetizing the latest innovations through productizing data.
These hacks include a case study of a professor who correctly predicted the outcome of every US presidential election since 1984 plus another surprising prediction he recently released. Click here to download!
Training
Following the success of our Creating Data-as-a-Service Products at Data Natives, Elizabeth Press from D3M Labs and Dataconomy are joining forces to give your organization the opportunity to jump start your next innovative data product. In addition to and in combination with our Creating Data-as-a-Service Products workshop, D3M Labs is offering specialized workshops for social media analysis, managing social media listening projects and combining social media listening with traditional analytical methods. Contact us to book a training specifically designed for your organization.
Like this article? Subscribe to our weekly newsletter to never miss out!
Image: Abi Begum, CC BY 2.0
