Big Data has an alluring promise: crunch all of your datas and find out all these very important unknown things you didn’t know before. You will find out all sorts of important things about your customers. You’ll discover what they’re buying, what they’re not buying, what they like, what they hate. All sorts things that sound very strategic and important. It’s all very sexy and appealing.
Mo’ Data Mo’ Problems
But Big Data has a Big Problem. And that is that it is most often directionless. It’s not directed at solving a specific problem. The problem with Big Data, in a nutshell, is that it is a solution looking for a problem.
I could tell you stories of large companies that I’ve worked with who have their own dedicated “Insights Team” whose only job is to hunt down these nuggets of strategic information. They routinely find entire books’ worth of information that is probably relevant to somebody–but they don’t know who. And so they send out spreadsheets that nobody reads and then grow progressively more frantic that their hard work goes unnoticed and unappreciated.
I’ve realized that their audience doesn’t respond to these insights for several reasons:
- They can reveal incompetence or, even worse, laziness
- They often create more work for busy people to actually act on them
- Sometimes they embarrass people because they reveal information that is obvious… but only in retrospect
You can probably guess that this goes over like a lead balloon. This is what some people (ahem, Gartner) call the “trough of disillusionment”.
I’ll admit I was completely smitten by the Big Data promise, and I jumped in headfirst. But I ran smack into the problems I just described, and it’s happening again, all over the world, every day. Businesses are becoming disillusioned with Big Data, and with good reason. They’ve been promised something earth-shattering and have received mountains of information that they don’t know what to do with.
So how do we move on to realize the promise?
There are a few problems that I’ve personally witnessed that need to be addressed:
1. Discovery efforts need to be targeted at business directives
2. Insights need to be routed to the subject matter expert to be vetted
3. Someone needs to translate vetted insights into the appropriate actions
4. Actions need to be implemented
5. Results need to be tracked
Right now the Big Data solutions we have aren’t really close to this. And the audience that’s interested in Big Data doesn’t really understand that this is what they’re signing on for. It’s a lot of work, and it creates a lot of work. People are busy, and most don’t have an appetite for yet more work.
I believe what we’re seeing is a schism between reactive and proactive analytics. Reactive currently rules the day in that people expect answers to questions they already know they should be asking. Proactive is the ultimate promise of Big Data, but the tools–and the audience–simply aren’t mature enough yet. You NEED reactive analytics to tell you WHY things are happening. You would BENEFIT from proactive analytics, to help you decide where you should be spending your attention.
Enter: Data Strategy
I believe what is needed is a cohesive data strategy for organizations. And I think you’ll start to hear a lot more about this, because the current haphazard approach simply isn’t working.
A viable Data Strategy should start from a menu of business topics that you KNOW your audience is interested in. Conversion rate optimization, marketing opportunities (to specific demographics), what creates customer churn–it’s going to be different for everyone. I’ve put together a comprehensive list for the book I’m writing and I’m up to 50 pages just describing the most common topics, so there are quite a few.
Once you’ve created a focused list of topics that the business needs to pay attention to, the first step is to take care of the reactive bits. This is where legacy technology like dashboards and alerts come in. They’re your first line of defense against the unknown creeping into your day-to-day operations and wreaking havoc–they tell you when something has changed.
Once the reactive intelligence systems have been established, then you can start getting into the more “out there” stuff. I liken this to peeling back the layers of an onion:
onion:
1. Level 1 is reactive, answers questions you’ve explicitly asked (“What is my churn rate for the week?”)
2. Level 2 starts to identify trends and patterns (“New customers are likely to churn within 1 month”).
3. Level 3 begins to predict things that are going to happen, and why (“New customers who buy your generic brand products are likely to churn within 1 month”)
4. Level 4 starts to tell you what you should do in order to address specific situations from Level 3 (“You should offer new customers who buy your generic brand a 10% off coupon to retain them long-term”).
5. Level 5 performs the actions recommended in Level 4 automatically
The problem lies in the fact that Big Data often gets stuck at Level 2. This is the emailed-spreadsheet problem–you’re just shuffling more work into users’ inboxes, and they’re not likely to thank you for that.
Data Strategy is about moving from Level 1 to Level 5. It’s your blueprint and roadmap to identify the questions that need to be answered and the resources needed to answer–and, ultimately, act on them–automatically. There are all sorts of topics that need to be addressed by your data strategy, and that need attention and careful consideration. Topics like:
- Should we let the patterns appear organically, or should we use set categories and labels that our organization is already familiar with?
- How much growth should we account for?
- Are there additional data sets available that we can use to answer other meaningful questions at the same time?
The list goes on for quite a while. But the point is, if you go into a data analytics project with eyes open and on the prize, you’re much more likely to be successful than if you go in looking for a mystery.
Jason Kolb is a big data architect who combines hands-on experience with big data and distributed architectures together with solid cross-functional business experience in some of the biggest companies in the world. He is the founder of Applied Data Labs and previously co-founded Latigent, which has been acquired by Cisco. He’s also working on a book on data strategy, which you can find out more about on his personal blog here.
Interested in more content like this? Sign up to our newsletter, and you wont miss a thing!
[mc4wp_form]
