The Hadoop community gathered this week in Amsterdam at the Hadoop Summit, the leading conference for the Apache Hadoop community in Europe. 800 participants ranging from Apache Hadoop enthusiasts, professionals, community members and ecosystem vendors met in the impressive venue Beurs van Berlage in central Amsterdam.

The recurring theme of the conference was that the Big Data data market is set for an impressive growth rally. Due to Forrester, also presenting at the event, this is mainly due to the fact that today only around 15% of companies actually use Hadoop technology in production while more than 40% are running POCs (proofs of concept) that will go into production over the coming years.

Big Data market expected to grow to $100B by 2020

Forrester also forecasts the market for Big Data to grow from $27B in 2015 to $100B in 2020. While in 2015 they expect that $6B (with a hardware, software and services each accounting for a third) will come from the Hadoop technology and ecosystem, that share will grow to $50B by 2020.

This trend is easy to understand if you look at the evolution of Hadoop. Hadoop 1.0 was largely about power web-scale batch applications during the early adoptions phase of 2006-2012 where customers wanted to see new technology and performance. Today Hadoop 2 caters successfully to customers looking for solutions & convenience, a clear sign of Hadoop having crossed the chasm.

For main drivers for this trend emerged again and again at the conference:

1. Business: Move from reactive analytics to pro-active customer interactions
2. Customers: Generation of insights that lead to competitive advantages
3. Technology: Data is everywhere now and the growing volumes overwhelm traditional systems
4. Financials: Cost of data systems as percentage of IT spend grows and commodity hardware becomes relatively cheap

The overall notion was that Hadoop has moved beyond being a mere tool for Big Data analyses to having finally become a true data operation system that will power the next generation of data architecture and applications. The arrival of YARN, Storm, Stinger, Tez and others have proven this impressively and were widely discussed by participants.