We have exploded into the age of big data. After years of big data hype everyone wants to know the state of the union. As, I noted in my last post, “BIG DATA 2.0: What is Driving the Big Shift?,”we are experiencing a major shift from Big Data 1.0 to 2.0. Big Data 1.0 has been all about introducing new technologies to take advantage of all the new data being created. One of the frontrunners that has emerged with great promise is Apache™ Hadoop. From a 50,000 foot view, Hadoop has gone from being an open source phenomenon like no other to becoming a common household name among technical and business people alike. Big data has evolved into something beyond what we would call a market; it’s a movement, a global movement.
The growth and adoption of Hadoop continues to astound the marketplace. From its first release in December of 2007 until now there widespread distribution of the new big data platform has exploded thanks to the open source community and companies like Hortonworks, Cloudera, and MapR. This first phase of big data brings us 5 significant achievements that change the way organizations all over the world approach data and analytics.
ACHIEVEMENT 1: Enormous, affordable scale compared to old storage paradigms
The traditional data storage market has been under attack from above and below. From above, cloud vendors like Amazon, Google, and Microsoft use cloud storage volumes to drive down the overall cost of simple storage. From below, HDFS (Hadoop Distributed File System) now offers a simpler way of storing data than the higher cost options provided by NAS, SAN, and databases. The result is that big data companies and organizations all over the world are replacing outdated storage systems with new file system storage at affordable rates.
ACHIEVEMENT 2: Capture and store all data without first determining its value
With the cost of storage at bargain prices, everyone from the phone company to the government has decided to keep data that had always been thrown away. Energy companies keep their historic data. Log files mount up at never before rates. And new data from sensors, social media, and the internet of things has found its way onto Hadoop storage systems everywhere. It is no longer necessary to establish the value of data in order to keep that data. Thanks to Big Data 1.0 companies now keep all their data and establish the value of that data at a later date.
ACHIEVEMENT 3: New types of data present new opportunities for analytics
All of the new data being amassed in Hadoop represents new opportunities for predictive analytics. Mobile apps produce huge amounts of behavioral data that can be used to drive hyper-segmentation. Social data provides never before captured context to what is happening around markets, brands, and segments. Sensors create micro-snapshots that can be pieced together into valuable predictive models. Emerging data types and new combinations of data are now driving a deeper, richer set of analytic results and creating even more value for organizations.
ACHIEVEMENT 4: Data discovery and data provisioning now common in most organizations
New storehouses of data pouring into file system storage have spawned new practices around data discovery and data provisioning. The first order of business around big data has been to discover what is actually in the data. The next order has been to provision that data to the different people and applications that need the data.
ACHIEVEMENT 5: Analytic applications emerge as the number one use for big data
For everyone involved in the big data movement there seems to be consensus around what to do with the oceans of data that have been collected and categorized. Everyone understands that analytics unlocks the value within the data. Data has been captured. Maps have been created to chart out the kinds of data captured and the potential for extracting value. Analytics are now set to unlock the more than $15 trillion of value locked in these new, massive data stores created in Hadoop.
These five achievements open the door for the next phase of big data where projects will move out of the lab and into production. Data captured means value waiting to be realized. More people will gain access to the data, time to value will be accelerated, and big data will become a prime mover in company and organizational success.