Deluge may not be the most apt word to use these days what with the biblical rainstorms and gusts of wind the UK has been experiencing of late, but you cannot escape the fact that data volumes are increasing at phenomenal rates as our lives and our world becomes more and more digitized. How businesses are flooded with bits and bytes, collecting and amassing data on an unprecedented scale never seen before. How objects are collecting, storing and transmitting information from various sources to umpteen targets around the clock, 24×7. For example, consider your car. Ever wondered how your car dealer seems to be able to know in advance what kind of work needs doing to the car without even opening the car hood? Yesterday, vehicles’ onboard computers stored the information in the key fob while today they transmit the data straight to the dealership on a constant basis for them to collate and make sense of it.
Indeed, objects and devices that were dumb yesterday are turning smart and clever today and tomorrow. Many call this the “Internet of Things” and this phenomenon is undoubtedly a great thing if it improves our lives. Yet, for organizations and businesses, the data deluge that follows in today’s era of billions of smart devices and super-fast connectivity still needs to be tackled if it is to be of any use. Indeed, who is going to turn the data into value? How can the deluge be turned to a company’s advantage in order to get transformational business outcomes?
One thing is clear. The Internet of Things is poised to break the back of established big data management solutions, many of which have been around since the 70s and 80s, and are just not engineered to cope with the data deluge. That’s why many organizations have yet to embrace big data because they can’t. Sure, they can amass the data but they struggle to do something with it. And the problem will only get worse how data volumes get exponentially bigger. Consider how long-term established retail organizations have taken some 60 years to amass 800TB of data and then contrast that with telecom providers in Asia who create 20TB every hour! The data tap is now firmly on; where once there was a data trickle, there is now a power jet of data flowing constantly.
One solution is Hadoop. It gives us scale but alas not the price and performance that businesses can tolerate. In fact, Hadoop can scale out at incredible rates, but it is immature, some might say a mess. Clearly, it needs to evolve, it needs user interfaces, robust data loaders and mature toolsets. But it is wrong to think that Hadoop is the answer to the problem. It isn’t. The clue is to pair Hadoop with next-generation massively parallel analytic database technology where you can store the data in Hadoop, then use a big data analytics platform to connect it other data sources, analyze it with technology that can work seamlessly with Hadoop to help you get a handle on the data deluge and to that nirvana of data-driven business decision making.
And the best bit: if you make your business more data-centric, it will become intrinsically more valuable. Indeed, where some might say that data is the new oil, data is in fact the new currency. Some might even say that the Holy Grail is to get to a point where you can take on the data deluge head-on and run closed-loop analytics that automates processes to bring about never-ending improvements to your business. Kind of take a back seat and let the technology do its thing itself.
So, while for many who are struggling with flooded homes and battling the elements the word deluge is anything but positive, organizations can rest assured that there are at least big data solutions for the data deluge that will help them run their businesses much more smartly than ever before. And it’s only going to get easier and better. Let’s just hope that agencies and administrations can use the data deluge to help its people deal and respond to the deluge of the rainfall variety.