Over on Bloomberg TV, Actian’s CEO Steve Shine made some strong points about the next generation of big data analytics, and that we really are ready for that second phase. In particular, he pointed out that there were some inherent weaknesses in the first phase of scale-out of large scale analytics infrastructures, and some serious pitfalls that had been road-blocking the success of a lot of big data projects. Nearly every point he made, he illustrated with a real world example. This isn’t just blue-sky thinking. It’s how the CEO of a data-centric company sees the industry and the technology shifting, and he isn’t alone in thinking that way.
I had heard several of the points Steve made expressed in recent posts by people completely unrelated to Actian. Lots of articles have identified the shortcomings of the first gen approach, and Actian is far from the only solution to see the need and seek to fill it. We just got a bit of a head start. Hadoop 2.0 with YARN is, in many ways, about opening up the basic big data analytics platform to fix some of the weak points. The Big Data Analytics 2.0 idea is that data analysis tools exists now to fit in that new, open framework and fill in the gaps.
There was an excellent article on Wired a couple weeks ago by Gary Nakamura: Why Hadoop Only Solves a Third of the Growing Pains for Big Data. He identifies some of the shortcomings of early big data processing technologies, and those tend to echo the problems we’ve been seeing at Actian. Pitfalls that sabotage big data projects before they get off the launching pad, like false starts and resource drains, and scarce, but essential expertise to avoid them, are well known. This Frost & Sullivan post on InformationWeek by Doug Henschen outlines 3 Big Data Pitfalls to Avoid, and spots some of the same issues that Steve Shine does. Frost & Sullivan are battle-scarred veterans in this often young industry. They see what we see. Companies like Opera Solutions have already made the shift to provide data science with lower latency and greater flexibility.
Accelerating Big Data 2.0 is certainly an Actian-sponsored term, but the ideas behind it are far deeper and more pervasive.