I have a love hate relationship with Legos. Love them because some amazing people now understand how to deconstruct the Death Star into a series of interlocking polygons so that your kids can re-assemble it (or in my case, Dad can, since my son is still a little guy). The hate comes from the fact that those sets, no matter how intricate, still leave a lot of the imagination unfulfilled. Gray pieces of the Death Star hardly allow for you to build more organic settings or even a simple brick house, and inevitably you run out of Legos. But what if that were not the case?
Actian recently had Josh, a new statistician, join the team from one of the legacy software vendors, and once he gained his footing, to him, it was like landing in a heap of Legos at Legoland. Every possible data combination, every kind of data, limitless analytical building blocks….think of the possibilities of what you could build. It is one thing to have incredible algorithmic depth or master builder knowledge…it’s another thing to implement those ideas without any thought of limitations! We talked about what specifically makes the Actian Analytics Platform a catalyst for innovation in the realm of data science.
The first was the ability to analyze all of the data. No longer was Josh bound by the notion of analyzing tiny statistical samples of the data set because the software could not scale. If you are trying to find the proverbial needle in the haystack, can you truly achieve your goal when you can’t look at more than 3% of your data to create your models? MPP databases and Hadoop provide ridiculously ample storage for your data and Actian’s modern parallel processing paradigms now allow you to process all of the data where it lives.
The second revelation was being able to examine data of any type and frequency. There are so many different types of structured and unstructured data now and you may not know where the next great insight may come from. Previously Josh would be bound by traditional methods and access to data that IT could accommodate easily, which was often batch data from legacy systems. Now social, machine, and SaaS application data are all within reach and fair game for advanced analytics.
The final point was that he could now collapse the iterative process of the analytics development cycle because of the efficiency and performance of the different pieces of the big data analytics platform. If up to 80% of your analytics development is preparing the data, then having a visual design environment for ETL and data quality that is massively parallel can significantly reduce that time. He spent many an hour in that previous life waiting impatiently on training analytics models while he read email. I heard him make an exasperated comment the other day about how much time he wasted in his old job. Just the thought of going back to the old serial world would probably provoke a bit of analyst road rage.
Those three points are all really cool from a technology perspective, and making statisticians like Josh happier people, but what is the potential impact to a business? Those three together create the potential for innovation that leads to transformational value. Building models against larger data sets potentially leads to more accurate models and subsequently better decisions made by the business. Augmenting traditional analytics with new sources of data at lower latencies allows new insights and more personalized predictions of behavior at a level of granularity not possible before thanks to predictive analytics. Innovating with these massive, diverse data sets is possible now because of the ability to experiment and iterate through the modeling process multiple times, in the time it used to take to wait for one slow model to train.
Big Data 2.0 has the potential to fundamentally change the way businesses operate. Whether it’s new and better answers to old business problems or fantastic new ways to do business. It’s like a whole kingdom to yourself, packed full of Legos of every size, shape and description, with no limits as to what you can build. The question is….
“How would you innovate if you had no constraints?”