The Flying Monkeys of Big Data


In his recent blog post Big Data and The Wizard of Oz Syndrome, Rick Sherman examined mistakes commonly made by enterprises when initiating their first big data projects, noting there is nothing new about these common mistakes, as they are the same ones enterprises have been making during every new technology wave.

As to why history repeats itself, Sherman explained, “The technology hype cycle is so powerful that enterprises continually stumble in their initial projects for the latest wave.  Often these stumbles are quite costly (and career damaging).  Vendors, analysts, and pundits typically pitch the latest technology wave as solving all the problems encountered in the previous cycles.  Just as the Wizard of Oz creates the myth of his power (don’t look behind the curtain), so do pundits proclaim the latest technology’s wizardry.”

I remember as a child being frightened by the flying monkeys in The Wizard of Oz movie, which were re-imagined as the far more frightening flying baboons in the new Oz the Great and Powerful movie. The juxtaposition, combined with Sherman’s remarks about common mistakes on big data projects being nothing new, made me wonder if the flying monkeys of big data just seem more frightening because of all the hype accompanying big data and its supporting technologies.

In his recent post, Richard Maddox provided a balanced perspective about MapReduce – Hadoop’s Mighty Data Hammer (presumably the weapon of choice for the flying monkeys of big data), including not only its positive impacts but also its limitations.  “The bottom line,” Maddox explained, “is that MapReduce isn’t ideal for all big data challenges and has particularly left much to desire in terms of real-time or near real-time event processing.  MapReduce has also fallen short in some cases when dealing with more complicated, deep-dive analysis projects.”

“Big data,” Sherman concluded, “is proclaimed as the data nirvana for enterprises that have been struggling with ’old fashioned‘ data and its associated tools such as relational databases and SQL.  This time it will be different and an enterprise need not worry about information management.  Big data will likely be similar to the initial web wave: the Internet Bubble.  The web did change the world as the people hyping it at the time proclaimed, but not before many enterprise failures and vendor bankruptcies.  It took much failure and more hype than the pundits thought, and it is likely big data will follow some of the same path.”

Just as the Yellow Brick Road eventually leads to the Emerald City, but not before passing through the Dark Forest, the path to big data success is fraught with peril.


Big data in its scariest information-overloading moments is reminiscent of the sky darkened by a descending swarm of flying monkeys.  You might like to believe that those flying monkeys will provide you an Emerald City shortcut by carrying your organization to successful big data projects, but it’s more likely that they will deliver you to the Wicked Witch of wasted money, time, and effort.

Just like theIdeal Chronicler, the Great and Powerful Oz is a myth.  It’s going to take a lot of work by the men and women behind the curtain in order to orchestrate successful big data projects.

About Jim Harris

Jim Harris is the Blogger-in-Chief at Obsessive-Compulsive Data Quality (OCDQ), which is an independent blog offering a vendor-neutral perspective on data quality and its related disciplines. He is a recognized thought leader with more than 20 years of enterprise data management industry experience. Jim Harris is an independent consultant, professional speaker, and freelance writer for hire.

View all posts by Jim Harris →