Embarking on a big data project can be daunting. If you believe that you need insight into your business activity, that you have to collect a large “big data” stash, and that you need to analyze it all to generate worthwhile insight, you will likely feel swamped by the task ahead to get any meaningful value from this project.

Big data, especially open data, has huge potential. Many businesses hold the same data – that is, data gathered from open-data sites – and combine it with their own data to find something unique. Being able to determine which way each of the elements in this data interact, and which way you can use it to determine a repeatable outcome, is the key to the Eureka moment.

The potential business inspiration can provide can only be attained if you know the why’s and where’s of what you are collecting. Just collecting data, often very repetitive data, will not help generate that insight.

Is Big Data Popular?

While virtually all growing companies today rely heavily on data, all data is not made equal.

Not all data can be joined to other data to make a relevant insight turn into a business decision. For example, you’re likely aware that supermarkets provide loyalty cards. Why? These cards allow them to hold details of where customers regularly shop and what they buy. The money-back offers those supermarkets extend to customers for providing that data could be seen as goodwill gestures, but they really serve as incentives to keep on using that loyalty card.

So is big data popular? It certainly is in the retail industry, at least by vendors. Perhaps customers that realize they are part of a bigger picture and accept the same loyalty or credit card on face value are also happy. There are, however, many people who do not consider “loyalty cards” fundamental drivers of big data generation. Some don’t see them as good ideas at all. If the data that is generated is not properly used, it has no benefit. Storing data for a long time and hoping it will be useful can make the idea unpopular. Most businesses now know the data they collect holds value – and pulling that value out at an industrial scale is becoming easier and less costly as cloud computing and storage have continued to evolve. More businesses are taking to that challenge every week.

Does that big data store have more than just a finite role of growing a business? Can it hold deeper meaning?

Exposing the Value of Big Data

Starting the exploratory big data journey means you are considering gathering a data set that is bigger than you have dealt with previously. Every business has a different threshold of what that is. A mom-and-pop store may be bringing up all of their customers’ orders into a single dataset – reloading, as is possible today, 25+ years of data from many backup sources. A new system with one terabyte of storage can seem intimidating. But by using today’s ingestion ETL/ELT tools, getting the data in and adding current data can be achieved.

The same principle applies to mid-size and large organizations. A company of 1,000 employees may embrace digital transformation, moving away from traditional on-premises data to a cloud-based system. But the key decision the company faces is not how to get 25 years of archives onto one storage server, but rather how to embody all of its current data centers into a cloud-based environment.

Why do companies move to the cloud? What is the driving force? Perhaps they have, from the data already analyzed, determined that growth for them will require greater flexibility than they can generate using their existing infrastructures. The power of modelling, insights from big data, more growth with less aging infrastructure, and the retraining of staff: these can also be seen as the positive actions of a company that is moving forward and marking itself as a place to work where best practice and delivery of high ROI is paramount.

So, how can companies ensure they are getting the most value from their big data projects?

Step 1: Hold the First Insight Meeting

What should companies want to know that they don’t already know? For one thing, is the business in good health? Many times this question is one we assume is a known quantity. However, that preconceived idea can be tested. Every business in this world faces or has faced some type of dilemma, but the key to understanding if it is a pending crisis or a beneficial inflection is big data and the analytics that comes with it. Without the analytics and deep discovery modelling, the data itself will not help.

Step 2: Mentor Your Data Engineers

Most owners of big data do not have the skills to understand the data, nor do they need them. Management’s key role relating to industry growth is acquiring the people with the right skills. This may be direct employment, a contractor/consultant or a service provided by a specialist company.

Big data empowers the business, but only if the right people are in place to analyze the data. Gathering the data and analyzing it should deliver benefits, ultimately helping businesses avoid a large crisis by predicting any pending declines.

Step 3: Conduct a Strategic Trend Analysis

Knowing the market and the current and prospective customer base can bring early benefits. Early benefits translate, when acted upon in a timely fashion, to greater turnover. And from that, greater profits come. Predicting the next trend in the market is likely to be a hybrid solution based on modelling, analytics and incorporation of the current social and environment semantics.

If a company fails to attract its target customer or misjudges the tremor of existing customers leaving, then all of the goodwill built up will drain away. All the big data the company has collected is worth nothing unless it is put to use. Keeping updated is of utmost important for businesses to stay alive and, with the right wind in their sails, move to a leading position in their chosen fields.

Big data projects may feel overwhelming when you start, but with careful planning and the adoption of tools that fit your business needs, you can overcome the challenges and take advantage of the insights data can provide.

About Keith Bolam

Keith Bolam is a veteran Data-Oli-gist at Actian, with nearly 40 years of hands-on experience in databases and operating systems. Since 1985, Keith has specialized in Data Application and Hierarchical Database design, focusing on Ingres technology and the x100 capability. He's guided teams through complex DB deployments, often presenting to user groups on data performance tuning. His Actian blog posts highlight best practices in database optimization and legacy system modernization. Check out his articles for deep dives into Ingres enhancements.