Why Real Time Data Integration is Hard, and Data Integration Technology More Important than Ever


A recent report by the Aberdeen Group highlighted that the average volume growth of data in the enterprise reached 56 percent in 2012.  For those of you keeping track, this is up from only 29 percent in 2009.  Of course, at the same time, organizations are capturing, analyzing, and using information from all corners of the business.  As we all know, the growth of data is an ongoing problem.

Back to the Aberdeen Group report, where there were a few key findings:

First, companies are integrating multiple types of data.  86 percent of companies manage structured data, while 61 percent maintain semi-structured information, and 35 percent integrate unstructured assets.  Externally, 53 percent of enterprises manage semi-structured data sets and 36 percent capture unstructured information.

Second, companies continue to incorporate multiple sources of information into their analytic and big data strategies.  There is a need for business decision makers to leverage real-time data integration to provide more relevant information to those in the company who need it.  Integrating resources in real time also ensures all employees access the same assets, which reduces risk and costs, and optimizes performance.

Finally, the Aberdeen Group revealed that there is a significant difference between firms that use real-time integration solutions and those that do not leverage the right approaches and technology.  The report noted that 89 percent of enterprises that use real-time integration have the power to provide managers with accurate information when it is needed, as opposed to only 73 percent of organizations that do not use real-time services.

The movement to real-time data integration requires an understanding that the use of data is going to be a bit different than with traditional approaches.  In the past, we accepted that data could flow occasionally from system to system, data store to data store, and as long as the data was replicated at some point, all was well.

However, the movement to real-time data integration takes a different mindset.  We need to interact with existing business transactional systems that drive the enterprise.  This means that we no longer wait for data to find its way to a back-end data store, but need to capture it directly from the core transactional systems as the data is gathered.  Moreover, the data must be transmitted and processed at the time of collection, and presented along with historical information within the BI tools or business applications.

Additionally, because the data is real time, there are data externalization issues that also must be dealt with.  This includes updating the real time data that is flowing to the decision maker, as the data changes over a given period of time.  For instance, the ability to track factory production over an afternoon, as the production data changes minute-to-minute.  In other words, we are moving from a report-oriented mentality to a dashboard-oriented mentality.

Core to the success of dealing with the massive growth of data, and the ability to leverage that data in real time is the proper use of data integration technology.  You need to select a data integration technology stack that is both adaptable and reliable around the integration of Big Data systems, traditional data stores, and, now, systems that drive business transactions.

As you approach this problem, understand your own requirements and correctly design a data integration solution.  This is critical to your success.  This means understanding all source systems, including interfaces you’re looking to leverage, and all details around those interfaces, including latency, resiliency, governance, security, etc..

Moving from the interface, you also need to understand how data flowing from the source system is changed to meet the needs of the target systems, or, perhaps the needs of the human.  This means accounting for the differences in data semantics and content, and doing so in real time as the information flows from the source(s) to the target(s).

The selection of the right data integration technology is key to your success.  Not all data integration technology providers can deal with real-time data integration, or an increasing volume of data.  Thus, you need to make sure you select a provider that will meet your requirements now, as well as in the future.  This requires careful analysis and an understanding of the existing problem and solution patterns.

The Aberdeen Group report is merely stating the obvious.  Data is becoming the single most important asset to our business.  The ability to understand and leverage this data to drive real-time operational changes to the business for efficiency and optimization will result in success.  Missing the value of this fact, most enterprises will find themselves lagging in their respective markets.  Who would have thought that data is so powerful?

About David Linthicum

Dave Linthicum is the CTO of Cloud Technology Partners, and an internationally known cloud computing and SOA expert. He is a sought-after consultant, speaker, and blogger. In his career, Dave has formed or enhanced many of the ideas behind modern distributed computing including EAI, B2B Application Integration, and SOA, approaches and technologies in wide use today. For the last 10 years, he has focused on the technology and strategies around cloud computing, including working with several cloud computing startups. His industry experience includes tenure as CTO and CEO of several successful software and cloud computing companies, and upper-level management positions in Fortune 500 companies. In addition, he was an associate professor of computer science for eight years, and continues to lecture at major technical colleges and universities, including University of Virginia and Arizona State University. He keynotes at many leading technology conferences, and has several well-read columns and blogs. Linthicum has authored 10 books, including the ground-breaking "Enterprise Application Integration" and "B2B Application Integration."

View all posts by David Linthicum →