Key Takeaways

  • Data lineage is a detailed representation of data over time: its origin, processes, and transformations.
  • Big data made centralized lineage hard, because data now lives across many technologies and storage systems.
  • A data catalog centralizes metadata from every storage system so you can visualize lineage in one place.
  • Dataset-level lineage shows the processes and datasets that created a final dataset.
  • Column-level lineage shows which columns and actions created a specific field.

Data lineage is a type of data life cycle. It is a detailed representation of any data over time: its origin, processes, and transformations. Although this is not a brand new concept, a paradigm shift is taking place.

Why is Data Lineage Harder in a Big Data Environment?

Getting data lineage from a data warehouse used to be a simple task. Because a data warehouse is centralized storage, you could obtain lineage by design from the data stored in one place.

The data ecosystem has evolved rapidly since the emergence of big data, as new technologies and storage systems have complicated enterprise information systems. It has become impossible to impose a single centralized tool across an organization. The software and methods that enterprise architects used in the “old world” have become harder to maintain, leaving their work obsolete and illegible.

How Do You Visualize Data Lineage in a Big Data Environment?

To get a global view of an enterprise’s data, new tools are emerging. One of them is the data catalog. It processes the maximum amount of metadata from all data storage through a user-friendly interface. By centralizing that information, you can build data lineage in a big data environment at two levels.

Dataset-level lineage Column-level lineage
What it shows The processes and datasets that created the final dataset The transformation stages of a dataset as a timeline of actions and events
Example sources A table in Oracle, a topic in Kafka, or a directory in the data lake A specific field you select
What you learn Which processes and datasets produced the result Which columns and actions created that field
Limit Does not answer questions about sensitive data or which columns were created and by what processes More granular, so it answers those questions

At the Dataset Level

A dataset can be a table in Oracle, a topic in Kafka, or a directory in the data lake. A data catalog highlights the processes and datasets that made it possible to create the final dataset.

However, this standard of data lineage on its own does not let data users answer all their questions. Among others, these remain: What about sensitive data? What columns were created, and with what processes?

At the Column Level

A more granular approach is to represent the different transformation stages of a dataset as a timeline of actions and events. By selecting a specific field, users can see which columns and actions created it.

What Does Data Lineage Look Like Today?

The Actian Data Intelligence Platform captures lineage automatically across cloud and on-premises environments and connects it to a federated knowledge graph, so lineage, glossary terms, owners, and policies sit against the same asset. Its smart lineage visualizes end-to-end data flow for impact analysis, root-cause tracing, and up-to-date documentation.

For a full overview, read Data Lineage: A Complete Guide for Enterprise Teams.