Databook: How Uber Turns Data into Exploitable Knowledge With Metadata

Uber is one of the most fascinating companies to emerge over the past decade. Founded in 2009, Uber grew to become one of the highest valued startup companies in the world. There is even a term for their success: “uberization”, which refers to changing the market for a service by introducing a different way of buying or using it, especially using mobile technology.

From peer-to-peer ride services to restaurant orders, it is clear that Uber’s platform is data-driven. Data is the center of Uber’s global marketplace, creating better user experiences across their services for their customers, as well as empowering their employees to be more efficient at their jobs.

However, Big Data by itself wasn’t enough; the amount of data generated at Uber requires context to make business decisions. So like many other unicorn companies, such as Airbnb with Data Portal, Uber’s Engineering team built Databook. This internal platform aims to scan, collect, and aggregate metadata to see more clearly the location of data in Uber’s IS and its referents. In short, a platform that wants to transform raw data into contextualized data.

How Uber’s Business (and Data) Grew

Since 2016, Uber has added new lines of business to its platform, including Uber Eats and Jump Bikes. Some statistics on Uber include:

• 15 million trips a day.
• Over 75 million active riders.
• 18,000 employees since its creation in 2009.

As the firm grew, so did its data and metadata. To ensure that their data & analytics could keep up with their rapid pace of growth, they needed a more powerful system for discovering their relevant datasets. This led to the creation of Databook and its metadata curation.

The Coming of Databook

Die Databook-Plattform verwaltet umfangreiche Metadaten über die Datensätze von Uber und ermöglicht es Mitarbeitern im gesamten Unternehmen, ihre Daten zu durchsuchen, aufzufinden und effizient zu nutzen. Die Plattform stellt auch sicher, dass der Kontext der Daten nicht unter Hunderttausenden von Menschen verloren geht, die versuchen, sie zu analysieren. Alles in allem ermöglichen Databook Metadaten allen Ingenieuren, Datenwissenschaftlern und IT-Teams, ihre Daten nicht nur zu visualisieren, sondern sie in verwertbares Wissen zu verwandeln

Databook enables employees to leverage automated metadata in order to collect a wide variety of frequently refreshed metadata. It provides a wide variety of metadata from Hive, MySQL, Cassandra and other internal storage systems.To make them accessible and searchable, Databook offers its consumers a user interface with a Google search engine or its RESTful API.

Databook’s Architecture

Databook’s architecture is broken down into three parts: how the metadata is collected, stored, and how its data is surfaced.

Conceptually, the Databook architecture was designed to enable four key capabilities:

• Extensibility: New metadata, storage, and entities are easy to add.
• Accessibility: Services can access all metadata programmatically.
• Scalability: Unterstützung von geschäftlichen Nutzern und technologischen Neuerungen.
• Execution: Power & speed.

To go further on Databook’s architecture, please read their article https://eng.uber.com/databook/

What’s Next for Databook?

With Databook, metadata at Uber is now more useful than ever.

But they still hope to develop other functionalities such as the abilities to generate data insights with machine learning models and create advanced issue detection, prevention, and mitigation mechanisms.

Sources

• Databook: Turning Big Data into Knowledge With Metadata at Uber
• The Story of Uber
• The Definition of Uberization, Cambridge Dictionary

About Actian Corporation

Actian empowers enterprises to confidently manage and govern data at scale, streamlining complex data environments and accelerating the delivery of AI-ready data. The Actian data intelligence approach combines data discovery, metadata management, and federated governance to enable smarter data usage and enhance compliance. With intuitive self-service capabilities, business and technical users can find, understand, and trust data assets across cloud, hybrid, and on-premises environments. Actian delivers flexible data management solutions to 42 million users at Fortune 100 companies and other enterprises worldwide, while maintaining a 95% customer satisfaction score.