Data Lakes, Data Warehouses, Data Hubs - Brauchen wir das?
There’s a long-standing debate, dating back to the early days of Hadoop, about what kind of data repository is best for a given data analytics use case. A data lake? A data hub? A data warehouse? Despite Hadoop’s fall from grace, the debate not only persists but grows more complicated. Today’s cloud-based repositories, including AWS S3, Microsoft Azure ADLS, and Google Cloud Storage, look very much like data lakes in the cloud. Similarly, cloud-based offerings like Snowflake look very much like enterprise data warehouses but in the cloud. Granted, for an apples-to-apples comparison for data lakes you’d need to pare down Hadoop to be just HDFS or add in the tools for data repository management, query, and so forth associated with the three public cloud providers’ portfolios.
At the same time, it should be noted that none of the vendors promoting these offerings are using those terms. . . Microsoft, Amazon, and Google identify their cloud repositories as “enterprise data hubs.” Snowflake positions itself as a cloud data warehouse but is pivoting to call itself a cloud data platform via its expansive ecosystem; but, standalone, it is really an “analytics engine.”
Die Änderung des Begriffs ändert jedoch nichts an der Frage, die der Debatte zugrunde liegt, und die einfache Wahrheit ist, dass keine Daten-LakeDie einfache Wahrheit ist, dass kein Daten-Lake, keine Datendrehscheibe und kein Data Warehouse - ob On-Premises oder in der Cloud - jemals in der Lage war, alle multidisziplinären Teams aus Geschäftsanalysten, Dateningenieuren, Datenwissenschaftlern und Power-Usern in verschiedenen Geschäftsbereichen effektiv zu unterstützen. Das war schon so, bevor es die Cloud gab, und es wird immer deutlicher, wenn Teams versuchen, neue Datensätze zu integrieren (z. B. Webservices und IoT) und halbstrukturierte Daten in strukturierte Repositories zu überführen. Lassen Sie mich gar nicht erst mit der Flut von Excel-Datenblättern anfangen, die verschwinden sollten (aber nie verschwunden sind), als wir in Sachen Analyse und Datenmanagement anspruchsvoller wurden.
But here’s the thing: There are real differences between these platforms, and it’s important to understand those differences. In the end, though — watch for it — the operational differences between these platforms aren’t the root cause of why they’re not providing the support that all the different stakeholders expect.
Level-Einstellung
Lassen Sie uns zunächst darüber reden, worüber wir eigentlich reden:
Datendrehscheibe
Hier definieren wir einen Daten-Hub als ein Gateway, über das virtuelle oder physische Daten zusammengeführt, umgewandelt und für die Weiterleitung an ein anderes Ziel in eine Warteschlange gestellt werden können. Bei diesem Ziel kann es sich um eine Anwendung, eine Datenbank oder eine andere Art von Lager handeln (z. B. ein Daten-Lake oder Data Warehouse). In jedem Fall sind die Daten in einer Datendrehscheibe flüchtig; sie werden nicht lokal gespeichert und haben keine Persistenz.
An example of a data hub would be something like Informatica, which can accommodate every imaginable data type and link both upstream and downstream data sources and destinations. Historically, data hubs have been managed and used by IT personnel who work with separate siloed groups from across the enterprise to create integrations where none naturally existed.
Daten-Lake
Unlike a data hub, a data lake acts as a repository for persistent data. It is not simply a pass-through. Data lakes can typically ingest and manage almost any type of data and, as exemplified by Hadoop (historically the most popular type of data lake), they provide tools for enriching, querying, and analyzing the data they hold. The problem is that Data Lakes are generally sandboxes for dumping large sets of data used in experimental projects by highly skilled technical resources, largely IT and developers.
Data Warehouse
A data warehouse differs from a data lake in that it acts as a repository for persistent and primarily structured data, incrementally built over time from multiple downstream data source silos. A data warehouse also differs from a data lake in that it requires some sort of data hub technology to prepare the data for ingestion. On-premise data warehouses such as those from the big legacy players like Oracle, IBM, and Teradata are very IT-centric, managed by one or more database administrators (DBAs). While the bulk of data used by business users may ultimately reside in a data warehouse, most of these users have no direct interaction with the data warehouse and may not even know they have one or what it is.
Virtueller Gummi trifft auf virtuelle Straße
Historically, data hubs, data lakes, and data warehouses all have several things in common: They each require personnel with specialized skills to set them up, maintain them, and manage them. Experts who can convert the requests of non-technical business users and analysts into queries and reports that can be run against these data repositories.
As an aside, the complexity of these platforms is one reason for the demise of Hadoop. Hadoop data lakes tended to become dumping grounds for data, and they were only manageable by developers and very skilled (and costly) IT personnel, which limited the business value a Hadoop data lake could generate. It’s not entirely surprising that, as a result, of the big three vendors formerly supporting Hadoop, only Cloudera remains the last “man” standing.
This need for specialized resources has affected the use of data hubs, data lakes, and data warehouses in other ways as well, and this in turn has further complicated the original question about which platform is best for different use cases. With the move from on-premises to cloud-based infrastructures, there’s been a reduction in demand for all these specialized resources. More and more operational support has been provided by the cloud vendors, which has helped to reduce operating costs. Moreover, the architectural changes in the most recent generations of cloud offerings (separate compute and storage offerings, pay for what you use, etc.) have created further incentives to move to the cloud to reduce costs.
Die Komplexität nimmt weiter zu
While all these structural changes have been taking place, though, the fundamental demand for data-based insights has not changed. The answer to the question about how best to gain these insights has only become more difficult to answer. The data that used to go into on-prem data lakes or data warehouses (via data hubs) is going to the cloud, but the offerings in the cloud are not quite the same as they were on-prem. Their object storage models differ. Microsoft, Amazon, and Google offer persistent data stores and, in that way, may resemble a data lake, but they rely on other tools to perform the data hub functions and cannot therefore be defined as anything more than data stores. They still require data integration or data hub functionality, and their business value is limited in the same way it always has been. The people who directly generate business value — the business analysts, data scientists, and (for lack of a specific title), the other line of business power users — still cannot easily access and unlock the insights bound up in the data.
These days, most business analysts and power users are using either the built-in analytics and visualization capabilities of siloed applications like Salesforce, Marketo, or whatever ERP platform they need to understand in terms of business operations or historical outcomes. At the same time, they strive to do more. Business users may try to incorporate data from flat files such as Excel or semi-structured JSON data exposed through web services APIs. Oftentimes, they will get help from IT to export data out of one or more systems and combine it with Excel spreadsheets and send it to a cube periodically. The result is painfully familiar: siloed data pipelines tied to siloed analytics and visualization results. Unbeknownst to these business users, when they employ help from IT, they may actually be leveraging a data hub, because there’s no data persistence in the hub; they’ve simply used the hub as a switch to tie a set of data silos and an analytics silo together to create an ad hoc organizational or project silo.
Data scientists and data engineers may be using many of the same data silos, but they may also be using data from semi-structured data sets such as clickstreams, IoT, and web services, and their destinations may include the same visualization tools but, of course, also include advanced analytics tools to support AI/ML. They may employ IT to support getting the data for them and, in turn, create the same point-to-point spaghetti network.
Anders ausgedrückt: Das von Data Lakes, Data Warehouses und Data Hubs versprochene gemeinsame Lager ist immer noch ein unerfüllter Traum. Ein echter Analytik-Hub muss erst noch realisiert werden - nicht vor Ort und nicht in der Cloud.
Dennoch.
Verlagerung des Schwerpunkts
Die Cloud beginnen, das Problem zu erkennen, und einige gehen schnell dazu über, es zu lösen. Die meisten von ihnen tun dies jedoch, indem sie sicherstellen, dass ein Cloud als vorgelagertes Lager für nachgelagerte Analyse-, Berichts- und Visualisierungstools fungieren kann. Oft wird dies über ein Partner-Ökosystem versucht, wie bei Snowflake. Dies ist notwendig, aber unzureichend für die Analysedrehscheibe, die wir alle wirklich brauchen.
Aber halt. Ein Analytics Hub? Wo war das in den obigen Definitionen?
Fact of the matter is that the cloud data warehouse is currently an analytics engine but without a data hub built-in on the back end and a focus on separate point-to-point connections to various BI and analytics tools on the front-end. Vendors like Snowflake do not mention analytics hubs, let alone claim to be one. Further, without the ability to easily get data from data sources and tie composite elements of data from those various sources for presentation out to the analytics tools, you don’t really have an analytics hub, chiefly because you don’t have a data hub.
Instead of just a data hub or analytics hub, both usable only by IT, what’s really needed is a data analytics hub that is used by a broad array of IT and business users. More on what this is and why it matters in the next blog.
Bleiben Sie in Verbindung
Datenanalysen direkt bei Ihnen.
(z. B. sales@..., support@...)
