Lacs de données, entrepôts de données, centres de données : en avons-nous besoin ?
There’s a long-standing debate, dating back to the early days of Hadoop, about what kind of data repository is best for a given data analytics use case. A data lake? A data hub? A data warehouse? Despite Hadoop’s fall from grace, the debate not only persists but grows more complicated. Today’s cloud-based repositories, including AWS S3, Microsoft Azure ADLS, and Google Cloud Storage, look very much like data lakes in the cloud. Similarly, cloud-based offerings like Snowflake look very much like enterprise data warehouses but in the cloud. Granted, for an apples-to-apples comparison for data lakes you’d need to pare down Hadoop to be just HDFS or add in the tools for data repository management, query, and so forth associated with the three public cloud providers’ portfolios.
At the same time, it should be noted that none of the vendors promoting these offerings are using those terms. . . Microsoft, Amazon, and Google identify their cloud repositories as “enterprise data hubs.” Snowflake positions itself as a cloud data warehouse but is pivoting to call itself a cloud data platform via its expansive ecosystem; but, standalone, it is really an “analytics engine.”
La vérité est qu'aucun lac de données, hub de données ou entrepôt de données - sur site ou dans le cloud - n'a jamais été en mesure de support efficacement toutes les équipes multidisciplinaires d'analystes commerciaux, d'ingénieurs de données, de data scientists et d'utilisateurs chevronnés au sein des différents secteurs d'activité. C'était évident avant l'existence du cloud, et cela ne fait que s'accentuer à mesure que les équipes tentent d'intégrer de nouveaux ensembles de données (pensez aux services web et à l'IoT) et de fusionner des données semi-structurées dans des référentiels structurés. Ne me parlez même pas du flux de feuilles de données Excel qui était censé disparaître (mais qui n'a jamais disparu) lorsque nous sommes devenus plus sophistiqués en matière d'analyse et de gestion des données.
But here’s the thing: There are real differences between these platforms, and it’s important to understand those differences. In the end, though — watch for it — the operational differences between these platforms aren’t the root cause of why they’re not providing the support that all the different stakeholders expect.
Réglage du niveau
Commençons par parler de ce dont nous parlons réellement :
Centre de données
Nous définirons ici un hub de données comme une passerelle par laquelle des données virtuelles ou physiques peuvent être fusionnées, transformées et mises en file d'attente pour être acheminées vers une autre destination. Cette destination peut être une application, une base de données ou un autre type de dépôt (comme un lac de données ou un entrepôt de données). En tout état de cause, les données d'un hub de données sont transitoires ; elles ne sont pas stockées localement et n'ont pas de persistance.
An example of a data hub would be something like Informatica, which can accommodate every imaginable data type and link both upstream and downstream data sources and destinations. Historically, data hubs have been managed and used by IT personnel who work with separate siloed groups from across the enterprise to create integrations where none naturally existed.
Lac de données
Unlike a data hub, a data lake acts as a repository for persistent data. It is not simply a pass-through. Data lakes can typically ingest and manage almost any type of data and, as exemplified by Hadoop (historically the most popular type of data lake), they provide tools for enriching, querying, and analyzing the data they hold. The problem is that Data Lakes are generally sandboxes for dumping large sets of data used in experimental projects by highly skilled technical resources, largely IT and developers.
Entrepôt de données
A data warehouse differs from a data lake in that it acts as a repository for persistent and primarily structured data, incrementally built over time from multiple downstream data source silos. A data warehouse also differs from a data lake in that it requires some sort of data hub technology to prepare the data for ingestion. On-premise data warehouses such as those from the big legacy players like Oracle, IBM, and Teradata are very IT-centric, managed by one or more database administrators (DBAs). While the bulk of data used by business users may ultimately reside in a data warehouse, most of these users have no direct interaction with the data warehouse and may not even know they have one or what it is.
Le caoutchouc virtuel rencontre la route virtuelle
Historically, data hubs, data lakes, and data warehouses all have several things in common: They each require personnel with specialized skills to set them up, maintain them, and manage them. Experts who can convert the requests of non-technical business users and analysts into queries and reports that can be run against these data repositories.
Soit dit en passant, la complexité de ces plateformes est l'une des raisons de la disparition de Hadoop. Les lacs de données Hadoop avaient tendance à devenir des décharges de données, et ils ne pouvaient être gérés que par des développeurs et du personnel informatique très qualifié (et coûteux), ce qui limitait la valeur commerciale qu'un lac de données Hadoop pouvait générer. Il n'est donc pas surprenant que, des trois grands fournisseurs qui soutenaient Hadoop, seul Cloudera reste le dernier "homme" debout.
This need for specialized resources has affected the use of data hubs, data lakes, and data warehouses in other ways as well, and this in turn has further complicated the original question about which platform is best for different use cases. With the move from on-premises to cloud-based infrastructures, there’s been a reduction in demand for all these specialized resources. More and more operational support has been provided by the cloud vendors, which has helped to reduce operating costs. Moreover, the architectural changes in the most recent generations of cloud offerings (separate compute and storage offerings, pay for what you use, etc.) have created further incentives to move to the cloud to reduce costs.
Accroître encore la complexité
While all these structural changes have been taking place, though, the fundamental demand for data-based insights has not changed. The answer to the question about how best to gain these insights has only become more difficult to answer. The data that used to go into on-prem data lakes or data warehouses (via data hubs) is going to the cloud, but the offerings in the cloud are not quite the same as they were on-prem. Their object storage models differ. Microsoft, Amazon, and Google offer persistent data stores and, in that way, may resemble a data lake, but they rely on other tools to perform the data hub functions and cannot therefore be defined as anything more than data stores. They still require data integration or data hub functionality, and their business value is limited in the same way it always has been. The people who directly generate business value — the business analysts, data scientists, and (for lack of a specific title), the other line of business power users — still cannot easily access and unlock the insights bound up in the data.
These days, most business analysts and power users are using either the built-in analytics and visualization capabilities of siloed applications like Salesforce, Marketo, or whatever ERP platform they need to understand in terms of business operations or historical outcomes. At the same time, they strive to do more. Business users may try to incorporate data from flat files such as Excel or semi-structured JSON data exposed through web services APIs. Oftentimes, they will get help from IT to export data out of one or more systems and combine it with Excel spreadsheets and send it to a cube periodically. The result is painfully familiar: siloed data pipelines tied to siloed analytics and visualization results. Unbeknownst to these business users, when they employ help from IT, they may actually be leveraging a data hub, because there’s no data persistence in the hub; they’ve simply used the hub as a switch to tie a set of data silos and an analytics silo together to create an ad hoc organizational or project silo.
Data scientists and data engineers may be using many of the same data silos, but they may also be using data from semi-structured data sets such as clickstreams, IoT, and web services, and their destinations may include the same visualization tools but, of course, also include advanced analytics tools to support AI/ML. They may employ IT to support getting the data for them and, in turn, create the same point-to-point spaghetti network.
En d'autres termes, le dépôt unique et partagé de données promis par les lacs de données, les entrepôts de données et les centres de données reste un rêve irréalisable. Un véritable centre d'analyse n'a pas encore vu le jour - ni sur site, ni dans le nuage.
Et pourtant.
Changement d'orientation
Les fournisseurs d'informatique en nuage commencent à prendre conscience du problème et certains s'efforcent rapidement de le résoudre. Cependant, la plupart d'entre eux le font en s'assurant qu'un entrepôt de données cloud peut agir comme un dépôt données en amont pour tout outil d'analyse, de reporting et de visualisation en aval. Souvent, cela se fait par le biais d'un écosystème de partenaires, comme dans le cas de Snowflake. Cela est nécessaire mais insuffisant pour créer le centre d'analyse dont nous avons tous réellement besoin.
Mais attendez. Un centre d'analyse ? Où était-ce dans les définitions ci-dessus ?
Fact of the matter is that the cloud data warehouse is currently an analytics engine but without a data hub built-in on the back end and a focus on separate point-to-point connections to various BI and analytics tools on the front-end. Vendors like Snowflake do not mention analytics hubs, let alone claim to be one. Further, without the ability to easily get data from data sources and tie composite elements of data from those various sources for presentation out to the analytics tools, you don’t really have an analytics hub, chiefly because you don’t have a data hub.
Instead of just a data hub or analytics hub, both usable only by IT, what’s really needed is a data analytics hub that is used by a broad array of IT and business users. More on what this is and why it matters in the next blog.
Restez connecté
Des informations tirées des données, à votre service.
(par exemple : sales@..., support)
