Data Hub Reviews

My main use case for Data Hub involves connecting a lot of data that is available and coming from upstream data points or data lakes like Kafka, or in BigQuery itself. We usually connect this data to Data Hub as it is a modern data catalog designed to streamline metadata management. We can put all the metadata of our data inside Data Hub, setting who the owner is and tracking where this data is coming from and where it is consumed downstream. We can have data discovery and governance as well.

My specific example of using Data Hub in my daily workflow involves an orders table, which is very large and is joined with several other tables. This data is populated by a Kafka consumer that consumes messages from a specific topic, and thereafter, a batch that runs once a day transfers this data to a history table in BigQuery. This allows us to manage visualizations and data management tasks. We usually put all this metadata in Data Hub to track the data lineage, profile datasets, and establish data contracts. This way, we know the lineage of each field, and if any batch fails the data contract check, it sends an email notification to the responsible person. We can add more contracts such as validations to the data as necessary.

The best features Data Hub offers include its integration capability with many popular tools like Apache Airflow, Snowflake, dbt, Looker, Apache Kafka, and BigQuery. These tools provide us with data in various places, and we commonly use Apache Airflow for the DAG, while utilizing BigQuery as our database and Apache Kafka for consuming messaging queues. Data Hub easily connects with all these tools and features excellent data discovery and visualization capabilities. We can see data visibility, where it comes from, its upstream and downstream relationships. If we remove a column, we can assess the impact of that change. Furthermore, if there are duplicate datasets being used by different teams that do not communicate regularly, onboarding all data to Data Hub allows us to identify these duplicates easily.

Out of all those features, I believe data discovery and impact analysis are the most valuable for my team because when we want to add or drop a column, we can assess the impact analysis to understand the downstream effects. This helps us know who owns a dataset, and we can easily contact the owner. Tracking the data lineage back to the source table is also a key benefit.

Data Hub has positively impacted my organization by significantly reducing manual work that was previously needed to identify upstream and downstream data relationships, as well as recognizing duplicate datasets. If a data contract is broken, we now easily get notified of those issues, making the process much easier and more efficient. It is particularly useful for data engineers and platform teams to check for problems directly within Data Hub.

Data Hub has saved our team a lot of time. For example, in a large company like Porch, if I want to know whether a specific dataset exists, I can check Data Hub, as it serves as a centralized point for managing the metadata of our data. While it does not contain all data, it does contain the metadata necessary for understanding the dataset's origin. If a dataset does not exist, I can simply see who the owner is and reach out to them, which reduces the dependency on others by providing direct access to information in Data Hub.

Regarding improvements for Data Hub, I think there is no scope for improvement. It is the best tool in the market currently. I have reviewed some other tools as well, but Data Hub stands out.

In terms of areas for improvement, I do not see anything lacking. Data Hub offers both cloud and self-hosted deployment options, and it has a robust community. They hold open Slack community sessions as well as webinars, typically once or twice a month, to share knowledge and updates, which is a significant benefit. I have not encountered any major issues with Data Hub.

I have been using Data Hub for around three years.

I have not seen any downtime within Data Hub.

In my experience, Data Hub's scalability is impressive. We have around 300 datasets from BigQuery, 400 from Kafka, and many more, yet I have not seen any downtime within Data Hub. We have successfully onboarded over 1000 datasets from various sources without any issues.

Customer support for Data Hub is very genuine, and they are responsive and attentive. If I raise a ticket today, they usually respond by the next day. Additionally, they host webinars monthly to discuss new features and updates. They also have an open Slack community where responses tend to be immediate.

I previously used OpenMetadata before adopting Data Hub, but I found Data Hub to be more user-friendly and easier to utilize than OpenMetadata.

Data Hub exceeds expectations in user-friendliness and functionality. It features a great user interface, an available SDK, APIs, and GraphQL previews, all complemented by a responsive Slack community and helpful customer support. The ease of documentation, website usability, and setup contributes to its overall effectiveness.

Additionally, we use some other data governance tools with Data Hub. We can add domains to any dataset, such as specifying that this is the orders domain or the customer domain. We can add more tags, manage data ownership by indicating which team owns specific data, and create glossary terms, which act as labels for different datasets.

I find myself relying on Data Hub for lineage checks and data contracts once a week.

Regarding Data Hub's AI capabilities, it exposes several MCP servers that easily integrate with LLMs such as Claude, Cursor, Gemini, or LangChain, along with the Agent Development Kit from Google. In terms of security, Data Hub ensures that no company data is exposed outside, and they maintain strict confidentiality regarding the metadata of the company, adhering to similar NDAs that prevent revealing sensitive information.

In terms of accuracy and reliability of output with Data Hub's AI capabilities, I find it exceeds 95% accuracy. Having utilized the MCP connectors with Claude and the ADK, I can confidently say that it performs flawlessly and retrieves data effectively.

My advice for others considering the use of Data Hub is to add more glossary labels and categorize datasets by domain. While it is manageable with a smaller dataset, as the amount of data scales, these glossary terms and domains become immensely helpful. Initially, we did not leverage them, but we found their value as we scaled up and needed to filter data efficiently. I would rate Data Hub a perfect 10 overall.

My main use case for Data Hub is to build data products inside Natura; primarily, I built data products for CRM, which is customer relationship management, and also for some data products for the product field, such as analytic fields. I used it by dividing the company into domains, and each domain has its own functionality and its own structure. With that approach, I used it extensively for building domains. I also used it to build data lineage across the entire data journey, from the ingestion of the data to the use of the data in the final part, such as in a data product or in a dashboard.

A specific example of how I used Data Hub for building domains and data lineage is the domain called GenAI, which is primarily built for products based on AI, mainly generative AI. To accomplish this, I used Data Hub to track the data from the ingestion field. I used some CDP tools such as Segment and I also have data in an S3 bucket that was ingested to Databricks using Airflow. With that setup, I track this lineage from the origin system. After that, I performed a lot of transformation of the data inside Databricks to clean the data and conduct some data augmentation. After that, this data is used to train some models using Databricks LLM. With that, I ingest all this metadata into Data Hub and I can see from where the data is coming from and to where the data is going. This is primarily for LLMs to help consultants at the end of the product.

The best features that Data Hub offers include the capability to make conversational questions inside the platform, which I believe is the best thing that they built in the past year. It is also easy to connect different data sources. Since data lakes, I have connectors to some databases and also to some business analyst tools and other tools. I can connect many types of data inside Data Hub and see what is going on and how we govern the data. Data Hub is a pretty good tool for that. I also value very much the open-source version because it is free and everyone can use it.

I do not have much experience using the conversational questions feature, but I do not need to go to the asset to see from where the data is coming from and where it is going. I can simply ask, 'How can we calculate the sales in this month?' and Data Hub will identify which table will be used and from where this data is coming from and where this data is going. This is very effective.

Data Hub has impacted my organization positively by helping us build a data governance environment and share the knowledge about the data for the entire company. As we used the open-source version, we have no limitation in how many people can use the tool, which is excellent. I can conduct many tests and test as quickly as possible. It is very good for building POCs, for example. Data Hub also helps to give this understanding of the data for the entire company. Everyone in the company can see the data and know where the data is coming from and where it is going. I believe it is very effective and all the people in the organization, not only the data field personnel, can understand more about the data and also help to build better products.

I believe Data Hub could provide more functionalities in the free version. I understand that we have to pay the persons who build the platform, but the free version has some limitations. Some capabilities of the paid version being included in the free version would be beneficial. Another improvement that is needed in Data Hub is how I can get data from Data Hub to build some metrics. I know that I have the API and the GraphQL API, but I believe it could be better. If this is improved, it would be very helpful.

I used Data Hub for over two years, mostly in the open-source version.

I do not have actual metrics to provide currently; I only have some metrics. I certainly improved the data discovery part of building a data product because it is really fast to know if the data product already exists or does not exist. In the past, I had many products that were the same, and with that, I had a lot of work doing it twice or three times in different parts of the process. This is very good. I do not actually know the exact number of time saved, but I certainly saved time. I have a metric that before Data Hub, I believe 20 to 30 persons used and had knowledge about the data. Currently, I have almost 250 persons using Data Hub.

I did not use many AI features in Data Hub, as I stopped using Data Hub before it started offering these functionalities.

My advice to others looking into using Data Hub is to start by trying the tool using the free version. If it is sufficient and you already understand how to use it, you can transition to the paid version. However, you can accomplish everything in the free version. I would rate this review an eight out of ten.

I work with Data Hub as a user, but I also have some administrative responsibilities there. I'm not a final user; the final users are business users, and I play some administrative roles in the tool to have the metadata information available for all Uber users.

I'm a Data Quality Engineer focused on data governance. I manage the metadata information for Uber, and I also use this to apply some data quality rules. My focus in my current job is to apply some rules and manage the metadata information and ensure it is accurate for the end users, which is why I'm using it.

One of the biggest advantages of Data Hub is the very good integration, for example, a department focused on development made the integrations between Data Hub and BigQuery. When this integration is very well done, it is possible to check data lineage, which I think is a very important subject in data governance. It's something that cannot be done manually, so having a tool that shows the data lineage from the source until the target tables helps us a lot. I think this is one of the best advantages that we have.

Data Hub helps to analyze data from various sources in my case.

I know that the integrations are not easy to do, and I believe it happens because it's a customized solution. There always needs to be software developers to work on this. It's complicated; every time we want to integrate new things or new sources, we need to generate a ticket or a request to another department. When I had my experience with Atlan, for example, I was able to connect different sources in a very user-friendly way. I just needed to set up some configurations and connect to the source without having to be a software developer or develop any code in the back end. It was just a feature in the data catalog that enabled me to connect with different kinds of sources. That's why I think the disadvantage of having a customized solution. Although I think Data Hub itself is a very good tool, years ago I had the opportunity to work with it, but with a clear interface and the open-source solution, which was very clear and easy to connect. At Uber, we need to have a request when we want to integrate new sources.

Regarding Data Hub's intuitiveness, regarding analytics, I would say that some quality dimensions are available for us. For example, for each field name or each column in a table, it's possible to see the frequency, how many values we have for a specific type or category, and we can see if there are new or null values, whether the columns are empty or not, along with some metrics. This is regarding the data quality dimensions, such as nullables and things of that nature. That is all we have for features. I remember when I was working with Atlan, there was a feature I liked very much—the possibility to have a sample. When I clicked on a table, I could see a short sample without needing SQL skills. I just clicked the table and could see some values or what the table represents; the data catalog would show a screen with some rows of the table. This feature was very good, but we don't have it in Data Hub the way it is implemented at Uber. I think it would be a very good feature for analytics, and we don't have it at the moment.

The integration part could be better, but again, it's because it's a customized solution. I think if they used the native version of the tool, it would be simpler. The integration part and the process of setting up new data quality rules would be important for data governance players like me.

I've been using Data Hub for one year and a half.

Since I've been using Data Hub, it has always been very stable; I can say it was one hundred percent stable. I never encountered issues trying to check datasets or columns and checking their numbers. It has always worked very well in that regard.

I think Data Hub can scale fast in its native way, but with a customized solution, it takes more time.

My support is internal when I have any questions or requests, so I direct it to a support team from Uber and not from the provider. When I was working with Atlan, and needed support, they were very good at attending to my requests directly. I had contact with the provider, so it was very fast. At the moment, I don't have that; I direct my requests to an internal department of Uber.

I'm not using Atlan anymore because the company that I was working with, I'm no longer there. I went to another consultancy group and now I'm working with other platforms. Atlan is not the one that I'm working with at the moment.

I am working with a different platform that is also regarding data governance and metadata management. The platform itself, the back end, is Data Hub. But the user interface is customized for this client. I'm currently working for Uber, the Uber company.

Because Data Hub is a customized solution, I don't have many details about the installation and deployment process. However, when I was using Atlan, I saw that they implemented very fast. In this way, I believe both tools have an easy way to implement, but because Uber chose to have a customized solution, it became more difficult and complex. However, in their native way, I think both tools are good.

In terms of ROI, I would say that Atlan is better. I had a very good experience using Atlan, and I believe it's faster. Velocity in organizations today is very important; people want to see things very fast. I believe Atlan has a better approach compared to Data Hub.

The way Data Hub is implemented at the moment, Atlan is much better. It's much, much faster.

I worked with Databricks, but I'm not sure if it is from Amazon; I don't think so. I think Databricks is from Microsoft.

I have experience with Data Hub to some extent.

I believe Data Hub uses a lot of APIs, but I don't think I'm the right person to answer that because it relies a lot on a technical aspect that I don't understand. I cannot provide you with a curated answer about it, but I know that the software development team that works with this customized solution uses APIs; I just don't know how to speak about their performance, whether it's good or not.

Real-time batch processing is very important for me and my organization because some datasets are very critical for the business. If we have batch processing, it's good for the organization to set up a very large dataset, for example, and have it available on the data catalog in a short time. I agree that this is important.

In both experiences I had, the integration with the catalog was with GCP. I don't have experience working with another data warehouse, so even in Atlan or now in Data Hub, it is connected with GCP.

I don't use anything else like CRM, storage, or any architecture management tools; just Data Hub.

I would give Data Hub a score of seven out of ten, summarizing everything that I've discussed about the product.

Our main use case for Data Hub is for data lineage and metadata governance for our UFC project, where we are utilizing multiple databases such as SQL Server, Databricks, and Snowflake. We have adopted Data Hub to create centralized metadata for all these databases.

A specific example of how we use Data Hub for metadata governance in our UFC project involves getting data from multiple sources including Excel files, CSV files, APIs, and external databases, storing that data first into Amazon S3 buckets, and then into Snowflake staging areas. We transform the raw data using a DVT model, create a silver layer, and then load data into the gold layer for reporting. With Data Hub, we have a centralized view of the data flow, which makes it easier to track issues in downstream applications such as Power BI reporting.

We also use Data Hub for onboarding new team members, as it was previously hectic to provide complete metadata details from our seven to eight data sources and over two hundred tables in our Snowflake database. Now, new team members can refer to the lineage of any table or column to understand the complete flow without relying solely on others.

One of the best features Data Hub offers is its ability to identify schema changes in the source side efficiently, especially when we pull data from multiple external databases such as SQL Server. It helps us quickly pinpoint necessary updates when columns are added or removed, streamlining what was previously a time-consuming manual process.

I find Data Hub quite manageable in the downstream application within the UFC data mart, mainly when issues are reported in Power BI. It provides a complete view of the data lineage, allowing us to backtrace the source of any discrepancies easily.

Data Hub has positively impacted our organization by reducing the knowledge transition period from three months to one month for new team members, enabling them to refer to the complete lineage without depending heavily on others, which is a substantial improvement.

In terms of improvements for Data Hub, it seems more useful for critical or large data pipelines, as small data architectures can be straightforward to understand without it.

Regarding enhancements for complex projects, I have noticed that sometimes Data Hub does not provide a complete picture of the lineage, particularly in complex data pipelines such as when we fetch data from an API to S3 and subsequently to Snowflake. We have to review the metadata in Data Hub closely.

I am working in the data engineering field for over twelve years.

Data Hub is stable in my experience.

Data Hub's scalability is advantageous, as we onboard data from over one hundred fifty tables in SQL Server to Snowflake, and adding new tables to Data Hub is not time-consuming.

Customer support for Data Hub is quite good; our infrastructure team received ample support during the initial setup within the given timelines.

Previously, we used the Snowflake inbuilt lineage graph to identify data flow, but we switched to Data Hub for its centralized governance capabilities across multiple databases.

The initial setup of Data Hub was completed by our infrastructure team, and I do not have complete visibility of how they made the purchase.

Regarding pricing, setup cost, and licensing for Data Hub, it was handled by our client infrastructure team, so I lack visibility into those aspects.

I have seen a return on investment with Data Hub, notably in reducing the knowledge transition period and improving our ability to troubleshoot production issues in Power BI, thus saving time.

We did not evaluate other options before choosing Data Hub since we were solely relying on the lineage functionalities of Databricks and Snowflake.

My advice for others considering Data Hub is to utilize it, as it is free and can significantly reduce time for production support and addressing data issues, while simpler data models can benefit from the inbuilt functionalities of their respective databases. I would rate this product eight point five out of ten.

We adopted Data Hub in the context of a large enterprise customer operating in a regulated industry with a strong focus on data governance, data discoverability, and ownership clarity across multiple cloud-native platforms. The solution was deployed on AWS, and the main business problem was the lack of a centralized, reliable view of data assets, including poor data discoverability, unclear data ownership and stewardship, limited lineage visibility across ingestion and transformation layers, and high dependency on tribal knowledge held by a few individuals. Data Hub was selected as an enterprise data catalog and metadata backbone with the goal of enabling both technical teams and business users to easily understand, trust, and reuse data.

We used Data Hub to create very good data discoverability, assign data ownership and stewardship, improve data quality processes, and establish good data governance for our customer in terms of data catalog, data lineage, and metadata management in general.

Our key benefits that we achieved include centralized metadata management across multiple AWS services and data platforms and improved data discoverability, significantly reducing the time required to find relevant data sets. Clear data ownership and stewardship improved accountability and collaboration between teams. End-to-end lineage visibility enabled faster impact analysis and safer changes, and faster onboarding of new data users through self-service access to documentation and metadata. From a governance perspective, Data Hub became a single source of truth for metadata, supporting both compliance requirements that are very important in a data governance environment and day-to-day operational needs.

The main strengths we experienced with Data Hub are a strong metadata model and its extensibility because Data Hub offers a rich and flexible metadata model that adapts well to complex enterprise scenarios. Excellent lineage capabilities are provided because the lineage visualization is clear, actionable, and extremely useful for impact analysis and governance workflow. The open source foundation with enterprise readiness is significant because the open architecture avoids vendor lock-in while still being suitable for production-grade environments.

Data Hub is very effective for us because we build the data lineage from the beginning, from origination to visualization, to the final use of the data. We follow and track a path of the data, which improves analysis and enables us to find where data is used and the impact of deleting data. This is also very important in a regulatory environment.

The impact is very positive, and there are many benefits for us using Data Hub because it was easier to make data governance, create centralized metadata management, improve data discoverability, and manage data in general. The areas for improvement, in my opinion, are the initial setup and configuration that can be complex without prior experience, especially in large-scale environments. User experience for non-technical users could be further simplified, particularly around advanced metadata concepts. The out-of-the-box governance workflow, for example, approvals and certification, could be more prescriptive for customers at early maturity stages.

Data Hub can be improved in the initial setup and configuration that is somewhat complex, and also in operational monitoring that could benefit from more native dashboards and alerts. However, these are not blockers, but areas where additional guidance or product enhancement would further accelerate adoption.

I have been using Data Hub since 2023.

Based on internal measurement and feedback from the data teams, there are many impacts. Time to locate and understand a data set was reduced by approximately 40-50 percent. Manual documentation effort was reduced by around 40 percent. Dependency on senior data engineers for data explanation dropped significantly. Data onboarding time for new team members decreased from weeks to days.

I would rate this product a 9 out of 10. I chose nine because Data Hub proved to be a robust, scalable, enterprise-ready data catalog that is well-suited for AWS-based architecture and complex organizational environments. It is always possible to improve and useful to maintain space for further optimization.

My advice is to use Data Hub to move from fragmented metadata and manual processes to a modern, governed, and self-service data ecosystem, delivering clear value in terms of efficiency, cost saving, and data trust. We would confidently recommend Data Hub to organizations looking to improve data governance, data discovery, and metadata management on AWS.

My main use case for Data Hub is one of the programs at Société Générale to migrate from the existing Oracle database to Azure cloud architecture, migrating the entire agency business from legacy to the target architecture.

In terms of data flow on the old system, there were many constraints with respect to the new processes that had to be onboarded due to system constraints. There was a four-year fee in the millions just to maintain the services of that particular database. To eliminate all these limitations, a project team was created to migrate all the processes and flows from the existing Oracle database to the Azure cloud architecture. The advantages of Azure are that you only pay for what you use. Additionally, we can streamline the resources and virtual machines that need to be used for a particular process, and security comes built in. This is why we moved to the Azure architecture.

Our primary objective was to ensure that Data Hub would be useful for multiple consumers, meaning the data we produce can be used by multiple consumers who can build their own process flows and architectures. The basic and main data is fed by the respective upstream applications, which in Société Générale terminology we call deal processing applications. These particular applications publish their data, and along with this main data set, we enrich it with other referentials. Consumers receive the required data along with the main data through these referentials, allowing them to build their processes with this one particular data set rather than subscribing to multiple applications. This is how Data Hub use case is currently maintained.

Data Hub fits into the workflow very aptly at the moment. Several things have helped in migrating to Azure. The tables, architecture, flows, views, and procedures that need to be written on Oracle database are quite complex and bulky to maintain, whereas on Azure, they are simple and easy to maintain. Processes and everything can be designed, and the resources assigned for a particular process can be increased or decreased depending upon the load at that given point in time.

In terms of consumption, on Oracle there was a view or package that would take approximately 25 to 30 minutes to load and return output. With Azure Data Lake, we have built a utility called OPAL, which stands for Open Platform Access Library. With this utility, users need to pass only a few parameters and within one or two minutes, they receive all the required data associated with their input. Data retrieval is significantly faster, and we have seen data retrieval improve by over 40 percent migrating to Azure Data Lake.

The best features are that we do not need to use multiple tables to create our own flow. With one enriched data set that we are creating by combining the main data set with referentials, consumers can cater to their needs through one specific data set rather than having various tables at our disposal.

The second best feature is that security is built in here, whereas on Oracle, we had to manage all security-related aspects by maintaining the RDLs, restricting users based on their access levels. On Azure, security is based on the Azure RDLs that are already in place for a public Azure data lake account.

The third feature, which I consider one of the best, is OPAL, which I have already explained as Open Platform Access Library. This helps users retrieve data much faster.

The fourth advantage for the application creating this data set is that one of the new versions of OPAL, called OPAL++, helps the application audit which particular user is using this data and what kind of queries they pass throughout the day. With this, statistics can be built to see which user is utilizing the most resources and how they are utilizing them.

Data Hub has saved approximately two million every four years, which is one of the major savings. In terms of a second savings metric, more consumers have onboarded to the Azure Data Lake compared to those onboarded before, as the process is more streamlined and structured. The third savings is in terms of data on tables. Previously, tables were added as per user request, but on Azure, we have established an Enterprise Data Model. Any feeder that publishes data into Azure Data Lake must adhere to this particular model. This way we have ensured that normalization and standardization are properly taken care of.

In my day-to-day work, most consumers raise concerns with respect to the enrichment that we perform on top of the main data set provided by the respective feeding application. This is where most of the work lies for us because consumers come with multiple data-related concerns regarding data quality issues, data mapping issues, or completely missing data during the enrichment process. It is my day-to-day responsibility to ensure that no records are dropped after the main data set is enriched with referentials. On Azure, we use an architecture on Spark that builds this particular enrichment using executors and driver memory. Sometimes when it goes out of memory due to multiple jobs in progress, some records are dropped because the executor is dropped without completing the entire process. A final solution is being built to address this particular use case.

On Oracle database, when you needed to create a new attribute or when a specific user needed a particular value available for their reporting use case, whether for regulatory reporting, compliance, audit, or any kind of requirements, we would determine which would be the most appropriate table and try to add this attribute to that particular flow. In this scenario, only the application creating these tables and the user are involved. To ensure that the table does not become bulky and the data inside it is not overloading, such cases are not ensured at all and there is no normalization.

When it comes to the Enterprise Data Model, version one was created looking at all the users who currently require this particular data set. Later, when one or multiple users request the addition of a new attribute or addition of a new block of information altogether, a team called the Data Management Office becomes involved. They take care of adding this attribute to the given model, publish a new version, and ensure that the new version created is backward compatible so that downstream impacts are minimized.

Data Hub would achieve a perfect rating once all consumers migrate to the Azure database and we address the majority of concerns they have.

Since a public data lake is being used on Azure, resources are coming from Azure itself. As a result, we cannot publish any confidential information onto the Azure Data Lake; private spaces have been created for this purpose. Once the public data lake is resilient enough to any online threats or issues, I believe Data Hub using the public data lake will become one of the major use cases where the majority of consumers can adopt it for going forward, instead of using a private lake or the Oracle database, which is entirely private and not hosted on any public platform.

I have been using Data Hub for approximately four years.

It is quite reliable and on par with what is created using the Oracle database. I do not see any issues in terms of the accuracy and reliability of the output.

For considering the use case of Data Hub, I would specifically say that for constructing a data hub, you should have a variety of consumers. The first requirement is that there should be multiple consumers who would want to access this particular data hub because a data hub is a collection of entire data which can be used by various consumers. The use case is built to help consumers get whatever data they need in one specific place, rather than going to individual applications, subscribing to them, and getting the data they need. That is one of the advantages. The second advantage of Data Hub is that it is more secure and reliable compared to other database architectures in place. I would rate this product an eight out of ten.

My main use case for Data Hub is data lineage tracking. With Data Hub, we track multiple sources, ingestion sources, and different sources where the data resides in S3. We bring all that metadata into Data Hub to track lineage on the data ingestion patterns that we perform or transformations that we do, and how they move from different tables or assets or the data pipelines. Whatever transformations we do with Spark and S3, Snowflake, all those are being tracked via Data Hub. We have S3 buckets and Snowflake tables, and all those lineage tracking is managed through the platform.

My main use case is mostly covered as we used Data Hub for metadata tracking and lineage for whatever transformations that we do so that we can track each transformation down the line.

In my experience, the best features Data Hub offers include lineage tracking, which is mostly on the asset level, a good glossary, and good connector support.

Regarding asset level and the good glossary, we need the glossary of our products so that it is easy to track which product, what went at what time on that particular product, how many assets are related, and so on. For asset integrations, Data Hub makes it easy to ingest all that metadata of those particular assets from S3 via connectors, which is quite easy. It has good connector support, although limited in some cases.

Overall, Data Hub is a good tool. If we talk about lineage, metadata, and observability on some high level, including domain descriptions, PII classification, datasets, and keeping datasets in one place along with policies, it is good in that particular sense. We do have a plan based on project-to-project usage, but in some of the projects, we do use Data Hub as well.

I would like to add that for the connectors, there is sometimes limited support for using wildcards to get the items or assets ingested from sources like S3; it does not support very good wildcard filters. Additionally, Data Hub has a problem with column-level lineage support, especially regarding non-pro users or those without any plans. If I talk about the free features of Data Hub open source, those two I found could be improved during my use case.

Regarding improvements needed for Data Hub, I have already mentioned the limitations on the usage of wildcards in the ingestion or connectors; that can be worked upon, especially regarding the open-source part of Data Hub. The rest is that I hope the UI is quite good.

I used Data Hub for one and a half years.

My advice for others looking into using Data Hub is that it is a good tool if you want to capture all that metadata, lineage, keep track of governance, security, and observability. It just depends on how you want to use it; you can choose the open-source version or the paid version and subscription-based model. The paid versions have more features, but open-source Data Hub, which most people will try to go for, has some limitations, such as the missing column-level lineage with Spark. You need to consider those points, but overall, it is good. I would rate this product an 8 out of 10.

I use Data Hub for our data lineage, data management, data heritage, and our data dictionary. Our organization is quite large, with about 2,000 people working on different initiatives, and everyone wants to connect to a database somehow. As the data engineering team, we are responsible for connecting every single data source that we have, defining each one, and providing an accurate single source of truth for the data so everyone can have the same understanding of the data they are discussing. Since we are ingesting every database that the company has into our own data infrastructure through different tools, we needed to have a clear understanding of data quality, data lineage, and the data discovery part of the process.

The Helm chart of Data Hub is designed really well, which makes our deployment strategies lean and operational. The UI is excellent, and I really appreciated the ability to treat data gathering and data ingestion as GitHub workflows. Data Hub is one of the services that was truly scalable, at least in the open-source version, which is one of the things we valued since you could scale every part of the system it used, including its internal MySQL or metadata database, Elasticsearch, and the search capabilities. Everything about Data Hub was quite scalable due to its excellent Helm chart, as they really focused on the Kubernetes aspect.

We encountered some issues when we wanted to connect our streaming infrastructure to Data Hub, which was somewhat problematic.

In our data streaming infrastructure, we had a database CDC'd through Kafka Connect to a Kafka topic, and at the end of the pipeline, it would go to either an OLAP or a data lakehouse. However, the problem with visualizing this data lineage was that while the connection between MySQL and Kafka worked, when we wanted to track data from Kafka to other services, we couldn't track everything back because the IDs were generated randomly and couldn't be connected. We had to fix this manually by stating where the data had gone, which was tedious.

Data Hub's GMS service, or General Metadata Service, is a good service that I used regularly, but the CLI version had considerable changes across different versions. When I installed a different version, there wasn't enough consistency to ensure that commands I used would work in future versions of Data Hub's GMS CLI, which was frustrating. I also recall that setting up Kafka without Zookeeper was not possible, which was inconvenient, though I should verify this as I don't remember if they fixed it. At least from my recollection, when I wanted to set it up one and a half years ago, they did not have direct support for KRAFT in their Helm chart.

I have been using Data Hub for approximately one and a half years.

Before adopting Data Hub, we considered moving forward with OpenMetadata but decided against it since it couldn't support MySQL version 5.

The setup of Data Hub was quite straightforward. One aspect of the architecture I appreciated is that Data Hub relies heavily on Cron jobs and jobs in Kubernetes. Whenever it needs to fix something, it initiates a job to repair its MySQL or its Elasticsearch. Operationally, I find it to be an excellent service, as they worked well on that aspect with the open-source version. However, the lack of support for KRAFT out of the box was somewhat problematic.

I previously evaluated OpenMetadata as a tool we considered before choosing Data Hub. In comparison to OpenMetadata, the lack of support for more databases and data sources was the deciding factor, whereas for Data Hub, we didn't encounter any problems; it worked really well.

Data Hub helped us by making it clear who owned which data and who needed to make changes to clean the deprecated data models and infrastructures we had, which was the most significant benefit. Using a tool that Data Hub provided made us visible to the faults and bugs in our different data sources.

I would recommend that organizations considering Data Hub adopt GitOps practices, as we implemented it where every single ingestion or transformation was triggered by GitLab CI/CD, making it straightforward for everyone to use. That was the most innovative approach we took by running every single ingestion job as a Cron job in Kubernetes through our GitOps.

I would rate this product a nine out of ten.

My main use case for Data Hub is to catalog the dataset across my company and to get the lineage of data in the my company pipeline.

To give an example of how I use Data Hub in my day-to-day work, suppose the data is flowing from a source to Kafka and then to some data storages. If some cross-team wants to use the data but there is a problem at the Kafka level, we are not sure who all are consuming that data. Data Hub is very useful for us in this scenario. It can generate the lineage from source to destination, and when there is an issue at the Kafka side, we will get to know what the end results and impacted data sources are.

I would add that sometimes when we do not want to share the data or when the customer or another team wants to consume the data, we are not sure what kind of data is there. We have to look at the schema. Data Hub is useful for us as we are doing the cataloging of all the datasets across my company, allowing us to later use and see the table information and schema information so that the team can identify what data is PII or non-PII.

The best features Data Hub offers include support for cataloging and lineage very well, as we are getting all the different types of connectors to consume and use across the my company dataset pipeline. Apart from that, the GraphQL APIs provided by Data Hub are very good, allowing us to get all the information we need programmatically whenever we need it.

Regarding how the GraphQL APIs help my team in day-to-day tasks, we sometimes use custom logic to check whether the data has PII or non-PII. We have some AI model running on top of it, which requires classification. Based on the dataset URL, we are getting information about the dataset using the GraphQL APIs. GraphQL APIs are very handy, allowing us to customize properties and pass on the necessary information. For example, if we need a structured property, we can get those structured properties. If we need tags or owners, we can retrieve that as well.

Data Hub positively impacts my organization by enhancing collaboration as previously, we had to ask the team to provide the schema information. my company operates in a cross-region environment, so a person in India could wait a day to receive information about the schema from someone in the US. However, with Data Hub, we have a centralized place where we can access all the schema of the datasets, making it very helpful. Additionally, whenever there is a problem, using the lineage helps us quickly identify the impacted team or dataset.

Whenever there is an incident, we first go to Data Hub to see the downstream teams impacted and stop any jobs running on those datasets. It helps us save around eighty percent of time, as we previously had to track down information manually to find the owners, but using Data Hub, we can tag the owners of the datasets directly in the tool.

For improvements to Data Hub, I feel the security is a bit on the weaker side. We have ingestion jobs that require exact permissions for different owners, but this setup does not align with the my company grouping system. We need to create some custom grouping to manage those permissions. I would appreciate it if there were a method to consolidate all the information on a single page, which would simplify sharing permissions for running ingestion jobs.

Additionally, I do feel that the metadata test we run daily takes too long. Initially, it takes one day, which I find excessive. Ideally, we should get information within one hour. These are the two main issues that would benefit from improvement for our use case.

I have been using Data Hub for one and a half years.

Data Hub is stable in my experience. However, there are times when we attempt to upgrade it, and it may go down for a couple of minutes, but not more than that.

Data Hub handles scalability effectively, accommodating growing data and users.

I have had to reach out to Data Hub customer support multiple times. For example, when we were setting up a private link to connect to Data Hub GraphQL APIs, we required our account to be whitelisted. I have also requested some future features for our use cases. For instance, when working with a metadata test scenario, I needed to have a range date column, which was not available. I requested the Data Hub team to make it public so we could use it.

I have seen a return on investment with Data Hub. For instance, I have noticed time savings during incidents and while looking up schemas. In terms of resources, Data Hub centralizes data cataloging and classification, saving us from having to disclose PII column information to teams not utilizing it. Regarding financial metrics, I do not have specific metrics available.

Before choosing Data Hub, we looked into Unity Catalog from Databricks, but we ultimately decided to stick with Data Hub.

My advice for others looking into using Data Hub is to use it for cataloging, classification, and centralizing all your schema. Data Hub supports a variety of connectors and has excellent lineage options. Additionally, make sure to utilize the well-written documentation that can guide you in building your product solutions. I would rate this product a nine out of ten.