Metadata management has streamlined lineage tracking and data discovery for our teamsThe best features Data Hub offers include its integration capability with many popular tools like Apache Airflow, Snowflake, dbt, Looker, Apache Kafka, and BigQuery. These tools provide us with data in various places, and we commonly use Apache Airflow for the DAG, while utilizing BigQuery as our database and Apache Kafka for consuming messaging queues. Data Hub easily connects with all these tools and features excellent data discovery and visualization capabilities. We can see data visibility, where it comes from, its upstream and downstream relationships. If we remove a column, we can assess the impact of that change. Furthermore, if there are duplicate datasets being used by different teams that do not communicate regularly, onboarding all data to Data Hub allows us to identify these duplicates easily. Out of all those features, I believe data discovery and impact analysis are the most valuable for my team because when we want to add or drop a column, we can assess the impact analysis to understand the downstream effects. This helps us know who owns a dataset, and we can easily contact the owner. Tracking the data lineage back to the source table is also a key benefit. Data Hub has positively impacted my organization by significantly reducing manual work that was previously needed to identify upstream and downstream data relationships, as well as recognizing duplicate datasets. If a data contract is broken, we now easily get notified of those issues, making the process much easier and more efficient. It is particularly useful for data engineers and platform teams to check for problems directly within Data Hub. Data Hub has saved our team a lot of time. For example, in a large company like Porch, if I want to know whether a specific dataset exists, I can check Data Hub, as it serves as a centralized point for managing the metadata of our data. While it does not contain all data, it does contain the metadata necessary for understanding the dataset's origin. If a dataset does not exist, I can simply see who the owner is and reach out to them, which reduces the dependency on others by providing direct access to information in Data Hub.
