My main use case for Datafold is to automatically compare data differences through data diffing, where I can compare datasets row-by-row, column-by-column, to surface unexpected changes. I also use it for CI/CD integration for data pipelines, where we embed data quality checks into GitHub and GitLab pull requests. Another interesting feature I use Datafold for is column-level lineage to trace how specific columns flow through SQL transformations across the data stack we have. Recently, for data diffing, there was one time we wanted to conduct a lot of comparisons on a very large dataset to be sure that all was in check before we migrated to Snowflake, which is a complex and resource-heavy process. We used Datafold to do the automatic checks across rows and columns to ensure that when we completed the migration, no data was lost and the migration into Snowflake was very smooth. That was in March, and it was very useful. For a friend who uses Datafold in their enterprise, an engineer made a change from an SQL model and did not realize it would cascade and alter downstream dashboard metrics. However, with the help of Datafold integrated into the pull request workflow, the data diff actually surfaced exactly which columns changed, which rows were affected, and by how much.
Data Quality solutions help businesses maintain the accuracy, completeness, and consistency of their data, enhancing decision-making processes and operational efficiency. These solutions are essential for ensuring data integrity across various enterprise systems and applications. Data Quality solutions provide organizations with the tools to cleanse, standardize, and validate data, reducing errors and enhancing reliability. With features like data profiling, these solutions facilitate...
My main use case for Datafold is to automatically compare data differences through data diffing, where I can compare datasets row-by-row, column-by-column, to surface unexpected changes. I also use it for CI/CD integration for data pipelines, where we embed data quality checks into GitHub and GitLab pull requests. Another interesting feature I use Datafold for is column-level lineage to trace how specific columns flow through SQL transformations across the data stack we have. Recently, for data diffing, there was one time we wanted to conduct a lot of comparisons on a very large dataset to be sure that all was in check before we migrated to Snowflake, which is a complex and resource-heavy process. We used Datafold to do the automatic checks across rows and columns to ensure that when we completed the migration, no data was lost and the migration into Snowflake was very smooth. That was in March, and it was very useful. For a friend who uses Datafold in their enterprise, an engineer made a change from an SQL model and did not realize it would cascade and alter downstream dashboard metrics. However, with the help of Datafold integrated into the pull request workflow, the data diff actually surfaced exactly which columns changed, which rows were affected, and by how much.