What is our primary use case?
Azure Databricks is our primary platform for building scalable data engineering and analytics pipelines for enterprise customers. We use it to inject, transform, and process large volumes of structured and unstructured data, then prepare curated data sheets for BI reporting, machine learning, and real analytics.
One project that stands out involved building a centralized data platform for a retail customer where we ingested POS, e-commerce, and inventory data into Azure Databricks. Using Spark and Delta tables, we transformed the data into curated data sheets that powered Power BI dashboards and demand forecasting models, reducing data processing time from several hours to under an hour.
What is most valuable?
The best features of Azure Databricks are its optimized Apache Spark engine, Delta Lake support, collaborative notebooks, and seamless integration with Azure services. We also value its auto-scaling clusters, strong performance for large-scale data processing, and built-in support for data engineering, analytics, and machine learning workloads on a single platform.
Azure Databricks has significantly improved our ability to process data and large data sets, and deliver analytics projects faster for our customers. We have reduced data processing time by 50 to 60%, improved data pipeline reliability, and enabled deployment teams to collaborate more efficiently on a single, scalable analytics platform.
What needs improvement?
From our experience, Azure Databricks could be improved with simpler cluster management and more predictable cost visibility and enhanced native monitoring for large enterprise environments. We would also like richer governance features, better debugging for distributed Spark jobs, and more granular controls for workload optimization over and across multiple teams, which we have at multiple customer environments and within our organization.
In day-to-day operations, troubleshooting failed Spark jobs can still be time-consuming, especially in complex distributed workloads. We would like clearer root cause diagnostics and more actionable performance recommendations within Azure Databricks. Better cost optimization insights at the job and cluster level would also help us manage large multiple team environments more efficiently.
For how long have I used the solution?
I have been using Azure Databricks for two years.
Which solution did I use previously and why did I switch?
We previously used traditional ETL platforms and on-premises Apache Spark environments for several customer projects. We switched to Azure Databricks because it offered the best scalability, simplified cluster management, and Azure-native integrations, and significantly improved our overall data processing capabilities.
How was the initial setup?
Our experience with Azure Databricks has been positive. The initial setup was straightforward for teams already using Azure, with no major upfront deployment cost beyond implementation effort. Licensing is flexible, but compute and license costs need to be monitored carefully. With proper cluster policies and auto-scaling, we have been able to balance performance with cost effectively.
What was our ROI?
We have seen a clear return on investment with Azure Databricks. The data engineering effort has been reduced by 30 to 40%. ETL processing times have been improved by 50 to 60%. New analytics use cases are typically delivered 30 to 40% faster. We did not reduce headcounts, but our existing teams are significantly more productive and support more customer projects within the same resources.
What other advice do I have?
We rely on Delta Lake most because it provides ACID transactions, schema enforcement, and reliable data versioning for production data pipelines in Azure Databricks. It has significantly reduced data quality issues and makes it much easier for multiple teams to build and maintain reliable ETL workflows while supporting rollback and audit requirements.
A great feature is the tight integration between Azure Databricks and the Azure ecosystem. Connecting with Azure Data Lake Storage, Azure Data Factory, Microsoft Entra ID, and Power BI is straightforward and simplifies end-to-end data platform management. We also find cluster auto-scaling very valuable because it optimizes resource usage during peak processing while helping control infrastructure costs.
We measured the 50 to 60% reduction in processing time by comparing end-to-end ETL execution times before and after migrating several customer data pipelines to Azure Databricks. For example, workflows that previously took four to five hours on traditional ETL platforms consistently completed in around two hours after optimizing with Spark, Delta Lake, and auto-scaling clusters. Similar improvements were observed across multiple customer projects and not just a single deployment.
The AI capabilities in Azure Databricks are reliable when the underlying data is well-governed and the models are properly trained. Azure Databricks delivers consistent results for ML workloads, and its support for versioning, reproducibility, and experiment tracking helps us maintain accuracy across production deployments.
Azure Databricks provides strong governance and security through role-based access control, Unity Catalog, data lineage, and integration with Microsoft Entra ID. These capabilities help us enforce consistent access policies, protect sensitive customer data, and meet enterprise compliance requirements for AI and analytics workloads.
Many of the needed improvements have already been discussed. I would rate this product an 8 out of 10 overall.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Microsoft Azure