From our experience, Azure Databricks could be improved with simpler cluster management and more predictable cost visibility and enhanced native monitoring for large enterprise environments. We would also like richer governance features, better debugging for distributed Spark jobs, and more granular controls for workload optimization over and across multiple teams, which we have at multiple customer environments and within our organization. In day-to-day operations, troubleshooting failed Spark jobs can still be time-consuming, especially in complex distributed workloads. We would like clearer root cause diagnostics and more actionable performance recommendations within Azure Databricks. Better cost optimization insights at the job and cluster level would also help us manage large multiple team environments more efficiently.
The biggest friction point I have experienced with Azure Databricks is its cost-effectiveness; for projects with less data volume, it is advisable to use Azure Fabric services instead, as Azure Databricks may not be suitable for low volume processing. I wish Azure Databricks offered the ability to further integrate advanced features such as Genie AI, which simplifies coding by automatically building PySpark code based on scenarios. From my perspective, I need to enhance my knowledge around Azure data lineage using Purview, and I find that downtime of Azure Databricks clusters significantly impacts the service and execution of existing pipelines, requiring careful management of backlog items.
My team handles those specific tasks, and I know that they are doing well with that. The only concern is perhaps related to the pricing and cost that Azure Databricks incurs.
Senior Business Intelligence Consultant at Stellar Consulting Group
MSP
Top 20
Mar 3, 2026
Overall, my experience has been positive with Azure Databricks; they have many features, but there is no use case for me to use those features, such as Delta Live Tables and Genie. In my opinion, I do not know how Azure Databricks can be improved, as I am not on the product side, but they are just building new features. I cannot think of any additional features I would like to see in the future.
From the improvement perspective, there is a feature in SAP for when we were doing dimensional modeling and the cube concept called navigational attributes. From that perspective, SAP BW is a little bit more mature because apart from RBAC, it gives data-level authorization, which is a little bit not that great in Azure Databricks at this point in time.
Azure Databricks is an advanced analytics platform combining the best of Microsoft's Azure and Apache Spark. It provides a powerful solution for big data processing, machine learning, and collaborative data projects, designed to help organizations unlock insights and foster innovation.Azure Databricks integrates seamlessly with Azure services, offering end-to-end data solutions for enterprises. Its collaborative environment supports data engineers and scientists, facilitating faster data...
From our experience, Azure Databricks could be improved with simpler cluster management and more predictable cost visibility and enhanced native monitoring for large enterprise environments. We would also like richer governance features, better debugging for distributed Spark jobs, and more granular controls for workload optimization over and across multiple teams, which we have at multiple customer environments and within our organization. In day-to-day operations, troubleshooting failed Spark jobs can still be time-consuming, especially in complex distributed workloads. We would like clearer root cause diagnostics and more actionable performance recommendations within Azure Databricks. Better cost optimization insights at the job and cluster level would also help us manage large multiple team environments more efficiently.
The biggest friction point I have experienced with Azure Databricks is its cost-effectiveness; for projects with less data volume, it is advisable to use Azure Fabric services instead, as Azure Databricks may not be suitable for low volume processing. I wish Azure Databricks offered the ability to further integrate advanced features such as Genie AI, which simplifies coding by automatically building PySpark code based on scenarios. From my perspective, I need to enhance my knowledge around Azure data lineage using Purview, and I find that downtime of Azure Databricks clusters significantly impacts the service and execution of existing pipelines, requiring careful management of backlog items.
My team handles those specific tasks, and I know that they are doing well with that. The only concern is perhaps related to the pricing and cost that Azure Databricks incurs.
Overall, my experience has been positive with Azure Databricks; they have many features, but there is no use case for me to use those features, such as Delta Live Tables and Genie. In my opinion, I do not know how Azure Databricks can be improved, as I am not on the product side, but they are just building new features. I cannot think of any additional features I would like to see in the future.
From the improvement perspective, there is a feature in SAP for when we were doing dimensional modeling and the cube concept called navigational attributes. From that perspective, SAP BW is a little bit more mature because apart from RBAC, it gives data-level authorization, which is a little bit not that great in Azure Databricks at this point in time.