Azure Databricks Reviews

My main use case for Azure Databricks is integrating different types of sources such as Oracle ERP, SAP ERP, API, SQL server, and Oracle databases, focusing on an ingestion platform, integrating and ingesting data into the raw layer, transforming that into a silver layer, and then creating a golden layer for the transformed data to be consumed by Power BI.

When I open Azure Databricks, the first thing I typically do is determine the services or applications to create a first end-to-end architecture platform using Azure services, decide which services are best for ingestion and transformation into the silver and golden layers, and then build a proof of concept to implement an end-to-end framework, which will guide the successful delivery of the entire project in a regular stream.

When I open Azure Databricks, my day-to-day workflow begins with validating the proof of concept framework, choosing the right service for the ingestion pattern, and then building or running notebooks for raw, silver, and gold layer transformations. I involve myself in building the data architecture, coordinating with my senior tech leads or COE groups if I have any questions to ensure we understand the required solutions, and after presenting the architecture to the customer, we progress on development and testing in the deliverables.

The best features of Azure Databricks I use the most include schema evolution, auto merge, and efficient handling of shuffles or transformations, along with Z-ordering and partitioning, which enhance the performance of data transformation. Additionally, Delta format, time traveling, versioning, and ACID properties are sophisticated features that provide great support for cloud Lakehouse architecture.

Azure Databricks has positively impacted my organization by allowing us to build medallion architecture or Kappa architecture for handling both scheduled and streaming data, and after establishing the ingestion framework, the existing data services handle transformations and cleansing effectively, which means we rarely need to change services. I also make use of Unity Catalog, RBAC, ACL, and Purview for data governance and security, enabling efficient management of scalable solutions across the organization.

The biggest friction point I have experienced with Azure Databricks is its cost-effectiveness; for projects with less data volume, it is advisable to use Azure Fabric services instead, as Azure Databricks may not be suitable for low volume processing.

I wish Azure Databricks offered the ability to further integrate advanced features such as Genie AI, which simplifies coding by automatically building PySpark code based on scenarios. From my perspective, I need to enhance my knowledge around Azure data lineage using Purview, and I find that downtime of Azure Databricks clusters significantly impacts the service and execution of existing pipelines, requiring careful management of backlog items.

I have been using Azure Databricks for more than four years.

When I first implemented Azure Databricks in my environment, setting it up took longer due to the need to coordinate with the customer for all the necessary data platform setups. However, if the customer already has an Azure data platform, it is easy to onboard new people quickly and start the development phases on time.

When my team started using Azure Databricks, they needed formal training to become comfortable with it, as IT organizations typically require newly onboarded or recruited individuals to undergo multiple training sessions on different platforms, including Azure, which lasts four to six months, after which they are assigned to projects and begin shadowing senior engineers or developers to pick up small initiatives.

Before standardizing on Azure Databricks, we were using traditional ETL services such as Informatica, Ab Initio, DataStage, Oracle ODI, and SAP BODS as data warehousing tools, and with data modernization, companies began migrating to the Azure data platform and other cloud services.

My pipelines are now significantly faster compared to older ETL tools. For example, what used to take over 12 to 14 hours to process 2 GB of source data in Synapse Analytics now completes within 5 hours using Azure Databricks framework for the transformation part, illustrating a substantial improvement in performance.

When evaluating modern options, I did consider other tools besides Azure Databricks, including Snowflake, Synapse, and AWS Glue, but I chose Azure Databricks specifically because it handles high volumes of source data efficiently, managing daily ingestion in gigabytes or terabytes, and facilitates the transformation of data into modern data models suitable for machine learning and AI purposes.

When I open Azure Databricks, my day-to-day workflow begins with validating the proof of concept framework, choosing the right service for the ingestion pattern, and then building or running notebooks for raw, silver, and gold layer transformations. I involve myself in building the data architecture, coordinating with my senior tech leads or COE groups if I have any questions to ensure we understand the required solutions, and after presenting the architecture to the customer, we progress on development and testing in the deliverables.

Although Delta Live Tables and streaming services are features I discussed and planned to use during implementation, I have utilized them very minimally for my applications.

My advice for someone considering Azure Databricks with a similar workflow is to ensure their solution can operate independently of cloud services, making it possible to work across Azure, GCP, or AWS, and to implement a plug-in/plug-out feature architecture that does not disrupt the overall framework. I would rate this product a seven out of ten.

I have constructed the Lakehouse on Azure Databricks for ETL, machine learning, deep learning, and all data science purposes. Azure Databricks is a very powerful tool when it comes to handling all these applications, and the Spark framework stands out significantly.

I utilize Azure Databricks for real-time data processing.

Regarding the collaborative features of Azure Databricks, they are working on it regularly. At this point in time, it is collaborative, and spinning up your own cluster is also easier. Comparatively, Snowflake is doing a little bit better based on my experience. I cannot judge on other people's experiences, but I feel Snowflake does a little bit better compared to Azure Databricks at this point in time. However, Azure Databricks itself has good enough collaboration.

I have integrated Azure Databricks with Power BI.

With integration of Azure Databricks, we use Power BI, Azure ML Studio, and Azure Data Factory. There are multiple other tools which we integrate as well. Additionally, with external tools outside Microsoft, such as Tableau, you can integrate. There are quite a lot of things you can do on top of Azure Databricks.

We use Azure Databricks mostly for ETL purposes. We use programming languages such as both SQL and Python. Scala is also possible, but I have never experienced Scala at this point in time. This is what its power is; it is mostly a code-based platform. There is something called agents, and also you have a capability recently developed called Agent Bricks, which is the GenAI capability that they are bringing up.

The real-time processing with Azure Databricks is supported through integration from external systems, for which we have to go with tools such as Matillion's HVR or Kafka. I have experience using HVR, high-volume replication. You get real-time data replicated into Azure Databricks using these tools.

When looking for performance metrics in Azure Databricks, it depends on the processing. It can process millions of records quickly, and it is driven by the Spark framework, which is pretty strong in terms of framework perspective. The columnar database is another strong feature which helps enhance its performance.

Prior to the introduction of Unity Catalog, there was no metadata capability in Azure Databricks. It was very simplistic, but now with the Unity Catalog introduction and Delta Sharing capabilities, Azure Databricks is at the top-notch at this point in time. In comparison, SAP BW is a little bit more mature because apart from RBAC, it gives data-level authorization, which is a little bit not that great in Azure Databricks at this point in time.

From the improvement perspective, there is a feature in SAP for when we were doing dimensional modeling and the cube concept called navigational attributes. From that perspective, SAP BW is a little bit more mature because apart from RBAC, it gives data-level authorization, which is a little bit not that great in Azure Databricks at this point in time.

I have been using Azure Databricks since 2021.

I find the deployment of Azure Databricks to be very simplistic. I can spin up an Azure Databricks environment within two minutes.

Regarding the licensing cost of Azure Databricks, it has evolved quite a lot. The compute is the biggest cost, as with any other big data solutions. The storage cost is almost minimal or negligible. Azure Databricks gives the capability to handle a lot of big data use cases and machine learning use cases, but machine learning use cases need quite a lot of compute power, and that is where the cost spikes up. There are features available for auto-scaling both up and down, vertically and horizontally. Still, in this regard, there are other tools such as Snowflake that are doing a little better with data warehousing workloads. The cost itself is based on t-shirt sizes. Earlier, it used to be different clusters and sizes. They are evolving as time progresses, which is helpful. At this point, I cannot comment on the cost being ideal; it is on the higher side, but in the cloud-based environment, compared to on-premise, it could be far lesser in cost.

Comparatively, Snowflake is doing a little bit better based on my experience. I cannot judge on other people's experiences, but I feel Snowflake does a little bit better compared to Azure Databricks at this point in time.

I think Azure Databricks is a very good product. I would rate it a nine out of ten.

I utilize Azure Databricks for both batch and real-time data processing. The impact of real-time data processing on my projects is significant, as my data analytics and reporting people, along with my business analyst group, generate real-time data reports. I'm using Azure Databricks for our production business applications.

The best features in Azure Databricks for me are that it's easy to use, flexible, and has fast processing, and you can use multiple data types. Implementing Azure Databricks is easy, as it doesn't present many challenges in terms of deployment, and deployment is not that time-consuming.

I would assess the collaborative features of Azure Databricks as a processing engine that improves our team's productivity 100%. Multi-language support in Azure Databricks helps increase my team's productivity, as it supports Python, JSON, and SQL.

I have only been using Azure Databricks for a year and a half, so I'm not the right candidate to comment on improvements; I'm still exploring and in discovery mode for all the features. I have the same concern regarding future updates; I don't know all the features myself, but I can say it's the future. Lower pricing is currently my only focus and I'm still exploring Azure Databricks, so it's too early to say something, but overall, I'm saying that it is the future.

I've been using Azure Databricks for the last one and a half to two years.

I can rate the support of Microsoft as an 8 or 8.5.

Positive

I have a team of developers who generally handle deployment, and if they face any issues injecting data or with the CI/CD pipeline or ETL pipeline, they resolve it without any assistance or consultants.

Azure Databricks is the right choice. 

The primary use cases for me are the reportings I have to do, so I need to ingest data from the file and create reports.

I do not utilize it for real-time data processing.

I have not integrated Azure Databricks with other Azure services, just Excel mainly; I put data in Excel, load it, ingest it in Azure Databricks and then use it.

I assess the collaborative features in improving team workflows as good; they are still making some updates, but it is good. I do not use all of the features, just limited features.

Regarding the learning curve, it is a good technology; it is the first time I am working on a cloud platform, and before that, I have not worked on any data engineering tool that is on cloud, so it is good learning.

Overall, my experience has been positive with Azure Databricks; they have many features, but there is no use case for me to use those features, such as Delta Live Tables and Genie.

In my opinion, I do not know how Azure Databricks can be improved, as I am not on the product side, but they are just building new features.

I cannot think of any additional features I would like to see in the future.

I have been working with Azure Databricks for almost two to three years.

I have not had any significant problems with the solution; it is a new technology, and when a new technology exists, you definitely need to learn more. There is an admin side, and there are still new features, along with an infrastructure team that handles many things. There is another team involved in different use cases, and I am in a different area.

I did not have any problems with the performance or with the scalability functionality; performance means if I want better performance, then I can go for a bigger warehouse or bigger compute power because they do parallel processing, but that will cost more. Performance-wise, it is fine.

The implementation time depends on the use case; if there is some complex data, it can take more time, but if there are easy data to load then it depends on what type of report they want and what type of data needs to be ingested.

In my region, the product is not very popular; Snowflake is more popular. People do know about it, but Snowflake is more widely recognized.

I can only recommend Azure Databricks to other companies when I know other products as well; I have not worked on Snowflake. People say that Snowflake is also good, but I do not have any list of what features are there in Snowflake or in Azure Databricks, and which one to use in which scenario. I do not have any information at this time; once I have that information, I can definitely recommend.

I do not use the multi-language support in Azure Databricks.

I do not use any metrics as such because these are small files; I am not loading millions of records, it has just some thousands of records.

The pricing of Azure Databricks is handled by the infrastructure team, so I am not involved in that; I am mainly a developer. If I have to use a warehouse which is more powerful, I need to make a request because that costs, otherwise I just do my work.

I think the interface is intuitive enough.

They provide online training; online training is available.

I have given the product a rating of six out of ten just because I do not use all of the functionalities, and I see some direction for improvement as well; also, every product has something to improve, and I have not used many features in this product.

I think this is related to our internal business, as we have a data warehouse and data lakes that we use Azure Databricks for. We work for Bosch, and we have the solution internally. We purchase everything through a big IT organization.

The concept of Azure Databricks is a very good one, especially for the data products concept and idea. However, to bring the concept into reality, we as an engineering department have to persuade others significantly. My suggestion is that Azure Databricks should believe in the concept of data products if it is needed. This would allow us to have more material to support our technical team when we are doing internal consultation.

My team handles those specific tasks, and I know that they are doing well with that. The only concern is perhaps related to the pricing and cost that Azure Databricks incurs.

My team has been working with it for about two years.

Onboarding is quite easy.

I think normally we just need two people to deploy this solution, but the whole team working on those topics is about eight. For setting up, it is just two people.

I gave this product a rating of eight out of ten.