Databricks and Amazon MSK serve different categories, with Databricks leading in data analytics and machine learning, and Amazon MSK leading in data streaming with Apache Kafka integration. Databricks might have an advantage in environments focusing on advanced analytics and machine learning capabilities, whereas Amazon MSK offers superior real-time data processing services.
Features: Databricks provides an integrated environment that supports multiple programming languages, collaborative notebooks, and machine learning integrations. It includes built-in optimization for data processing and interactive querying. Amazon MSK simplifies real-time stream management with a fully managed Kafka service, reducing administrative tasks and offering seamless integration with other AWS services.
Room for Improvement: For Databricks, improvements could be made in simplifying administrative setup, expanding multi-cloud compatibility, and enhancing cost transparency in data processing. For Amazon MSK, enhancements might include broadening support for custom integrations outside of AWS, improving user interfaces for complex setups, and optimizing latency for large-scale streaming environments.
Ease of Deployment and Customer Service: Databricks offers deployment across various cloud environments and is praised for effective customer support. In comparison, Amazon MSK offers tight integration within the AWS ecosystem but may face challenges in multi-cloud deployments despite benefiting AWS users through streamlined service integration.
Pricing and ROI: Databricks employs flexible pricing models catering to diverse business needs, which can result in high ROI from data insights. Amazon MSK's pay-as-you-go pricing appeals to existing AWS customers, with cost efficiency tied to data throughput, offering a budget-friendly option for streaming services. Both solutions offer competitive pricing, with Databricks potentially requiring higher initial investment but yielding substantial returns through efficient data strategies, whereas Amazon MSK provides an economical choice within the AWS ecosystem.
For a lot of different tasks, including machine learning, it is a nice solution.
When it comes to big data processing, I prefer Databricks over other solutions.
They can manage most of our queries, and for what they cannot manage, they guide us through the process of finding out.
Amazon's support is excellent.
Whenever we reach out, they respond promptly.
As of now, we are raising issues and they are providing solutions without any problems.
I rate the technical support as fine because they have levels of technical support available, especially partners who get really good support from Databricks on new features.
The functionality for scaling comes out of the box and is very effective.
As a B2B enterprise client, our clientele consists of large ticket clients but low amounts of users.
The patches have sometimes caused issues leading to our jobs being paused for about six hours.
Databricks is an easily scalable platform.
I would rate the scalability of this solution as very high, about nine out of ten.
It doesn't require any maintenance on my end yet, as I haven't had any issues.
They release patches that sometimes break our code.
Although it is too early to definitively state the platform's stability, we have not encountered any issues so far.
Databricks is definitely a very stable product and reliable.
The increase in cloud costs by 50% to 60% does not justify the savings.
The only issue with Amazon MSK that we are facing is the configurations.
I had to remove and drop all the clusters and recreate them again, which is complicated in a production environment.
Adjusting features like worker nodes and node utilization during cluster creation could mitigate these failures.
We prefer using a small to mid-sized cluster for many jobs to keep costs low, but this sometimes doesn't support our operations properly.
We use MLflow for managing MLOps, however, further improvement would be beneficial, especially for large language models and related tools.
Once we started using Kafka, our cloud costs rose by 50% to 60%.
We use Kafka M5 Large instance, and depending on the instances, that is the cost we have, along with storage cost and data transfer costs.
It is not a cheap solution.
The scalability and usability are quite remarkable.
The best features of Amazon MSK are the real-time analytics that are excellent.
Amazon MSK is basically Kafka in the cloud, and when you need to create a cluster of Kafka brokers, Amazon MSK helps with that by automatically creating all the brokers according to the configuration you provide.
Databricks' capability to process data in parallel enhances data processing speed.
The platform allows us to leverage cloud advantages effectively, enhancing our AI and ML projects.
The Unity Catalog is for data governance, and the Delta Lake is to build the lakehouse.
Product | Market Share (%) |
---|---|
Databricks | 12.5% |
Amazon MSK | 5.9% |
Other | 81.6% |
Company Size | Count |
---|---|
Small Business | 4 |
Midsize Enterprise | 7 |
Large Enterprise | 4 |
Company Size | Count |
---|---|
Small Business | 25 |
Midsize Enterprise | 12 |
Large Enterprise | 56 |
Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data. Amazon MSK provides the control-plane operations, such as those for creating, updating, and deleting clusters.
Databricks offers a scalable, versatile platform that integrates seamlessly with Spark and multiple languages, supporting data engineering, machine learning, and analytics in a unified environment.
Databricks stands out for its scalability, ease of use, and powerful integration with Spark, multiple languages, and leading cloud services like Azure and AWS. It provides tools such as the Notebook for collaboration, Delta Lake for efficient data management, and Unity Catalog for data governance. While enhancing data engineering and machine learning workflows, it faces challenges in visualization and third-party integration, with pricing and user interface navigation being common concerns. Despite needing improvements in connectivity and documentation, it remains popular for tasks like real-time processing and data pipeline management.
What features make Databricks unique?In the tech industry, Databricks empowers teams to perform comprehensive data analytics, enabling them to conduct extensive ETL operations, run predictive modeling, and prepare data for SparkML. In retail, it supports real-time data processing and batch streaming, aiding in better decision-making. Enterprises across sectors leverage its capabilities for creating secure APIs and managing data lakes effectively.
We monitor all Streaming Analytics reviews to prevent fraudulent reviews and keep review quality high. We do not post reviews by company employees or direct competitors. We validate each review for authenticity via cross-reference with LinkedIn, and personal follow-up with the reviewer when necessary.