Amazon MSK Reviews

We are recently working with Amazon MSK at Fortis, where we have multiple dashboards in our revenue intelligence platform. We are streaming data from different apps into those dashboards. The data flow goes from applications through Spark and then to our dashboard. We are using Amazon MSK for changes in the CDC pipelines, in the DBs, and for logging. We use it for every microservice communication that we are building for the product. We decouple the services with async event streaming at Amazon MSK.

We have used Amazon MSK for IoT data streaming because in our product, we collect audio from retail shop floors. We have an IoT device on stream and a Raspberry Pi on stream. We can set up clouds on tracking devices and track everything from there through Amazon MSK, including their Docker health. It streams through Amazon MSK into our common database.

The best features of Amazon MSK are the real-time analytics that are excellent. Event sourcing is good, and log and metric ingestion is good. These are three things we are using very heavily.

Amazon MSK has contributed positively to our real-time analytics capabilities because Fortis's dashboards have dashboard health that needs to be maintained, user logs that need to be maintained, and usage tracking. That data comes through Amazon MSK into our platforms in real-time and gets converted into actionable outputs for different teams including customer support. We have an implementation team that is getting the dashboard usage up and supports adoption. This data also goes into multiple other sources in the tech background where customer issues and logs are being identified. Customer support and tech collaborations are done through Amazon MSK for shifting data.

The integration capabilities of Amazon MSK are not very flexible. If you have your own self-managed Kafka, that helps significantly because you can set up configurations. We are considering self-managed Kafka since our product is only one year old. The Kafka integrations are fine, but the configurations are an issue.

The only issue with Amazon MSK that we are facing is the configurations. There are preset configurations and limited configurations that we can set for our unique use case. The product could improve by allowing us to set different configurations.

I would also like to see Amazon MSK improve in the area of connectors. We are considering Confluent Cloud because they have many more connectors. They have KSQL DB and governance features. It is slightly costlier, but Confluent offers more flexibility with their connectors.

I have been working with Amazon MSK for two to three years.

With Amazon MSK, we have not experienced any downtimes or crashes.

When we have large volumes of data, we find it manageable because we manage it in a different way. We do not put that on Amazon MSK currently. As a B2B enterprise client, our clientele consists of large ticket clients but low amounts of users. The data problem is not present in our use case.

I have escalated questions regarding Amazon MSK to the tech support because the tech team uses it. I am not certain what they use it for as our back-end lead handles that. The tech support is good usually. AWS tech support has been quite helpful whenever we have used it. They can manage most of our queries, and for what they cannot manage, they guide us through the process of finding out.

Positive

Regarding the event data collection process, I evaluate Amazon MSK's impact as functioning fine. These are normal tools that work on an open source framework. Setting it up is easier because we are completely based on AWS. Our product is completely based on AWS, so Amazon MSK makes sense because integrations with the ecosystem are much easier.

My advice for users who are looking into Amazon MSK is that they can use it according to their needs. It is a simple platform to set up. They should identify if AWS MSK is needed for their workstream. It is not always that this service is usable for everyone. It is very specific if you are on AWS continuously and your whole system is on the AWS Cloud and you need integrations with AWS Cloud. If not, you can use open source platforms such as Kinesis that will also do more things.

Regarding the pricing aspect of Amazon MSK, we have basic costs as we are billed per hour for the broker instance hours. We use Kafka M5 Large instance, and depending on the instances, that is the cost we have, along with storage cost and data transfer costs, which are the standard data transfer cost for AWS.

We are working with Amazon MSK product, Managed Streaming for Apache Kafka.

The main benefits I have seen from using Amazon MSK is that as a B2B enterprise client, we have many SLAs to fulfill. Tracking those SLAs and then building out and detecting it early is something that Amazon MSK helps us do very effectively.

On a scale of one to ten, I rate Amazon MSK a six out of ten.

The main use case I am working with is actually with Debezium, which is a tool to extract information from one source and put it in another source. I am using Debezium to extract information from an Oracle database, and this information is stored in Kafka topics; that is the use of Amazon MSK because I need some way to process this in AWS.

Auto scaling is automatically configured in Amazon MSK, which is great because we don't need to worry about how to configure it. Just some basic configuration was needed, and the application starts to scale automatically, which is beneficial in my case.

Amazon MSK is basically Kafka in the cloud, and when you need to create a cluster of Kafka brokers, Amazon MSK helps with that by automatically creating all the brokers according to the configuration you provide.

What I appreciate most about Amazon MSK is that it doesn't require extensive concern about the configurations. It starts checking how the brokers are functioning, and automatically, Amazon MSK tries to resolve all the problems. The lack of worry about configurations is very advantageous.

The downside of Amazon MSK is that when I needed to make a small change in the MSK configuration, it wasn't possible. I had to remove and drop all the clusters and recreate them again, which is complicated in a production environment.

Configuration changes on the fly would be a good feature to implement.

I have been working for two to three years with AWS, specifically with EC2, and in the last six months, I have been working with Amazon MSK. In the application I am managing, we need to deploy Kafka in the cloud, so Amazon MSK appears to be the better option for deploying Kafka in a cluster in AWS.

For stability, we are not in production, and my load test with Amazon MSK was the bigger test I performed with 60,000 requests, which showed 91% of the requests executed. Any issues seemed to relate to Debezium and Oracle, not Amazon MSK.

It doesn't require any maintenance on my end yet, as I haven't had any issues, and any problems I have encountered seem to be related to Debezium, not Amazon MSK.

I haven't contacted technical support regarding anything else yet.

My company has an agreement with another company who are representatives of AWS.

I haven't actually tried using any commercial competitors of Amazon MSK. I was using just Kubernetes in a Docker Compose environment.

The initial setup of Amazon MSK was pretty easy. I just researched some basic configurations online, and it's not complicated, just very small things.

It took me about three days to fully set up Amazon MSK.

I set up Amazon MSK by myself because trying to contact support requires getting in touch with another team, making it more complicated, and I was just trying to do it independently.

I would rate Amazon MSK an 8 out of 10. I am interested in being a reference for Amazon, so the AWS team can contact me later via email.

We are currently using Amazon MSK to transfer data from our PostgreSQL database to our DynamoDB, acting as a mediator between those two databases for migration purposes.

Our data is in an on-premise PostgreSQL database, and we are moving it to DynamoDB in AWS using Amazon MSK.

I have Amazon MSK integrated with other AWS services such as S3 and Lambda.

We are using Amazon MSK for encryption during our migration process.

I'm not sure exactly what benefit we have because we are using multiple AWS tools. We have AWS DMS, which is also the same as Amazon MSK, and we have Fivetran, which is a third-party website providing similar features to Amazon MSK. Since we are working in the financial domain, we want to ensure everything meets security requirements, which is why we chose Amazon MSK.

We need to create connectors in Amazon MSK, but there are no default connectors in AWS for that purpose. We need to create custom plugins, which requires writing Java code and uploading it to the S3 bucket. From the S3 bucket, we need to create a custom plugin, making it a lengthy process. In comparison, Fivetran has all plugins readily available. When creating a source for Snowflake and DynamoDB in Fivetran, there's no need to create custom plugins. Everything is available, requiring only the selection of source and target, and configuration of URLs and IP addresses. With Amazon MSK, we must create custom plugins and upload zip files to the S3 bucket, making it a long and hectic process.

I have been working with Amazon MSK for almost six months.

I never had any issues with stability or downtime.

I never contacted AWS support regarding Amazon MSK as I did not have any issues.

Positive

The cost of Amazon MSK is reasonable compared to other tools. However, people focus more on time and flexibility rather than cost.

Setting up Amazon MSK is difficult because AWS lacks proper documentation. For completing the proof of concept, it took me six days as a newcomer to Amazon MSK, and I needed step-by-step guidance which was not available.

The implementation process is lengthy. In Fivetran, everything can be created within 10 to 15 minutes. However, with Amazon MSK, it took almost six days because I had no prior knowledge of plugin creation. I had to refer to the documentation and create my code since we needed to develop our own plugins.

I have not encountered any other significant challenges with Amazon MSK beyond the custom plugin creation process.

On a scale of 1-10, I rate Amazon MSK a 9.

We used Amazon MSK to manage high-volume third-party data entering our system. It served as a buffer when our system was unable to consume data at high speeds in real-time. The data initially went to Kafka and was then consumed by multiple consumers. One consumer sent data to the master database, and others sent it to related applications for further processing.

It provides installations, scaling, and other functionalities straight out of the box. The scalability and usability are quite remarkable. The ability to handle everything, including installation, with minimal human intervention, is a strong point. Stability is another valuable feature, with no notable latency or performance issues encountered.

The cost of using Amazon MSK is high, which is a significant disadvantage, as the increase in cloud costs by 50% to 60% does not justify the savings. There were no other notable issues.

We used it for about two to four months.

The solution is stable enough. There are no latency or performance issues.

Scalability is great with Amazon MSK. The functionality for scaling comes out of the box and is very effective.

Amazon's support is excellent. They are available anytime we need assistance.

Positive

The setup process is very smooth as it comes with out-of-the-box installations, which makes it straightforward.

Even one person should be able to handle the installation and operation.

There is some savings, however, the cost increase of 60% does not justify these savings.

I can see the cost increase in our tracking system. Once we started using Kafka, our cloud costs rose by 50% to 60%. I am not aware of the specific licensing model.

I'd rate the solution eight out of ten.

I have used Confluent Cloud and Amazon MSK in my company. We are not using it for analytics and it is more for CDC processes, so we change the capture processes. It is used to extract data from a database and make it available in other parts of our systems or produce events that inform us of data updates.

From AWS, I would consider more MSK schema validation is needed. It is easy to integrate if you have an application, but on-topic integration is more complex. You can do it with EvenBridge very easily, but not with MSK. One of the reasons why we prefer Kafka is because the support is a little bit difficult to manage with Amazon MSK. You have Azure and AWS there, but if you use a connector from the market, the support model is marketplace-based, which means you have to go to the developer of this connector.

I have experience with Amazon MSK.

Amazon MSK offers 99.9 percent availability and stability. I rate the tool's stability an eight out of ten.

Amazon MSK's scalability is very good. There are mostly two aspects to look at on both platforms. If you use AWS, serverless is optional for MSK, and it is very easy. You have to define the number of instances, and you don't manage the infrastructure or anything. If you use the classic MSK, you have to define the number of brokers, and if you have to increase and scale, it is something that you have to do manually based on the number of processes.

If you look at the market today's support, though it is not the fault of AWS, based on the costs, I rate the technical support as a six out of ten.

Neutral

Amazon MSK is a very good option when you have to integrate with other AWS services. It is easy to scale, and it is easy to manage, though it is a little biased by the fact that one should have AWS family products. The main feature that we use in Amazon MSK is the broker. We don't use many MSK connectors, which are the connector part of the AWS service.

When you look at Amazon MSK, it is a service from AWS. What you get is Kafka, and that is about it. You have to recreate the entire ecosystem around it for proper enterprise adoption of a database or event-driven architecture. If you buy if you go for Confluent, you don't just get Kafka, but the full set of services that enables you to deploy every given architecture for enterprise. What it means is that if you use Amazon MSK, you won't have connectors, and so you have to employ MSK connectors well together with MSK. If you use Amazon MSK, you have to employ Glue, which is another AWS service. If you want to have routing, you have to employ EventBridge, meaning you don't have routing directly on MSK. You have to connect together various services.

I honestly didn't work on the setup part and was more on the architecture side. A part of the delivery team does the setup, but it was pretty easy on both sides, as with AWS and Confluent, the team didn't have much trouble.

What we really see in Amazon MSK is that it has flexibility. We see ROI related to innovation. We are able to innovate at a faster speed. We are able to do testing faster and fail faster. At the end of the day, it saves quite a bit of hours and quite a bit of money, which we would otherwise have spent on Confluent. If you need a connector, and if it has a license in Confluent, the model that we have is the cost, meaning even if we use it for a week, we have to pay for it for a year, while with AWS, we pay for what we use, and that allows us to save money on experimentation and move a little faster because there is a process for onboarding new technology. There is a process for being allowed to make some expenditures. There are some processes that we have to respect and work on as well, during which AWS will suffer less.

When you create a complete enterprise-driven architecture that is deployable on an enterprise scale, I would say that the prices of Amazon MSK and Confluent Platform become comparable.

With Amazon MSK, we don't use or handle large data volumes. From what we have seen in terms of our volumes, both Confluent Cloud and AWS are pretty much in line, and they both handle it pretty well.

The AI strategy is finalized, but we have to do some PoCs, although we really don't have any deep AI usage. There are areas of the business that I don't touch, but AI is used for detection. In my area, I am just involved with the PoCs. AI for the business strategy has not yet been finalized.

I would recommend Amazon MSK and Confluent Cloud, but I will make sure that I understand the use case. If you are not in need of connectors and are not in need of deploying a fully-fledged EDA platform, I would say go for Amazon MSK, as you can save some money.

I rate the tool a seven out of ten.

We use Amazon MSK for communication and notification purposes, such as notifying users about new game releases and managing customer support interactions, thus handling large data volumes.

Amazon MSK's separation of concerns and ease of creating and deploying new features are highly valuable. It just requires assigning them to the topic, and then anyone who needs to consume these messages can do so directly from Amazon MSK. This separation of concerns makes it very convenient, especially for new feature development, as developers can easily access the messages they need without having to deal with complex server communications or protocol setups.

In my opinion, there are areas in Amazon MSK that could be improved, particularly in terms of configuration. Initially setting it up and getting it connected was quite challenging. The naming conventions for policies were updated by AWS, and some were undocumented, leading to confusion with outdated materials. It took us weeks of trial and error before discovering new methods through hidden tutorials and official documentation.

I've been using this solution for more than a year.

The scalability of Amazon MSK hasn't had a significant impact on our operations because we are primarily focused on customer support, and we only have a limited number of CSRs working at any given time.

I found the technical support and customer service from Amazon for MSK to be excellent. They were very knowledgeable and helpful in addressing configuration issues.

Positive

Setting up Amazon MSK required six to eight weeks for deployment. The setup involved collaborating with our networking department to define and configure the networking aspects. The deployment process involved multiple people and departments to ensure everything was set up correctly.

The return on investment with Amazon MSK is positive, however, it is challenging to assess the full impact. In terms of handling large data volumes, MSK has performed well without any significant issues compared to other services I've used.

I highly recommend AWS to any company or team. It offers fast delivery, easy deployment for new features, and a clear separation of concerns. The interface is user-friendly and requires minimal worry. 

I would rate it around an eight out of ten due to initial setup challenges that are overcome with familiarity. 

I use the solution in my company for our CDC pipelines to asynchronously collect data from databases and systems.

I use the solution in my company for our CDC pipelines to asynchronously collect data from databases and systems.

Horizontal scale-out is actually not easy, making it an area where improvements are required.

There is a limitation on the size of the message that you can put and it needs to be increased.

I have been using Amazon MSK for four years. My company is a customer of the tool. My organization uses Amazon MSK quite a lot. The tool also uses Apache Kafka 2.12.

The solution's technical support was helpful. I rate the technical support a seven out of ten.

Neutral

Amazon MSK's features that have proven most effective for handling large volumes of data stem from the fact that it acts as Kafka's underlying capability to support millions of rides per minute, and it is what helps us scale very easily.

Amazon MSK's scalability impacts our data processing tasks, and one thing for sure is that it's a highly scalable system. Usually, we don't see any challenges because it supports millions of rides per minute with Kafka underlying. One challenge is if you want to scale it out by adding new nodes or brokers.

People should finalize the number of partitions before they start using MSK and create a topic because changing partitions is quite challenging.

I rate the tool an eight out of ten.

We use the software to facilitate building integrations between systems.

Amazon MSK has significantly improved our organization by building seamless integration between systems.

The product's schema support needs enhancement. It will help enhance integration with many kinds of languages of programming languages, especially for environments using languages like .NET. They should build schema access as efficiently as developed by Confluent Cloud.

We have been using Amazon MSK for two years. At present, we use the latest version.

I rate the product's stability a nine out of ten.

I rate the product's scalability an eight out of ten.

The technical support services need improvement.

Positive

The software can be deployed on the cloud and has a simple setup process.

Amazon MSK generates a return on investment in terms of reducing integration time.

The platform has better pricing than one of its competitors.

We evaluated Confluent as well. However, we decided to go with Amazon MSK due to its good pricing.

I advise others to understand the exact business requirements before purchasing. I rate Amazon MSK a six out of ten.

My use cases were primarily in the banking domain, where we tracked real-time transactions.

Apart from real-time transactions, we also had another project in the retail domain. Our client's name was McDonolad, and their focus was on package delivery. By "package," I mean the customized burgers created at their restaurant that needed to be delivered to various restaurant locations. When customers visit their restaurant, they would order their personalized burgers. Sometimes, customers would request specific ingredients that were not available in the pre-made burgers displayed at the counter. The restaurant faced the challenge of understanding why customers were not opting for ready-made burgers or patties. To address this issue, they aimed to enhance their product by offering customized burgers. These customized burgers had to be prepared according to the customer's specific preferences. To achieve this, they started taking orders from customers. 

However, it was difficult to determine the exact taste or requirements the customers were looking for. To overcome this, they decided to transfer all the customer's order data to the Kafka server for streaming. This allowed the restaurant's backend system to analyze the data and understand the customer's new tastes and requirements. By collecting this data, they were able to enhance their product and make necessary changes to improve customer satisfaction.

Using Exabeam has provided one valuable aspect, which is the availability of separate agents for Windows and Linux. However, there is a difference when it comes to Linux. On Linux, we need to perform certain command line operations, whereas, for Windows, we have the convenience of an MSI package. One thing I personally feel is that it would be beneficial to have command line installation for Windows as well. 

Additionally, we have to install the package one by one on each server. It would be really helpful if Amazon MSK could provide a single installation that covers all the servers. If there are multiple servers, I would otherwise need to perform individual installations on each one. All of my servers are either VDI or virtual machines, so I have to log in to each virtual machine, copy the necessary files, and then initiate the installation. In future releases, I would like to see if there a probability of having a one-click installation that can be applied to all servers.

I have been using Amazon MSK for more than two years. 

It is a stable product. However, the only problem that we faced was with one of the servers running an older version of Windows.

We faced an issue where the latest package of the log collector or agent collector could not be installed on these older versions.  So at that time only, we connected with DuoSAPI from the IBM side. And he has to go through those issues.

I would rate the scalability a seven out of ten. We faced some issues, but I am not sure what caused those issues. 

Overall, I would rate Amazon MSK a seven out of ten. The only reason I am giving it seven is because I have not used all the aspects of this solution. 

Only thing I would suggest is to go through the documentation before starting anything. Amazon has already provided documentation on its site, which is highly informative. If there are still any issues or questions, users can directly raise a support ticket with the support center. In fact, they may not even need to raise a support ticket because the documentation already covers everything, including instructions on how to install agent collectors on Windows or Linux machines.

Amazon MSK has good integration because our team has been undergoing significant changes. Coupling it with MSK within AWS is helpful. We don't have to set up additionals or monitor external environments. This coupling streamlines our processes, and we don't need to configure any VPCs; it's automatic.

We don't have many use cases involving ingesting large amounts of data and scaling up and down. We have a clear understanding of our data volume, which remains relatively constant throughout the week. While we're aware of other features Amazon MSK offers, we feel confident in our current setup. If our requirements change significantly in the future, we'll reassess our needs and consider adopting Amazon MSK.

I rate the solution’s stability an eight out of ten.

The solution have some scalability issues, but we are trying to develop a new platform.

Mostly data science team uses it. 20 users are using this solution.

I rate the solution’s scalability an eight or nine out of ten.

Customer service have a very good response time.

Amazon MSK is very easy to use because I already had the setups and brokers in place. Obtaining all the necessary information was effortless through AWS commands. So, from a user perspective, it was very straightforward.

The solution is expensive.

We evaluated Confluent and MSK. We wanted everything in the same architecture and trying to connect and schema to the receiver was painful. Confluent has started to move towards their own registry, but until then it was difficult.

Maintenance is pretty low. There's only one person needed for the initial setup. We wanted to build something on our own instead of using an already existing solution because it's quite expensive.

If you want to build something like a custom data platform, MSK is very easy to use. It offers multiple products. For example, you can add multiple features like Glue, streaming, and more. So, it's very easy to plug in and use.

I rate the solution an eight out of ten.