Amazon MSK Reviews

We used Amazon MSK to manage high-volume third-party data entering our system. It served as a buffer when our system was unable to consume data at high speeds in real-time. The data initially went to Kafka and was then consumed by multiple consumers. One consumer sent data to the master database, and others sent it to related applications for further processing.

It provides installations, scaling, and other functionalities straight out of the box. The scalability and usability are quite remarkable. The ability to handle everything, including installation, with minimal human intervention, is a strong point. Stability is another valuable feature, with no notable latency or performance issues encountered.

We use Amazon MSK for communication and notification purposes, such as notifying users about new game releases and managing customer support interactions, thus handling large data volumes.

Amazon MSK's separation of concerns and ease of creating and deploying new features are highly valuable. It just requires assigning them to the topic, and then anyone who needs to consume these messages can do so directly from Amazon MSK. This separation of concerns makes it very convenient, especially for new feature development, as developers can easily access the messages they need without having to deal with complex server communications or protocol setups.

I have used Confluent Cloud and Amazon MSK in my company. We are not using it for analytics and it is more for CDC processes, so we change the capture processes. It is used to extract data from a database and make it available in other parts of our systems or produce events that inform us of data updates.

From AWS, I would consider more MSK schema validation is needed. It is easy to integrate if you have an application, but on-topic integration is more complex. You can do it with EvenBridge very easily, but not with MSK. One of the reasons why we prefer Kafka is because the support is a little bit difficult to manage with Amazon MSK. You have Azure and AWS there, but if you use a connector from the market, the support model is marketplace-based, which means you have to go to the developer of this connector.

Amazon MSK has good integration because our team has been undergoing significant changes. Coupling it with MSK within AWS is helpful. We don't have to set up additionals or monitor external environments. This coupling streamlines our processes, and we don't need to configure any VPCs; it's automatic.

We don't have many use cases involving ingesting large amounts of data and scaling up and down. We have a clear understanding of our data volume, which remains relatively constant throughout the week. While we're aware of other features Amazon MSK offers, we feel confident in our current setup. If our requirements change significantly in the future, we'll reassess our needs and consider adopting Amazon MSK.

I use the solution in my company for our CDC pipelines to asynchronously collect data from databases and systems.

I use the solution in my company for our CDC pipelines to asynchronously collect data from databases and systems.

We use the software to facilitate building integrations between systems.

Amazon MSK has significantly improved our organization by building seamless integration between systems.

My use cases were primarily in the banking domain, where we tracked real-time transactions.

Apart from real-time transactions, we also had another project in the retail domain. Our client's name was McDonolad, and their focus was on package delivery. By "package," I mean the customized burgers created at their restaurant that needed to be delivered to various restaurant locations. When customers visit their restaurant, they would order their personalized burgers. Sometimes, customers would request specific ingredients that were not available in the pre-made burgers displayed at the counter. The restaurant faced the challenge of understanding why customers were not opting for ready-made burgers or patties. To address this issue, they aimed to enhance their product by offering customized burgers. These customized burgers had to be prepared according to the customer's specific preferences. To achieve this, they started taking orders from customers. 

However, it was difficult to determine the exact taste or requirements the customers were looking for. To overcome this, they decided to transfer all the customer's order data to the Kafka server for streaming. This allowed the restaurant's backend system to analyze the data and understand the customer's new tastes and requirements. By collecting this data, they were able to enhance their product and make necessary changes to improve customer satisfaction.

We chose MSK over Kinesis Data because it was mainly cost driven based on volumes. So the amount of workflow that flows through MSK compared to Kinesis, the Kafka clusters were much cheaper. Overall, it is very cost-effective based on the workflow. But it will be cost-effective to connect to the support for a smaller workload.