Apache Kafka Reviews

I worked with Apache Kafka for customers in the financial industry and OTT platforms. They use Kafka particularly for data streaming. Companies offering movie and entertainment as a service, similar to Netflix, use Kafka.

Apache Kafka offers unique data streaming. It allows the use of data in motion, allowing data to propagate from one source to another while it is in motion. This is valuable when data is not simply residing in a database.

We primarily use Kafka for EventStream. We publish messages into the Kafka queues, and then at the other end, we consume those messages to perform tasks.

We have seen a return on investment with respect to improvements in productivity and ease of use, although these benefits are not easily quantified in dollar terms.

My company uses Apache Kafka to keep some intermediate data in the workflow.

Apache Kafka's most valuable features include clustering and sharding. Though we have not started using Apache Kafka Streams in our company, I have heard that it is one of the good features of the product we plan to use. The good features let you replay and reconsume messages in Apache Kafka, allowing you to have multiple consumer groups. The rebalancing feature of the product is also useful since if one consumer dies, then Apache Kafka does a rebalancing. With Apache Kafka, we use clustering, sharding, and partitioning features in our company.

We have different use cases for different clients. For example, a banking sector client wants to use Apache Kafka for fraud detection and messaging back to the client. For instance, if there's a fraudulent swipe of a credit or debit card, we stream near real-time data (usually three to five minutes old) into the platform. Then, we use a look-up model to relay the data to the messaging queue to the customers. This is just one use case.

We also have data science engineers who use Kafka to feed on the data (usually within the last five to seven minutes of transactions) to detect any fraudulent transactions for internal consumption of the bank. This is not for relaying back to the customer. This client is based in the Middle East, and they have some interesting use cases.

We are still in the cluster database phase. Based on the use cases captured during the advisory phase, there will be a mix of 40 to 60% of users. 40% will be internal data science and IT teams, while 60% will be end users. So, the total number of users we have seen is 25. Out of these, around 15 business users will make decisions based on reports generated by Kafka analytic data. The remaining users are internal, who analyze this data daily to identify more use cases from a predictive and AI perspective for the future banking domain.

Moreover, our current client is an enterprise business. It is a globally renowned bank that has entered Saudi.

We use Kafka for a stage event-driven process from a process perspective. Our platform is an ID platform, so after registration data is received, it has to be stored from various registration locations. The process includes stages like quality checking, consistency, format, biometric data checking, and so on. We are using both Kafka and ActiveMQ for almost three years now.

Using Kafka has saved us costs compared to proprietary solutions. It has allowed for significant customizations internally, which would have been much costlier with other solutions. We have saved a lot by employing Kafka and only needing basic salary agreements for our experts.

We have multiple use cases for our Kafka system. One is Kafka Connect, which is used to facilitate communication between different regions with Grid Deal. Another is to distribute events and projects to multiple downstream. We publish all the messages to Kafka and other listeners subscribe and write them to different MQs. Lastly, Kafka Connect is used especially for inter-application communication.

We had been using a lot of expensive licenses earlier, such as SOLEIL, as well as some legacy versions, which were not only costly but also caused memory issues and required highly technical personnel to manage. This posed a huge challenge in terms of resourcing and cost, and it simply wasn't worth investing more in. However, Kafka was comparatively free as it was open source, and we were able to build our own monitoring system on top of it. Kafka is an open-source platform that allows us to develop modern solutions with relative ease. Additionally, there are many resources available in the market to quickly train personnel to work with this platform. Kafka is user-friendly and does not require an extensive learning curve, unlike other tools. Furthermore, the configuration is straightforward. All in all, Kafka provides us with a great platform to build upon with minimal effort.

Apache Kafka can be deployed on the cloud and on-premise.

We use Apache Kafka internally to build a service on a cluster. Additionally, we use the intermediate persistence layer for events. There are many teams who leverage it as a message queue and further their microservice connections.

Apache Kafka has helped out the organization because we leverage it for all our eCommerce real-time analytics use cases.