Apache Kafka Reviews

We use Kafka for Elastic Stack and Kafka SCRAM login.

I have many users of Apache Kafka. It's like a subject to study in enterprises. However, we have not decided if the systems should generalize Apache Kafka for every application and every IT system.

We use Kafka for mapping and ThoughtSpot data from one IT system source to the destination. We also prefer it to exchange data from our internal IT systems.

Kafka is a new method we opted to apply to our need for data exchange. Also, we use the solution's integration capabilities.

Irovement-wise, I would like the solution to have more integration capabilities. Also, the solution's setup, which is currently complex, should be made easier.

I have experience with Apache Kafka.

It is a stable solution.

A lot of my experience indicates that Apache Kafka is scalable. We can have ten or even fifty hundred users on the solution. So, it's possible because we are a big enterprise.

I have experience with Apache Kafka's technical support.

The solution's initial setup process was complex. The deployment process took three or four years.

Right now, I can't deliver the planning process required for deployment.

For deployment and maintenance, we have a manager and an operational person. However, I can't give an exact count of the people required for deployment and maintenance.

To be able to recommend Kafka to others, especially considering every context, we will have to set a benchmark and compare Kafka with other tools.

I rate the overall solution a seven out of ten.

My primary use case for Apache Kafka is replacing ETL and doing data transformations.

Kafka allows you to handle huge amounts of data and classify it into different categories. If you have huge amounts of data, Kafka is a very good solution for data classification. When you need to route it in different directions, you have to take a look at the messages that you get, interfile them, and then send them to the correct place. Kafka is a good product to use in the backend.

The feature I find most valuable is the classification feature. Kafka enables you to tag content with a category.

Kafka contains two components. The component that does the synchronization between the rest of the components, that's an older version of the software and it causes all kinds of configuration problems. The Confluent, which is the company that sells a commercial version of Kafka is getting away from that component precisely because of that. Kafka is a nightmare to administer.

In the next release, I would like to see that one troublesome component that causes configuration issues removed.

I have been using Apache Kafka for a couple of years.

The stability of this solution depends on whether it is properly configured. Having said that, Kafka is incredibly complex to configure, set up, administer, and maintain.

My opinion is that Apache Kafka is a scalable solution. In our organization, there are hundreds of thousands of users using Kafka.

The initial setup was extremely complex. In our case, it took a team of 12 two months to deploy.

These systems were installed by somebody else, not me.

I would advise others to schedule a month or two to just set it up and have it up and running.

There are other options. For example, Databricks is a Kafka alternative. We decided to go with Kafka because one of our clients already chose Kafka.

While evaluating, we found out Databricks is more expensive, for the level of activity that Kafka handles (in this case, millions of requests per day). Databricks could do it, but it would be overly expensive.

I would rate Apache Kafka's pricing a seven out of ten, with one being cheap and 10 being very expensive.

Since it has become so popular, large enterprises especially want to do it. For smaller enterprises, Kafka would probably be too expensive because they would have to hire people to maintain it.

I would rate the Apache Kafka solution a seven out of ten.

We have got this product, which is meant for integration. So our use cases are essentially integrating with other systems, using any messaging stack. We use these products in Dev and QA and we have connectors for various different messaging applications. Apache Kafka just happens to be one of the messaging applications that we connect with. We also have our own messaging, it's called Enterprise Messaging Server and Rendezvous, we connect to those also. Our product is essentially used for integration. So we connect to almost all messaging applications.

The most valuable feature is the speed at which the solution can be deployed.

The solution can improve its cloud support.

I have been using the solution in Dev and QA for a few years.

The solution is stable.

The solution is scalable.

Since we are supporting various different messaging applications, we tend to use and support all the messaging applications that are popular. Like SQS, Google pops up, Active MQ, Rapid MQ, MQTT, and IBM MQ.

The open-source version is relatively straightforward to set up and only takes a few minutes.

We typically implement the solution in-house.

The solution is open source.

I give the solution an eight out of ten.

We test all the supported versions of the solution based on our customers' use.

We support our integration product. So we need to do dev and QA with Apache Kafka or any other messaging applications. But we do not provide support. The solution can be supported by someone else.

We don't need to have any specific staff for deployment. All the developers in QA can install and configure the solution. We don't have a separate person for maintenance.

Our team and our product dev and QAs all use the solution.

I think Apache Kafka is a good solution and I recommend it to others.

We use Apache Kafka to process messages, specifically payment type messages, and incorporate the data from those messages into our analytics and reporting. It utilizes data from additional sources in real-time for our analytics and reporting purposes.

The real-time nature and the ability to use multiple offsets are the most beneficial features of Apache Kafka for our data streaming needs. This allows us to replay the same messages using different offsets. Although I haven't set up Kafka's scalability and fault tolerance myself, I know it can be configured with redundancy and fallback options. We primarily consume the messages using different clients, so the setup for fault tolerance and redundancy is transparent to us.

It's quite flexible and comparable to other solutions like ActiveMQ in terms of features and guarantees, especially with offsets for message handling. While ActiveMQ may be preferred in some use cases requiring guaranteed message delivery, Kafka's offset management provides similar functionality. Overall, I would recommend Kafka for real-time data streaming without hesitation.

The main challenge we faced while integrating Apache Kafka with other tools was setting up SSL and securing connections. Managing certificate changes and ensuring all clients connect smoothly, especially outside Kubernetes environments, posed ongoing challenges. Once initially set up, maintaining and sharing these security configurations became more manageable, but ensuring compatibility across different environments remained a continuous effort.

I have been using Apache Kafka for the last five years.

I would rate the scalability nine out of ten.

The technical support, typical for open-source solutions, is also responsive and helpful.

Positive

We switched to Kafka from paid solutions like IBM's MQ due to cost considerations, finding Kafka's multiple offsets and popularity advantageous.

As for pricing, Kafka is open-source, so it's free to install and use.

I primarily use Kafka in the investment banking sector to update prices and inform clients of updates.

Kafka's most valuable feature is its user-friendliness.

There are some latency problems with Kafka.

I've been using Kafka for more than three years.

I would give Kafka a rating of seven out of ten.

Apache Kafka is used for more than only a messaging bus but also served as a database to store information. It functioned as a streamer, similar to ETL, to manipulate and transform events before migrating them to other systems for use. The database could also act as a cache. Apache Kafka is used as a database broker, streamer, and source of truth for multiple systems due to its ability to maintain events for at least 10 days. It provided both synchronous and asynchronous communication, making it a complex system that would be easier to understand through diagrams or sketches.

We use reactive frameworks.

From my experience with Apache Kafka, one of the most notable advantages is its ability to maintain a comprehensive record of historical data that includes every update, alteration, and version of information, unlike a conventional relational database. This feature allows for seamless tracking and analysis of the progression and transformation of the data over time, enabling users to easily review and analyze the history of the information.

The solution has the capability for various systems to effortlessly interact with one another without prior knowledge of their existence, current operational status, or specific configurations. By utilizing service buses and dynamic integration, data can be distributed across networks and retrieved in a way that is most suitable for each system's requirements. In addition, Apache Kafka allows for the modification of data to provide diverse clients, consumers, or observers with unique and varying data. The replication of data can produce multiple versions, and this data can be adjusted to fit various needs. With the use of probes, one can alter the behavior of the transformation process, thereby changing the way in which data is transformed and the output produced. Overall, working with Apache Kafka has brought about an array of benefits, enabling seamless system interactions and allowing for the customization and modification of data to meet individual requirements.

There are numerous possibilities that can be explored. While it may be challenging to fully comprehend the potential advantages, one key aspect is the ability to establish a proper sequence of events rather than simply dealing with a jumbled group of occurrences. These events possess their own timestamps, even if they were not initially provided with one, and are arranged in a chronological order that allows for a clear understanding of the progression of the events.

There have been some challenges with monitoring Apache Kafka, as there are currently only a few production-grade solutions available, which are all under enterprise license and therefore not easily accessible. The speaker has not had access to any of these solutions and has instead relied on tools, such as Dynatrace, which do not provide sufficient insight into the Apache Kafka system. While there are other tools available, they do not offer the same level of real-time data as enterprise solutions.

One additional area that I think could benefit from improvement is the deployment process on OpenShift. This particular deployment is quite challenging and requires the activation of certain security measures as well as integration with other systems. It's not a straightforward process and typically requires engineers who are highly skilled and have extensive experience with Apache Kafka to carry out these tasks. Therefore, I believe that there is a need for progress in this area, and some tools that can provide information, assistance, and help make the whole process easier would be greatly appreciated.

I have been using Apache Kafka for approximately four years.

The solution is stable if you have set it up correctly.

Apache Kafka is a scalable solution.

I have not escalated any questions to technical support because Apache Kafka is an open-source system. However, Confluent and other companies sell support and enterprise solutions to make it more convenient and streamline the work. They offer tools, such as a monitoring tool with a visual interface, which provides a lot of information and buttons to press for correction or change without touching the code. Each of those buttons hypothetically could have helped the situation, but it is unclear what they do exactly, it is best to call the data center and ask. If you buy their service, you have access to all the enterprise comforts.

Setting up Apache Kafka is, is not an easy task, especially when trying to containerize it and make it controllable. This is because Apache Kafka has its own distributed mechanism for staying alive, checking readiness, replicating, and scaling. Ensuring that it complies with Kubernetes or OpenShift Orchestrator requires careful attention, as there is a risk of two masters attempting to perform the same task and ultimately undoing each other's work.

In comparison to Kubernetes, OpenShift is a highly skilled and advanced implementation infrastructure that automatically manages and orchestrates all the steps required for an application setup. It operates at a higher level of abstraction and eliminates the need for manual operations that are required with Kubernetes. While Kubernetes can run an application with some pipeline and configuration, OpenShift takes care of everything from finding the required images to creating ports and connecting databases. Although manual changes can be made, it's not necessary as OpenShift offers a much more course-grained management approach.

One skillful DevOps engineer can implement the solution.

Apache Kafka is an open-source solution.

The maintenance of Apache Kafka is crucial due to the complexity of the system with numerous microservices and systems communicating through Apache Kafka, requiring proper integration and configuration to prevent overloading and ensure a healthy cluster. The task is not easy and requires knowledge of the various adjustable parameters, as misadjusting even one of them can greatly slow down the cluster. For example, if the consumer group changes frequently, the messages must be regrouped and reassigned, causing significant delays. Therefore, configuring Apache Kafka correctly is essential to avoid high latency issues.

I would strongly suggest others give Apache Kafka a chance and explore the various advantages that it can offer, especially since it should not be perceived as a message bus or broker but rather an enterprise bus designed for data manipulation. It has the ability to transform data, store and reject it, and even maintain different versions of the same data simultaneously. Moreover, it operates on a pull mechanism rather than a push mechanism, which takes away the risk of losing data and places the responsibility for data loss on the consumer. On the other hand, it also ensures that the data is always available within the specified window and allows for easy replication of the past, which is extremely helpful in situations such as those involving a hacked bank database. With Apache Kafka, you can efficiently go back in time, obtain the required status and events, and make changes accordingly, without the need to go through each transaction separately. Thus, using this solution can make data management much more efficient and convenient.

I rate Apache Kafka an eight out of ten.

In order to improve its user-friendliness, engineer-friendliness, and DevOps-friendliness, the system must undertake various tasks, such as enhancing the overall operation and configuration, ensuring seamless integration with other systems, and adapting to security layers in a more comprehensive and generic manner. This will require significant efforts to make the system more functional, secure, and efficient.

The primary use case of the solution is for asset communication through our microservices.

The solution has allowed us to take the use cases provided by another communication tool and resolve those issues.

The most valuable feature is how persistent it is. For example, we are able to reprocess messages when we need to, we're able to recover methods to consume them.

The solution can be improved by reducing the cost to run it on the premises.

I have been using the solution for four years.

The stability of the solution is good.

The solution is scalable.

The initial setup was straightforward.

The implementation was through a vendor.

I have seen a return on investment with this solution.

I give the solution a nine out of ten.

We have 80 people using the solution and five people are required to maintain it.

I suggest using cloud services because the solution is expensive if you are using it on-premises.

We utilize Apache Kafka in several areas, including financials, logistics, and client management to name a few.

We used to lose some of our messages when we integrated them in bulk, this solution has stopped that happening.

It is a useful way to maintain messages and to manage offset from our consumers. 

I would like to see an improvement in authentication management.

We have been using the solution for around four years.

The stability is good; the solution operates on our clusters without a big impact.

It is easy to scale.

We used to use a different solution, but our increased throughput meant we needed a product that would allow for a larger queue.

The initial setup was complex for us because we built it internally. This meant that full deployment took around a month.

The implementation was carried out in-house.

I would recommend that other businesses do the deployment themselves, but manage the tool with the aid of a service provider, rather than in-house.

I would rate this product seven out of ten.