Apache Kafka Reviews

We utilize Apache Kafka in several areas, including financials, logistics, and client management to name a few.

We used to lose some of our messages when we integrated them in bulk, this solution has stopped that happening.

It is a useful way to maintain messages and to manage offset from our consumers. 

I would like to see an improvement in authentication management.

We have been using the solution for around four years.

The stability is good; the solution operates on our clusters without a big impact.

It is easy to scale.

We used to use a different solution, but our increased throughput meant we needed a product that would allow for a larger queue.

The initial setup was complex for us because we built it internally. This meant that full deployment took around a month.

The implementation was carried out in-house.

I would recommend that other businesses do the deployment themselves, but manage the tool with the aid of a service provider, rather than in-house.

I would rate this product seven out of ten.

Our primary use case of this solution is for data integration and for real-time data consumption. I'm a senior staff engineer for data and infrastructure and we are customers of Apache. 

I love the simplicity of the library and the design as well as the architectural concept which is like horizontal scaling.

When compared to other commercial competitors, Kafka doesn't have the ability to scale down, the elasticity is lacking in the product. The other issue for us is the delayed queue, which was available to us in the commercial software but not in Kafka. It's something we use in most of our applications for deferred processing and I know it's available in other solutions. I'd like to see some tooling support and language support in the open source version. 

I've been using this solution for four years.

The stability is good. 

The solution scales horizontally and scales better than its competitors. We have around 400 to 500 microservices consuming this cluster and the company has around 600 employees. We have four different verticals, each with around 100 engineers with 100 to 150 microservices. 90% of the microservices have a touchpoint with Kafka.

I think the community is very good and will respond if you raise a ticket. We also use external third-party libraries that were built in GitHub. It would be good to have some direct support from Apache.

Four years ago we were using Rabbit MQ but we switched to Kafka because Rabbit was designed for a very narrow use case. It became difficult for us to run and maintain that server and our client libraries. We had a huge outage, so we shifted to Kafka because of the simplicity in the architecture.

The initial setup was simple although we had a couple of hiccups. It took around a week but that was several years ago and we haven't had any problems since. Our team carried out the deployment and we currently have a few engineers who deal with maintenance. 

We are currently using the open-source version. 

There is room for improvement with this solution so I rate it eight out of 10. 

Apache Kafka is a messaging solution where you have topics to pass on your information. You can send messages to multiple topics.

We need to manage limited resources. Additionally, we can send or push messages through a specified port. This is a significant feature because, unlike traditional queues, Kafka uses a cluster of nodes, making it easy to integrate with various algorithms. This clustering is an advantage and a key feature of Kafka, providing good interaction and scalability.

For example, when you want to send a message to inform all your clients about a new feature, you can publish that message to a single topic in Apache Kafka. This allows all clients subscribed to that topic to receive the message. On the other hand, if you need to send billing information to a specific customer, you can publish that message on a topic dedicated to that customer. This message can then be sent as an SMS to the customer, allowing them to view it on their mobile device.

Apache Kafka is different in its design. If you have topics around the front end of clusters in the facility, it is scalable. The software is scalable to handle and process data. However, it might not be suitable for handling specific types of images or media files. Other than that, it should handle the rest of the data processing needs.

There are no multiple versions, which simplifies the process of granting access with Kaspersky. Every message is accurately delivered. However, Kafka does not support sending messages directly. You need to publish messages finalization. If you want to resend a message, you must resend it manually. Kafka does not automatically handle this. Another thing is the need for a redo option if an issue occurs. If a message is not sent properly, it can be retransmitted within the core system. You should enable the gateway in your program for it to function correctly. Messages will not be delivered or refreshed unless you enable the direct replay option in the product settings.

I have been using Apache Kafka since 2020-21

The initial setup of Apache Kafka is challenging and requires experience. Each message should always receive a response, so prioritizing traffic is essential. Furthermore, the client or consumer must always be in sync, or the message will not be processed.

One pair of nodes is sufficient for the system. If our other system requires more than five nodes, it might not be feasible. Currently, other components are functioning as expected. The Kafka setup won't take much time.

When using Apache Kafka, it’s important to manage different environments carefully to avoid confusion. For instance, you can configure different client applications for producing and consuming messages. Ensure that the configurations for each environment (development, testing, production, etc.) are separated. This includes managing source code and data appropriately to maintain security and efficiency. Proper management of Kafka assets and operations phases is crucial for a smooth workflow.

I recommend Apache Kafka since it is extremely fast, stable and has been used for a very long time. We haven't encountered any major issues or concerns regarding its performance and customer service.

Overall, I rate the solution a nine out of ten.

We use Kafka for Elastic Stack and Kafka SCRAM login.

I have many users of Apache Kafka. It's like a subject to study in enterprises. However, we have not decided if the systems should generalize Apache Kafka for every application and every IT system.

We use Kafka for mapping and ThoughtSpot data from one IT system source to the destination. We also prefer it to exchange data from our internal IT systems.

Kafka is a new method we opted to apply to our need for data exchange. Also, we use the solution's integration capabilities.

Irovement-wise, I would like the solution to have more integration capabilities. Also, the solution's setup, which is currently complex, should be made easier.

I have experience with Apache Kafka.

It is a stable solution.

A lot of my experience indicates that Apache Kafka is scalable. We can have ten or even fifty hundred users on the solution. So, it's possible because we are a big enterprise.

I have experience with Apache Kafka's technical support.

The solution's initial setup process was complex. The deployment process took three or four years.

Right now, I can't deliver the planning process required for deployment.

For deployment and maintenance, we have a manager and an operational person. However, I can't give an exact count of the people required for deployment and maintenance.

To be able to recommend Kafka to others, especially considering every context, we will have to set a benchmark and compare Kafka with other tools.

I rate the overall solution a seven out of ten.

My primary use case for Apache Kafka is replacing ETL and doing data transformations.

Kafka allows you to handle huge amounts of data and classify it into different categories. If you have huge amounts of data, Kafka is a very good solution for data classification. When you need to route it in different directions, you have to take a look at the messages that you get, interfile them, and then send them to the correct place. Kafka is a good product to use in the backend.

The feature I find most valuable is the classification feature. Kafka enables you to tag content with a category.

Kafka contains two components. The component that does the synchronization between the rest of the components, that's an older version of the software and it causes all kinds of configuration problems. The Confluent, which is the company that sells a commercial version of Kafka is getting away from that component precisely because of that. Kafka is a nightmare to administer.

In the next release, I would like to see that one troublesome component that causes configuration issues removed.

I have been using Apache Kafka for a couple of years.

The stability of this solution depends on whether it is properly configured. Having said that, Kafka is incredibly complex to configure, set up, administer, and maintain.

My opinion is that Apache Kafka is a scalable solution. In our organization, there are hundreds of thousands of users using Kafka.

The initial setup was extremely complex. In our case, it took a team of 12 two months to deploy.

These systems were installed by somebody else, not me.

I would advise others to schedule a month or two to just set it up and have it up and running.

There are other options. For example, Databricks is a Kafka alternative. We decided to go with Kafka because one of our clients already chose Kafka.

While evaluating, we found out Databricks is more expensive, for the level of activity that Kafka handles (in this case, millions of requests per day). Databricks could do it, but it would be overly expensive.

I would rate Apache Kafka's pricing a seven out of ten, with one being cheap and 10 being very expensive.

Since it has become so popular, large enterprises especially want to do it. For smaller enterprises, Kafka would probably be too expensive because they would have to hire people to maintain it.

I would rate the Apache Kafka solution a seven out of ten.

We have got this product, which is meant for integration. So our use cases are essentially integrating with other systems, using any messaging stack. We use these products in Dev and QA and we have connectors for various different messaging applications. Apache Kafka just happens to be one of the messaging applications that we connect with. We also have our own messaging, it's called Enterprise Messaging Server and Rendezvous, we connect to those also. Our product is essentially used for integration. So we connect to almost all messaging applications.

The most valuable feature is the speed at which the solution can be deployed.

The solution can improve its cloud support.

I have been using the solution in Dev and QA for a few years.

The solution is stable.

The solution is scalable.

Since we are supporting various different messaging applications, we tend to use and support all the messaging applications that are popular. Like SQS, Google pops up, Active MQ, Rapid MQ, MQTT, and IBM MQ.

The open-source version is relatively straightforward to set up and only takes a few minutes.

We typically implement the solution in-house.

The solution is open source.

I give the solution an eight out of ten.

We test all the supported versions of the solution based on our customers' use.

We support our integration product. So we need to do dev and QA with Apache Kafka or any other messaging applications. But we do not provide support. The solution can be supported by someone else.

We don't need to have any specific staff for deployment. All the developers in QA can install and configure the solution. We don't have a separate person for maintenance.

Our team and our product dev and QAs all use the solution.

I think Apache Kafka is a good solution and I recommend it to others.

I primarily use Kafka in the investment banking sector to update prices and inform clients of updates.

Kafka's most valuable feature is its user-friendliness.

There are some latency problems with Kafka.

I've been using Kafka for more than three years.

I would give Kafka a rating of seven out of ten.

Apache Kafka is used for more than only a messaging bus but also served as a database to store information. It functioned as a streamer, similar to ETL, to manipulate and transform events before migrating them to other systems for use. The database could also act as a cache. Apache Kafka is used as a database broker, streamer, and source of truth for multiple systems due to its ability to maintain events for at least 10 days. It provided both synchronous and asynchronous communication, making it a complex system that would be easier to understand through diagrams or sketches.

We use reactive frameworks.

From my experience with Apache Kafka, one of the most notable advantages is its ability to maintain a comprehensive record of historical data that includes every update, alteration, and version of information, unlike a conventional relational database. This feature allows for seamless tracking and analysis of the progression and transformation of the data over time, enabling users to easily review and analyze the history of the information.

The solution has the capability for various systems to effortlessly interact with one another without prior knowledge of their existence, current operational status, or specific configurations. By utilizing service buses and dynamic integration, data can be distributed across networks and retrieved in a way that is most suitable for each system's requirements. In addition, Apache Kafka allows for the modification of data to provide diverse clients, consumers, or observers with unique and varying data. The replication of data can produce multiple versions, and this data can be adjusted to fit various needs. With the use of probes, one can alter the behavior of the transformation process, thereby changing the way in which data is transformed and the output produced. Overall, working with Apache Kafka has brought about an array of benefits, enabling seamless system interactions and allowing for the customization and modification of data to meet individual requirements.

There are numerous possibilities that can be explored. While it may be challenging to fully comprehend the potential advantages, one key aspect is the ability to establish a proper sequence of events rather than simply dealing with a jumbled group of occurrences. These events possess their own timestamps, even if they were not initially provided with one, and are arranged in a chronological order that allows for a clear understanding of the progression of the events.

There have been some challenges with monitoring Apache Kafka, as there are currently only a few production-grade solutions available, which are all under enterprise license and therefore not easily accessible. The speaker has not had access to any of these solutions and has instead relied on tools, such as Dynatrace, which do not provide sufficient insight into the Apache Kafka system. While there are other tools available, they do not offer the same level of real-time data as enterprise solutions.

One additional area that I think could benefit from improvement is the deployment process on OpenShift. This particular deployment is quite challenging and requires the activation of certain security measures as well as integration with other systems. It's not a straightforward process and typically requires engineers who are highly skilled and have extensive experience with Apache Kafka to carry out these tasks. Therefore, I believe that there is a need for progress in this area, and some tools that can provide information, assistance, and help make the whole process easier would be greatly appreciated.

I have been using Apache Kafka for approximately four years.

The solution is stable if you have set it up correctly.

Apache Kafka is a scalable solution.

I have not escalated any questions to technical support because Apache Kafka is an open-source system. However, Confluent and other companies sell support and enterprise solutions to make it more convenient and streamline the work. They offer tools, such as a monitoring tool with a visual interface, which provides a lot of information and buttons to press for correction or change without touching the code. Each of those buttons hypothetically could have helped the situation, but it is unclear what they do exactly, it is best to call the data center and ask. If you buy their service, you have access to all the enterprise comforts.

Setting up Apache Kafka is, is not an easy task, especially when trying to containerize it and make it controllable. This is because Apache Kafka has its own distributed mechanism for staying alive, checking readiness, replicating, and scaling. Ensuring that it complies with Kubernetes or OpenShift Orchestrator requires careful attention, as there is a risk of two masters attempting to perform the same task and ultimately undoing each other's work.

In comparison to Kubernetes, OpenShift is a highly skilled and advanced implementation infrastructure that automatically manages and orchestrates all the steps required for an application setup. It operates at a higher level of abstraction and eliminates the need for manual operations that are required with Kubernetes. While Kubernetes can run an application with some pipeline and configuration, OpenShift takes care of everything from finding the required images to creating ports and connecting databases. Although manual changes can be made, it's not necessary as OpenShift offers a much more course-grained management approach.

One skillful DevOps engineer can implement the solution.

Apache Kafka is an open-source solution.

The maintenance of Apache Kafka is crucial due to the complexity of the system with numerous microservices and systems communicating through Apache Kafka, requiring proper integration and configuration to prevent overloading and ensure a healthy cluster. The task is not easy and requires knowledge of the various adjustable parameters, as misadjusting even one of them can greatly slow down the cluster. For example, if the consumer group changes frequently, the messages must be regrouped and reassigned, causing significant delays. Therefore, configuring Apache Kafka correctly is essential to avoid high latency issues.

I would strongly suggest others give Apache Kafka a chance and explore the various advantages that it can offer, especially since it should not be perceived as a message bus or broker but rather an enterprise bus designed for data manipulation. It has the ability to transform data, store and reject it, and even maintain different versions of the same data simultaneously. Moreover, it operates on a pull mechanism rather than a push mechanism, which takes away the risk of losing data and places the responsibility for data loss on the consumer. On the other hand, it also ensures that the data is always available within the specified window and allows for easy replication of the past, which is extremely helpful in situations such as those involving a hacked bank database. With Apache Kafka, you can efficiently go back in time, obtain the required status and events, and make changes accordingly, without the need to go through each transaction separately. Thus, using this solution can make data management much more efficient and convenient.

I rate Apache Kafka an eight out of ten.

In order to improve its user-friendliness, engineer-friendliness, and DevOps-friendliness, the system must undertake various tasks, such as enhancing the overall operation and configuration, ensuring seamless integration with other systems, and adapting to security layers in a more comprehensive and generic manner. This will require significant efforts to make the system more functional, secure, and efficient.