Apache Kafka Reviews

Apache Kafka is used for more than only a messaging bus but also served as a database to store information. It functioned as a streamer, similar to ETL, to manipulate and transform events before migrating them to other systems for use. The database could also act as a cache. Apache Kafka is used as a database broker, streamer, and source of truth for multiple systems due to its ability to maintain events for at least 10 days. It provided both synchronous and asynchronous communication, making it a complex system that would be easier to understand through diagrams or sketches.

We use reactive frameworks.

From my experience with Apache Kafka, one of the most notable advantages is its ability to maintain a comprehensive record of historical data that includes every update, alteration, and version of information, unlike a conventional relational database. This feature allows for seamless tracking and analysis of the progression and transformation of the data over time, enabling users to easily review and analyze the history of the information.

The solution has the capability for various systems to effortlessly interact with one another without prior knowledge of their existence, current operational status, or specific configurations. By utilizing service buses and dynamic integration, data can be distributed across networks and retrieved in a way that is most suitable for each system's requirements. In addition, Apache Kafka allows for the modification of data to provide diverse clients, consumers, or observers with unique and varying data. The replication of data can produce multiple versions, and this data can be adjusted to fit various needs. With the use of probes, one can alter the behavior of the transformation process, thereby changing the way in which data is transformed and the output produced. Overall, working with Apache Kafka has brought about an array of benefits, enabling seamless system interactions and allowing for the customization and modification of data to meet individual requirements.

There are numerous possibilities that can be explored. While it may be challenging to fully comprehend the potential advantages, one key aspect is the ability to establish a proper sequence of events rather than simply dealing with a jumbled group of occurrences. These events possess their own timestamps, even if they were not initially provided with one, and are arranged in a chronological order that allows for a clear understanding of the progression of the events.

There have been some challenges with monitoring Apache Kafka, as there are currently only a few production-grade solutions available, which are all under enterprise license and therefore not easily accessible. The speaker has not had access to any of these solutions and has instead relied on tools, such as Dynatrace, which do not provide sufficient insight into the Apache Kafka system. While there are other tools available, they do not offer the same level of real-time data as enterprise solutions.

One additional area that I think could benefit from improvement is the deployment process on OpenShift. This particular deployment is quite challenging and requires the activation of certain security measures as well as integration with other systems. It's not a straightforward process and typically requires engineers who are highly skilled and have extensive experience with Apache Kafka to carry out these tasks. Therefore, I believe that there is a need for progress in this area, and some tools that can provide information, assistance, and help make the whole process easier would be greatly appreciated.

I have been using Apache Kafka for approximately four years.

The solution is stable if you have set it up correctly.

Apache Kafka is a scalable solution.

I have not escalated any questions to technical support because Apache Kafka is an open-source system. However, Confluent and other companies sell support and enterprise solutions to make it more convenient and streamline the work. They offer tools, such as a monitoring tool with a visual interface, which provides a lot of information and buttons to press for correction or change without touching the code. Each of those buttons hypothetically could have helped the situation, but it is unclear what they do exactly, it is best to call the data center and ask. If you buy their service, you have access to all the enterprise comforts.

Setting up Apache Kafka is, is not an easy task, especially when trying to containerize it and make it controllable. This is because Apache Kafka has its own distributed mechanism for staying alive, checking readiness, replicating, and scaling. Ensuring that it complies with Kubernetes or OpenShift Orchestrator requires careful attention, as there is a risk of two masters attempting to perform the same task and ultimately undoing each other's work.

In comparison to Kubernetes, OpenShift is a highly skilled and advanced implementation infrastructure that automatically manages and orchestrates all the steps required for an application setup. It operates at a higher level of abstraction and eliminates the need for manual operations that are required with Kubernetes. While Kubernetes can run an application with some pipeline and configuration, OpenShift takes care of everything from finding the required images to creating ports and connecting databases. Although manual changes can be made, it's not necessary as OpenShift offers a much more course-grained management approach.

One skillful DevOps engineer can implement the solution.

Apache Kafka is an open-source solution.

The maintenance of Apache Kafka is crucial due to the complexity of the system with numerous microservices and systems communicating through Apache Kafka, requiring proper integration and configuration to prevent overloading and ensure a healthy cluster. The task is not easy and requires knowledge of the various adjustable parameters, as misadjusting even one of them can greatly slow down the cluster. For example, if the consumer group changes frequently, the messages must be regrouped and reassigned, causing significant delays. Therefore, configuring Apache Kafka correctly is essential to avoid high latency issues.

I would strongly suggest others give Apache Kafka a chance and explore the various advantages that it can offer, especially since it should not be perceived as a message bus or broker but rather an enterprise bus designed for data manipulation. It has the ability to transform data, store and reject it, and even maintain different versions of the same data simultaneously. Moreover, it operates on a pull mechanism rather than a push mechanism, which takes away the risk of losing data and places the responsibility for data loss on the consumer. On the other hand, it also ensures that the data is always available within the specified window and allows for easy replication of the past, which is extremely helpful in situations such as those involving a hacked bank database. With Apache Kafka, you can efficiently go back in time, obtain the required status and events, and make changes accordingly, without the need to go through each transaction separately. Thus, using this solution can make data management much more efficient and convenient.

I rate Apache Kafka an eight out of ten.

In order to improve its user-friendliness, engineer-friendliness, and DevOps-friendliness, the system must undertake various tasks, such as enhancing the overall operation and configuration, ensuring seamless integration with other systems, and adapting to security layers in a more comprehensive and generic manner. This will require significant efforts to make the system more functional, secure, and efficient.

Our primary use cases allow software developers, and application developers, the option to not have to code in their own logic for the retry mechanism. A lot of software, and applications, have this feature of retry built in some way or the other, but they all have some kind of a pre-alpha version of Kafka, up to a certain extent.

So it is a good backbone for microservices. So basically you want to write microservices, which you can shut down and bring it up whenever you want. You want to be able to shut it down to actually replace it with a newer version and bring it up. The bottom line is you can kill the microservice and bring it back up and do all the things that you want to do with it. But whenever it comes back up, it should pick up and run from where it had left off. That is what everybody tries to do. And in order to build such a system, they have to write several logical pieces of code, and most of that code has already been built for in Kafka so that you don't have to do it yourself.

One of the most valuable features I have found is Kafka Connect.

Basically, the bootup time, if you have large messages, sometimes takes up more time than I would really like it to. So that is the area that Kafka can actually improve upon. But that is okay, the way we get around it is to make sure that Kafka has started up first and warmed up before anything else starts up. I would also like to see monitoring service tools.

I have been using Apache Kafka for the past three years.

The stability is good as long as you have a short retention period.

Confluent is the cloud version of Apache Kafka and it is scalable.

We do the implementation in-house.

If you are managing your own implementation the return on investment is pretty good. What you need is good developers.

Apache Kafka has open-source pricing.

Apache Kafka is a good choice, so I would recommend people not have a real-time application if they do not have to. It is better to have a very fast batch operation than a real-time operation. I would rate Apache Kafka a nine on a scale of one to ten.

My primary use case for Apache Kafka is replacing ETL and doing data transformations.

Kafka allows you to handle huge amounts of data and classify it into different categories. If you have huge amounts of data, Kafka is a very good solution for data classification. When you need to route it in different directions, you have to take a look at the messages that you get, interfile them, and then send them to the correct place. Kafka is a good product to use in the backend.

The feature I find most valuable is the classification feature. Kafka enables you to tag content with a category.

Kafka contains two components. The component that does the synchronization between the rest of the components, that's an older version of the software and it causes all kinds of configuration problems. The Confluent, which is the company that sells a commercial version of Kafka is getting away from that component precisely because of that. Kafka is a nightmare to administer.

In the next release, I would like to see that one troublesome component that causes configuration issues removed.

I have been using Apache Kafka for a couple of years.

The stability of this solution depends on whether it is properly configured. Having said that, Kafka is incredibly complex to configure, set up, administer, and maintain.

My opinion is that Apache Kafka is a scalable solution. In our organization, there are hundreds of thousands of users using Kafka.

The initial setup was extremely complex. In our case, it took a team of 12 two months to deploy.

These systems were installed by somebody else, not me.

I would advise others to schedule a month or two to just set it up and have it up and running.

There are other options. For example, Databricks is a Kafka alternative. We decided to go with Kafka because one of our clients already chose Kafka.

While evaluating, we found out Databricks is more expensive, for the level of activity that Kafka handles (in this case, millions of requests per day). Databricks could do it, but it would be overly expensive.

I would rate Apache Kafka's pricing a seven out of ten, with one being cheap and 10 being very expensive.

Since it has become so popular, large enterprises especially want to do it. For smaller enterprises, Kafka would probably be too expensive because they would have to hire people to maintain it.

I would rate the Apache Kafka solution a seven out of ten.

We used Kafka as a central message bus, transporting data from SNMP through to a database. Some of the processing in between was handled by other components.

We built a solution for a client and the client was happy with the solution.

The message bus capabilities, basically sending messages to it, and the way it handles events or messages is pretty good. The throughput was good. Generally, it was a good component.

We struggled a bit with the built-in data transformations because it was a challenge to get them up and running the way we wanted. There was a bit of a learning curve. It may be that we didn't fully grasp the information.

Also, the documentation covering certain aspects was a bit poor. We had to trawl around different locations to try to find what we needed. When we were able to find documentation on transformation, for example, there wasn't a good set of documentation examples we could use, and the examples we had weren't quite meeting the need. Better examples would've helped us.

I used this solution for about a year and a half. 

It seemed pretty stable and didn't have any issues at all.

I don't know how many people were using it on the client's side, but we had a four-person team doing the development work. 

Our team handled the deployment in-house.

Kafka is an open-source solution, so there are no licensing costs. There are third-party companies who support and provide add-ons to Kafka, but we didn't need to use any of those. Confluence, for example, provides plug-ins for Kafka. 

There were other solutions, like Apache MQ, but there were a number of components we looked at that were based around being a message bus, and Kafka was the winner from that review work.

The documentation can be a challenge. There are quite advanced capabilities of Kafka, like the transformations that you can build to modify the data as needed. We found that the biggest challenge was documentation and being able to gain the knowledge of exactly how to do stuff. We also struggled on the transformation, but other components were fine, so some parts are good, and some parts are bad.

I would rate this solution as an eight out of ten. 

We use it extensively in our data pushing, for analytics and all of this type of data that is pushed, rather than on a real-time and payment basis. However, we are using it for offline messages, pushing it for processing, and for heavy, heavy usage, rather than extensively using it for financial data.

The main advantage is increased reliability, particularly with regard to data and the speed with which messages are published to the other side. 

The connectivity from the application is straightforward, as is the API integration.

These are some of the most valuable features of this solution. 

In terms of partition tolerance, message reliability is also present, which is a very good feature from the customer's perspective.

The area for improvement in Kafka is difficult to say because it's a solid product that works well in its intended applications. And, we are looking for something that can be used as part of financial implementations, because we don't want too many messages to be delivered to the other side, which is one of the areas I am looking at as well.

One of the things I am mostly looking for is that once the message is picked up from Kafka, it should not be visible or able to be consumed by other applications, or something along those lines. That feature is not present, but it is not a limitation or anything of the sort; rather, it is a desirable feature.

The next release should include a feature that prevents messages from being consumed by other applications once they are picked up by Kafka.

Then there is message dependability because a message is of no use if cannot be consumed. Alternatively, if the message is consumed but not committed, it should not be recorded in the Kafka queues. It should be because that is one of the features that is existing in MQs consistently provide: if the message is not committed, it will be committed back to the queues.

I have not seen that in Kafka.

We have been using Apache Kafka for approximately three years in the organization.

I believe we are working with version 10. Confluent Kafka is what we are using.

It's a stable solution. Once completed, it is a very stable solution.

The scalability is very good. It is scalable horizontally rather than vertically. 

It can scale up to any level horizontally. However, if the message, once used horizontally scalable, cannot be shrunk once the requirement is reduced, some process is actually taking place. That is one thing that is lacking.

I believe there are approximately 10 to 15 people who use it.

This is being used by the data migration, data team, data analytical team, and data engineer. It's being used by all application architects who are just looking into it, as well as middleware integrators and middleware application integrators.

We have big plans to increase the use of various other innovations and stuff like that. We are using it in relation to data activities. 

Also, we are only planning to use the financial part for publishing it, subscribing, and publishing a pop-up model for various use cases.

Apache usually has a community deployment. If you use Apache or any other software, you will usually receive community support. Otherwise, some companies are taking it and beginning to process it. For example, in Kafka, there is a version of Confluent that they use and support. Or, as we call it, the Oracle Big Data platform.

It will be included with Hadoop, Spark, and other similar technologies. That is coming as, one of the back software packages that are part of that offering, and it is supported by Oracle. Depending on the type of open source, there are various types of support available. Other than the community, we will not receive assistance. Otherwise, it's free enterprise, and we can take it from Confluent or other vendors who offer similar products.

Prior to implementing this solution, we were not using another solution. We have been using, Kafka from the beginning with regard to these use cases. However, we are using other queuing solutions, such as MQ, ActiveMQ, IBM IQ, and Q, but the use cases are different. This is primarily due to the large volume, faster processing, and other benefits of using Kafka.

It is not deployed on-premises. 

We use Kafka as part of the OCI Oracle Cloud platform and the Oracle Big Data platform because Kafka is included.

The Apache Kafka setup will take some time because it is not simple, and we have a lot of other components to install. It's fine because we needed all the plugins and other things for the simple implementations, but the containers' implementation is simple. The only difference is that when it comes to Zookeeper, there are a lot of supporting applications running on top of it, such as Zookeeper. As part of their area, Apache Kafka is running on top of Zookeeper. What do they think? As part of their... manageability, the Kafka area, and Apache Zookeeper. As a result, everything must be removed. And it will be preferable if the implementation is simple.  I believe Confluent is doing this, but we have not yet begun.

The deployment, and configuration, will take one hour to complete. However, it is also dependent on the fact that you require a large number of configurations, which we have.

The deployment was completed in-house.

Currently, there is a team of three to maintain this solution. There are application support personnel in charge of access control.

It will be included in the Oracle-specific platform. It is approximately $600,000 USD.

When it comes to Apache Kafka, they must understand how it works and what its internals are. There could be numerous challenges associated with the product and its entire life cycle. You will have to have a good understanding and knowledge of the configuration. You will need a technical person who is knowledgeable in Kafka which will be an advantage and on an ongoing life partner.

It's a very good solution, I would rate Apache Kafka a nine out of ten.

We have a scalable architecture where we need multiple workers to handle some processing. To make it possible, the backend catches the request and puts it in a common medium, which is the queue of Apache Kafka. The workers then can share and process it.

The most valuable feature of Apache Kafka is the clustering which is very easy to scale and we have multiple servers all over our platforms. It has been useful for stability and performance.

Apache Kafka can improve by providing a UI for monitoring. There are third-party tools that can do it, but it would be nice if it was already embedded within Apache Kafka.

I have been using Apache Kafka for approximately two years.

Apache Kafka is stable. We have not had any issues.

the scalability of Apache Kafka is good. We have parts of the information we use in different geographical sites and it doesn't pose any problem.

I have not used technical support.

I previously used RabbitMQ. We switched because Apache Kafka was more stable and had better performance.

The initial setup of Apache Kafka was easy because it is Dockerized. However, if you were to install it yourself it would be difficult. Having it Dockerized makes it worth it. 

The first deployment took approximately two hours. The updates of the solution can be done in a matter of minutes.

Our DevOps team in our IT department did the deployment of the solution. It was mostly virtual work. The maintenance of the solution does not take a lot of time.

We are using the free version of Apache Kafka.

We had a good experience with the solutions, the maintainability and scalability are good. I would recommend the solution to others.

I rate Apache Kafka a nine out of ten.

We use Apache Kafka for patching real-time data that we receive over a data transport layer and for putting the data into Apache Kafka. From Apache Kafka, we use several applications to subscribe to topics from different applications that we serve directly to browsers. Additionally, we use these applications inside our solution and have Apache Kafka Stream, which is connected to MongoDB.

Since we receive data in real-time consisting of IoT devices, running vehicles, their locations, their states, and their VNs, the solution is helpful.

The product could be improved with proper documentation. Proper documentation should be the SSE. We have a challenge with configuration, so it isn't easy to configure a standalone Apache Kafka on the premises. It needs to be set up on-premises and surveys being provided in the market want to be excluded. Hence, being a developer and configuring Apache Kafka is very hard. It is user-friendly, but initially, we found it challenging. Improving the documentation in this solution would be much better if documents were provided on GitHub for different things. As the market is growing, Spring solution is working hard to get products in the market so when Python, React JS, and Node.Js came, they were lacking. But today, Spring Boot has a solid framework. So the support on Apache Kafka could be improved, but finding some configurations with Spring Boot isn't easy.

We have been using this solution for over three years and are currently using the latest version.

The solution is stable, and the most fantastic thing about it is its throughput. For example, I have tried MQs, which also have Apache Kafka Streams. So the throughput of Apache Kafka Stream is impeccable and impressive.

The solution is very scalable, and its availability is brilliant. We have approximately 32,000 people on our customer base.

We do not have any experience with customer service and support.

We have tried different MQs, but the subscription and charting available on this solution are better. We have used Queues previously, but this solution is more stable, so we chose it.

The initial setup is dependent on the individual. For example, it would be straightforward if a person practices these things a lot and understands the documentation correctly. However, since most people prefer examples instead of reviewing documentation, it would be easy to set up if they find steps on the internet but difficult if they do not have examples.

I rate the pricing for this solution an eight out of ten. It could be a bit cheaper.

I rate this solution an eight out of ten. It is good, but the documentation could be improved.

We utilize Apache Kafka in several areas, including financials, logistics, and client management to name a few.

We used to lose some of our messages when we integrated them in bulk, this solution has stopped that happening.

It is a useful way to maintain messages and to manage offset from our consumers. 

I would like to see an improvement in authentication management.

We have been using the solution for around four years.

The stability is good; the solution operates on our clusters without a big impact.

It is easy to scale.

We used to use a different solution, but our increased throughput meant we needed a product that would allow for a larger queue.

The initial setup was complex for us because we built it internally. This meant that full deployment took around a month.

The implementation was carried out in-house.

I would recommend that other businesses do the deployment themselves, but manage the tool with the aid of a service provider, rather than in-house.

I would rate this product seven out of ten.