Coming October 25: PeerSpot Awards will be announced! Learn more

Apache Kafka OverviewUNIXBusinessApplication

Apache Kafka is #2 ranked solution in top Message Queue Software. PeerSpot users give Apache Kafka an average rating of 8.0 out of 10. Apache Kafka is most commonly compared to IBM MQ: Apache Kafka vs IBM MQ. Apache Kafka is popular among the large enterprise segment, accounting for 77% of users researching this solution on PeerSpot. The top industry researching this solution are professionals from a financial services firm, accounting for 25% of all views.
Apache Kafka Buyer's Guide

Download the Apache Kafka Buyer's Guide including reviews and more. Updated: September 2022

What is Apache Kafka?

Apache Kafka is a distributed streaming platform, with the following capabilities:

  • It lets you publish and subscribe to streams of records. In this respect it is similar to a message queue or enterprise messaging system.
  • It lets you store streams of records in a fault-tolerant way.
  • It lets you process streams of records as they occur.

Apache Kafka gets used for two broad classes of application:

  • Building real-time streaming data pipelines that reliably get data between systems or applications.
  • Building real-time streaming applications that transform or react to the streams of data.
Apache Kafka Customers
Uber, Netflix, Activision, Spotify, Slack, Pinterest
Apache Kafka Video

Apache Kafka Pricing Advice

What users are saying about Apache Kafka pricing:
  • "It is approximately $600,000 USD."
  • "Apache Kafka is free."
  • "The price for the enterprise version is quite high. For on-premise, there is an annual fee, which starts at 60,000 euros, but it is usually higher than 100,000 euros. The cost for a project including the subscription is usually between 100,000 to 200,000 euros. The cost also depends on the level of support. There are two different levels of support."
  • "It's a bit cheaper compared to other Q applications."
  • "We use the free version."
  • Apache Kafka Reviews

    Filter by:
    Filter Reviews
    Industry
    Loading...
    Filter Unavailable
    Company Size
    Loading...
    Filter Unavailable
    Job Level
    Loading...
    Filter Unavailable
    Rating
    Loading...
    Filter Unavailable
    Considered
    Loading...
    Filter Unavailable
    Order by:
    Loading...
    • Date
    • Highest Rating
    • Lowest Rating
    • Review Length
    Search:
    Showingreviews based on the current filters. Reset all filters
    George Thomas - PeerSpot reviewer
    Lead Architect at a financial services firm with 1,001-5,000 employees
    Real User
    Good partition tolerance, message reliability, and API integration
    Pros and Cons
    • "The main advantage is increased reliability, particularly with regard to data and the speed with which messages are published to the other side."
    • "One of the things I am mostly looking for is that once the message is picked up from Kafka, it should not be visible or able to be consumed by other applications, or something along those lines. That feature is not present, but it is not a limitation or anything of the sort; rather, it is a desirable feature. The next release should include a feature that prevents messages from being consumed by other applications once they are picked up by Kafka."

    What is our primary use case?

    We use it extensively in our data pushing, for analytics and all of this type of data that is pushed, rather than on a real-time and payment basis. However, we are using it for offline messages, pushing it for processing, and for heavy, heavy usage, rather than extensively using it for financial data.

    What is most valuable?

    The main advantage is increased reliability, particularly with regard to data and the speed with which messages are published to the other side. 

    The connectivity from the application is straightforward, as is the API integration.

    These are some of the most valuable features of this solution. 

    In terms of partition tolerance, message reliability is also present, which is a very good feature from the customer's perspective.

    What needs improvement?

    The area for improvement in Kafka is difficult to say because it's a solid product that works well in its intended applications. And, we are looking for something that can be used as part of financial implementations, because we don't want too many messages to be delivered to the other side, which is one of the areas I am looking at as well.

    One of the things I am mostly looking for is that once the message is picked up from Kafka, it should not be visible or able to be consumed by other applications, or something along those lines. That feature is not present, but it is not a limitation or anything of the sort; rather, it is a desirable feature.

    The next release should include a feature that prevents messages from being consumed by other applications once they are picked up by Kafka.

    Then there is message dependability because a message is of no use if cannot be consumed. Alternatively, if the message is consumed but not committed, it should not be recorded in the Kafka queues. It should be because that is one of the features that is existing in MQs consistently provide: if the message is not committed, it will be committed back to the queues.

    I have not seen that in Kafka.

    For how long have I used the solution?

    We have been using Apache Kafka for approximately three years in the organization.

    I believe we are working with version 10. Confluent Kafka is what we are using.

    Buyer's Guide
    Apache Kafka
    September 2022
    Learn what your peers think about Apache Kafka. Get advice and tips from experienced pros sharing their opinions. Updated: September 2022.
    632,779 professionals have used our research since 2012.

    What do I think about the stability of the solution?

    It's a stable solution. Once completed, it is a very stable solution.

    What do I think about the scalability of the solution?

    The scalability is very good. It is scalable horizontally rather than vertically. 

    It can scale up to any level horizontally. However, if the message, once used horizontally scalable, cannot be shrunk once the requirement is reduced, some process is actually taking place. That is one thing that is lacking.

    I believe there are approximately 10 to 15 people who use it.

    This is being used by the data migration, data team, data analytical team, and data engineer. It's being used by all application architects who are just looking into it, as well as middleware integrators and middleware application integrators.

    We have big plans to increase the use of various other innovations and stuff like that. We are using it in relation to data activities. 

    Also, we are only planning to use the financial part for publishing it, subscribing, and publishing a pop-up model for various use cases.

    How are customer service and support?

    Apache usually has a community deployment. If you use Apache or any other software, you will usually receive community support. Otherwise, some companies are taking it and beginning to process it. For example, in Kafka, there is a version of Confluent that they use and support. Or, as we call it, the Oracle Big Data platform.

    It will be included with Hadoop, Spark, and other similar technologies. That is coming as, one of the back software packages that are part of that offering, and it is supported by Oracle. Depending on the type of open source, there are various types of support available. Other than the community, we will not receive assistance. Otherwise, it's free enterprise, and we can take it from Confluent or other vendors who offer similar products.

    Which solution did I use previously and why did I switch?

    Prior to implementing this solution, we were not using another solution. We have been using, Kafka from the beginning with regard to these use cases. However, we are using other queuing solutions, such as MQ, ActiveMQ, IBM IQ, and Q, but the use cases are different. This is primarily due to the large volume, faster processing, and other benefits of using Kafka.

    How was the initial setup?

    It is not deployed on-premises. 

    We use Kafka as part of the OCI Oracle Cloud platform and the Oracle Big Data platform because Kafka is included.

    The Apache Kafka setup will take some time because it is not simple, and we have a lot of other components to install. It's fine because we needed all the plugins and other things for the simple implementations, but the containers' implementation is simple. The only difference is that when it comes to Zookeeper, there are a lot of supporting applications running on top of it, such as Zookeeper. As part of their area, Apache Kafka is running on top of Zookeeper. What do they think? As part of their... manageability, the Kafka area, and Apache Zookeeper. As a result, everything must be removed. And it will be preferable if the implementation is simple.  I believe Confluent is doing this, but we have not yet begun.

    The deployment, and configuration, will take one hour to complete. However, it is also dependent on the fact that you require a large number of configurations, which we have.

    What about the implementation team?

    The deployment was completed in-house.

    Currently, there is a team of three to maintain this solution. There are application support personnel in charge of access control.

    What's my experience with pricing, setup cost, and licensing?

    It will be included in the Oracle-specific platform. It is approximately $600,000 USD.

    What other advice do I have?

    When it comes to Apache Kafka, they must understand how it works and what its internals are. There could be numerous challenges associated with the product and its entire life cycle. You will have to have a good understanding and knowledge of the configuration. You will need a technical person who is knowledgeable in Kafka which will be an advantage and on an ongoing life partner.

    It's a very good solution, I would rate Apache Kafka a nine out of ten.

    Which deployment model are you using for this solution?

    Private Cloud

    If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?

    Other
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    Flag as inappropriate
    PeerSpot user
    Solution Architect at a manufacturing company with 10,001+ employees
    Real User
    Good performance when a high throughput is required, but they need to implement a portal
    Pros and Cons
    • "The processing power of Apache Kafka is good when you have requirements for high throughput and a large number of consumers."
    • "They need to have a proper portal to do everything because, at this moment, Kafka is lagging in this regard."

    What is our primary use case?

    I am a solution architect and I used Apache Kafka in this role.

    What is most valuable?

    The processing power of Apache Kafka is good when you have requirements for high throughput and a large number of consumers. 

    What needs improvement?

    They need to have a proper portal to do everything because, at this moment, Kafka is lagging in this regard. It could be used to do the preprocessing or the configurations, instead of directly doing it on the queues or the topics. If you look at Solace, for example, they have come up with a portal where you don't need to touch these activities. You don't need to access the platform beyond the portal.

    For how long have I used the solution?

    I have used Apache Kafka for between one and one and a half years.

    What do I think about the stability of the solution?

    Apache Kafka is stable.

    What do I think about the scalability of the solution?

    This is certainly a scalable product. There are currently 30 or more people using it but we expect to scale beyond this. It is going to be an enterprise tool within the company.

    How are customer service and technical support?

    I am not directly interacting with the service people at this moment. It is limited for now because we are still exploring and effecting our architecture and design, and deciding how to align it with our existing strategy. There is not much progress in this regard and it will take more time.

    Which solution did I use previously and why did I switch?

    Prior to working with Apache Kafka, there was no messaging queue system. For many projects, they were using the Azure Event Hub, but it was not serving the purpose. So, we started moving towards Kafka, and that's why we have procured Confluent Kafka.

    Several months ago, I stopped working on Apache Kafka. I am now working on Confluent Kafka. It was not my decision to switch solutions.

    My current organization has chosen Confluent Kafka for various reasons. One is that we have a large number of streaming requirements, and Confluent Kafka has one more layer on top of Apache Kafka to do this transformation and connecting with other multiple lane systems.

    There are out-of-the-box features along with the KSQL features. For example, things like fetching the events are kind of query-based. So, that seems to be a good feature for our requirements. That is why we ultimately procured Confluent Kafka.

    For some time, I have also worked with Solace and it has an advantage. Given that my core strength is integration, I work with integration platforms such as MuleSoft, Azure functions, then TIBCO. Based on our requirements, I found that the event-driven APA implementation with Solace was easier.

    Solace also has a top-notch solution for portal management and you register your producers, consumers, and preprocessing logic. All of these things are pretty easy to do. This is an area where Kafka could use some enhancement.

    How was the initial setup?

    I don't think that the initial setup was a complex process.

    Which other solutions did I evaluate?

    MQ messaging systems are not my core strength but for any integration platform where we have a large number of APIs and events, to integrate with an IoT platform, for example, I found Kafka is better than ActiveMQ.

    I'm not getting into in MQTT or other things but comparatively, when you compare ActiveMQ and Kafka, Kafka has done better.

    What other advice do I have?

    I think that many people are using Apache Kafka just as a publishing and subscription model, but I feel that Kafka is better than that. Furthermore, Confluent Kafka is even more than that.

    Confluent Kafka is offering features that are equal to those of a data lake. You can do lots with data, and huge data can be persisted. However, many people are not using that feature. Rather than make use of persistence logic, they are pushing the messages and consuming them. Maybe if people were using it for persistence, they would see the impact or real power of Kafka.

    I would rate this solution a seven out of ten.

    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    PeerSpot user
    Roger Sabourin - PeerSpot reviewer
    Roger SabourinSenior Manager, Analyst Relations at a tech vendor with 201-500 employees
    Real User

    You're in luck, Solace's PubSub+ Event Portal for Kafka does all the things you're looking for, specifically for your Kafka environments, be they open source Kafka, Confluent or Amazon MSK.  Check it out, or request a free trial at https://solace.com/products/po...

    Buyer's Guide
    Apache Kafka
    September 2022
    Learn what your peers think about Apache Kafka. Get advice and tips from experienced pros sharing their opinions. Updated: September 2022.
    632,779 professionals have used our research since 2012.
    Principal Technology Architect at a computer software company with 5,001-10,000 employees
    Real User
    Events and streaming are persistent, and multiple subscribers can consume the data
    Pros and Cons
    • "With Kafka, events and streaming are persistent, and multiple subscribers can consume the data. This is an advantage of Kafka compared to simple queue-based solutions."
    • "Kafka's interface could also use some work. Some of our products are in C, and we don't have any libraries to use with C. From an interface perspective, we had a library from the readies. And we are streaming some of the products we built to readies. That is one of the requirements. It would be good to have those libraries available in a future release for our C++ clients or public libraries, so we can include them in our product and build on that."

    What is our primary use case?

    It's a combination of an on-premise and cloud deployment. We use AWS, and we have our offshore deployment that's on-premise for OpenShift, Red Hat, and Kafka. Red Hat provides managed services and everything. We use Kafka and a specific deployment where we deploy on our basic VMs and consume Kafka as well.

    We publish or stream all our business events as well as some of the technical events. You stream it out to Kafka, and multiple consumers develop a different set of solutions. It could be reporting, analytics, or even some data persistence. Later, we used it to build a data lake solution. They all would be consuming the data or events we are streaming into Kafka.

    What is most valuable?

    With Kafka, events and streaming are persistent, and multiple subscribers can consume the data. This is an advantage of Kafka compared to simple queue-based solutions.

    What needs improvement?

    We are still on the production aspect, with our service provider or hyper-scalers providing the solutions. I would like to see some improvement on the HA and DR solutions, where everything is happening in real-time. 

    Kafka's interface could also use some work. Some of our products are in C, and we don't have any libraries to use with C. From an interface perspective, we had a library from the readies. And we are streaming some of the products we built to readies. That is one of the requirements. It would be good to have those libraries available in a future release for our C++ clients or public libraries, so we can include them in our product and build on that.

    For how long have I used the solution?

    We've been using Apache Kafka for the past two to three years.

    What do I think about the stability of the solution?

    Kafka is stable. It's a great product. 

    What do I think about the scalability of the solution?

    We did some benchmarking, but we are still looking further to scale up some of the benchmarking and performances. So far, it meets all our business requirements. We are just developers, so everything goes to the clients, who will deploy it at their scale and use it for their end customers. So were are looking at it from a developer's perspective. Those who are developing the products are working on this.

    How are customer service and support?

    We haven't really contacted technical support, but some of our clients have subscribed to support from the vendors. We generally look for open-source solutions. From there, we try to figure out if there are any issues. There's a good online community where you can ask questions.

    How was the initial setup?

    We were able to deploy and use it with no problems for our use case. We didn't find it so complex. We work with so many applications, databases, Postgres, and so many other things, so we could manage it easily. We deployed Kafka in a few hours. We have an infrastructure team and DevOps. Those teams are pretty capable, and they've completely automated the whole deployment. It always takes time the first time you upgrade any application, not just Kafka. We might discover some issues, such as configuration, parameters, compatibility, etc. Once that becomes standard, it is stable, and then they only need to replicate it to the different environments or different developers groups. We have a sophisticated process.

    What other advice do I have?

    I rate Apache Kafka eight out of 10. There are so many products on the market, so my advice is to consider if Kafka suits your business requirements first. If it's suitable, the next step is to check whether all the technical requirements are met. If everything checks out, I would say that Kafka is a relatively stable, sound, and scalable product, so they can try it out. 

    Which deployment model are you using for this solution?

    Hybrid Cloud
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    PeerSpot user
    Sr Technical Consultant at a tech services company with 1,001-5,000 employees
    Real User
    Top 5
    Effective stream API, useful consumer groups, and highly scalable
    Pros and Cons
    • "The most valuable features are the stream API, consumer groups, and the way that the scaling takes place."
    • "would like to see real-time event-based consumption of messages rather than the traditional way through a loop. The traditional messaging system works by listing and looping with a small wait to check to see what the messages are. A push system is where you have something that is ready to receive a message and when the message comes in and hits the partition, it goes straight to the consumer versus the consumer having to pull. I believe this consumer approach is something they are working on and may come in an upcoming release. However, that is message consumption versus message listening."

    What is our primary use case?

    One of our clients needed to take events out of SAP to stream them through Apache Kafka while applying data enrichment before reaching the consumers.

    How has it helped my organization?

    The solution can handle more speed and has horizontal scalability for both messaging, but more specifically stream processing and data enrichment. By using this solution it can reduce the number of components required in the tech stack. For example, we were taking data events out of SAP and sending them to consumers without having to go through multiple processors that were outside of the KAFKA space. Additionally, we are using Kafka from GoldenGate to propagate database updates in real-time.

    What is most valuable?

    The most valuable features are the stream API, consumer groups, and the way that the scaling takes place. 

    What needs improvement?

    I would like to see real-time event-based consumption of messages rather than the traditional way through a loop. The traditional messaging system works by listing and looping with a small wait to check to see what the messages are. A push system is where you have something that is ready to receive a message and when the message comes in and hits the partition, it goes straight to the consumer versus the consumer having to pull. I believe this consumer approach is something they are working on and may come in an upcoming release. However, that is message consumption versus message listening.

    Confluent created the KSQL language, but they gave it to the open-source community. I would like to see KSQL be able to be used on raw data versus structured and semi-structured data.

    For how long have I used the solution?

    I have been using this solution for approximately one year.

    What do I think about the stability of the solution?

    The solution is stable.

    What do I think about the scalability of the solution?

    I have found the Apache Kafka to be highly scalable

    How are customer service and technical support?

    The project we were working on was open-source, we were using Confluent as support and they were great.

    How was the initial setup?

    Apache Kafka on AWS is a bit complex. There is a third-party company called Confluent and they have the support that makes their installation much easier, especially for the on-premise deployment. You install Apache Kafka alone it can be a little complex compared to other queuing messaging solutions.

    The on-premise deployment takes approximately a few days. The cloud or hybrid deployments including all the permissions, typologies, firewalls, and networking configuration can take weeks for all the accessibility issues to be resolved. However, the delay could have been client-related and not necessarily the solution.

    What about the implementation team?

    We provide the implementation service.

    What's my experience with pricing, setup cost, and licensing?

    Apache Kafka is free. My clients were using Confluent which provides high-quality support and services, and it was relatively expensive for our client. There was a lot of back and forth on negotiating the price.

    Confluent has an offering that has Cloud-Based pricing. There are different packages, prices, and capabilities. The highest level being the most expensive. AWS provides services to their market, for example, to have Kafka running. I do not know what the pricing is and I am fairly confident, Azure and GCP provide similar services.

    What other advice do I have?

    My advice to others wanting to implement this solution is to start with data streaming projects, not simple messaging projects because while it is very good at general-purpose messaging, it is more suited and geared for when you are using it as a streaming solution.

    I rate Apache Kafka an eight out of ten.

    Which deployment model are you using for this solution?

    Public Cloud

    If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?

    Amazon Web Services (AWS)
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    PeerSpot user
    Mario Estrada - PeerSpot reviewer
    CTO at Estrada & Consultores
    Real User
    Top 20
    Great scalability with a high throughput and a helpful online community
    Pros and Cons
    • "The solution is very easy to set up."
    • "While the solution scales well and easily, you need to understand your future needs and prep for the peaks."

    What is our primary use case?

    We primarily use the solution for upstreaming messages with different payload for our applications ranging from iOT, Food delivery and patient monitoring. 

    For example for one solution we have a real-time location finding, whereby a customer for the food delivery solution wants to know, where his or her order is on a map. The delivery person's mobile phone would start publishing its location to Kafka, and then Kafka processes it, and then publishes it to subscribers, or, in this case, the customer. It allows them to see information in real-time almost instantly.

    How has it helped my organization?

    Apache Kafka has became our main component on almost all our distributed solutions. It has helped us to delivery fast distributing messages to our customer's applications.

    What is most valuable?

    The solution is good for publishing transactions for commercial solutions whereby a duplicate will not affect any part of the system.

    The solution is very easy to set up.

    The stability is very good.

    There's an online community available that can help answer questions or troubleshoot problems. 

    The scalability of Kafka is very good.

    It provides high throughput.

    What needs improvement?

    Kafka can allow for duplicates, which isn't as helpful in some of our scenarios. They need to work on their duplicate management capabilities but for now developers should ensure idempotent operations for such scenarios.

    While the solution scales well and easily, you need to understand your future needs and prep for the peaks. 

    For how long have I used the solution?

    I've been using the solution for four years so far.

    What do I think about the stability of the solution?

    The stability is excellent. There are no bugs or glitches. It doesn't crash or freeze. It's reliable. 

    What do I think about the scalability of the solution?

    Scaling is not really a problem with Kafka. We have used Kubernetes clusters and it is working very well. It scales up and down, almost automatically almost unnoticeable to the consumers, based upon our configuration. Kafka is just one pod inside of our cluster that scales horizontally.

    We have a couple of customers that also have vertical scaling, meaning that, there's more CPU, more memory available to the Kafka pod.

    How are customer service and technical support?

    For Kafka, we don't actually require support from the company. We usually have people experienced in-house and sometimes we just ask in the community. 

    How was the initial setup?

    The initial setup is easy. The majority of the tools today are really very easy to configure and setup. Docker Containers and Kubernetes, actually, have made life easier for architects as well as developers.

    Nowadays, you just install the container, and then you don't have to really manage the internals at libraries, OS levels, et cetera. You just run the container. Everything is containerized.

    What's my experience with pricing, setup cost, and licensing?

    Apache Kafka is OpenSource, you can set it up in your own Kubernetes cluster or subscribe to Kafka providers online as a service.

    What other advice do I have?

    New users should understand the product capabilities. Often, people will start putting their hands in new products without knowing the capabilities and the disadvantages in specific scenarios. In our case for example, We haven't used Kafka for financial transaction processing, for which we still use IBM MQ, but It really depends upon your knowledge and experience with the product. My advice is to understand the product very well, its pros and cons and work from there.

    Finally I'd rate the solution at a nine out of ten.

    Which deployment model are you using for this solution?

    Hybrid Cloud
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    PeerSpot user
    CEO at a comms service provider with 11-50 employees
    Real User
    Reliable for working with a huge amount of data and has many options for building applications on top of it
    Pros and Cons
    • "The high availability is valuable. It is robust, and we can rely on it for a huge amount of data."
    • "The price for the enterprise version is quite high. It would be better to have a lower price."

    What is our primary use case?

    We deploy it for our customers. The main use case is related to log management and metrics because we are a partner of Elastic Stack, and we usually collect information through Kafka.

    What is most valuable?

    The high availability is valuable. It is robust, and we can rely on it for a huge amount of data. 

    The Kafka Streams capability is also valuable. We get many options to build applications on top of Kafka.

    What needs improvement?

    The price for the enterprise version is quite high. It would be better to have a lower price.

    For how long have I used the solution?

    I have been working with this solution for four or five years.

    What do I think about the stability of the solution?

    It is absolutely stable.

    What do I think about the scalability of the solution?

    It is very scalable. It is easy to scale it. 

    It doesn't matter how many users are using it. The licenses are calculated based on the number of nodes. It is not based on the number of users who are using it. We have between 10 to 20 nodes on average in an organization.

    How are customer service and support?

    It is quite good, but they don't speak Italian. In Italy, we have to provide support in the Italian language. It is a problem for customers to have support in English. This is the reason why we provide direct support to customers.

    How was the initial setup?

    I am into pre-sales and project management. I don't usually install Apache Kafka, but its basic installation seems quite simple.

    Its deployment is usually quite short. Usually, we are able to deploy it in a few days, but data management and application development can take a few months.

    What about the implementation team?

    We have our own team to deploy it. We also take care of its maintenance. We have a team of five or six employees to provide 24/7 support to our customers.

    What was our ROI?

    It depends on the project. For log management projects, the ROI is not very quick, but we have other projects where we used Kafka for high-value applications, and the ROI was very quick. We got an ROI in a few months.

    What's my experience with pricing, setup cost, and licensing?

    The price for the enterprise version is quite high.

    For on-premise, there is an annual fee, which starts at 60,000 euros, but it is usually higher than 100,000 euros. The cost for a project including the subscription is usually between 100,000 to 200,000 euros. The cost also depends on the level of support. There are two different levels of support.

    What other advice do I have?

    Kafka is a really good product. To be able to keep it running in the long term, you need to know very well how it works. You should have good knowledge about it. It isn't about just knowing how to install it because it is quite simple to install it. It is important to have the right knowledge and experience to do a good installation and let it run for a long period. You can also go for someone who has the right experience and knowledge.

    We are very satisfied with Kafka. I would rate it an eight out of 10. It is not perfect, but it is a really good product.

    Which deployment model are you using for this solution?

    On-premises
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    Flag as inappropriate
    PeerSpot user
    Salvatore Campana - PeerSpot reviewer
    CEO & Founder at XAUTOMATA TECHNOLOGY GmbH
    Real User
    Top 5Leaderboard
    Allows us to ingest a lot of data and make tech decisions in real time
    Pros and Cons
    • "The stability is very nice. We currently manage 50 million events daily."
    • "The repository isn't working very well. It's not user friendly."

    What is our primary use case?

    We use Apache Kafka to ingest a lot of data in real time that Apache Spark processes, and the result is used for a tech decision in real time – in the IT environment, infrastructure environment, and IOT environment, like for a  manufacturing plant.

    This is an open-source framework. We also sell professional services on this solution and specifically create a business application for customers. 

    The application is called Sherlogic. We have two kinds of customers. We have end-user customers that use the Sherlogic solution, and maybe customers don't know that there is Spark and Kafka in Sherlogic. But we have another kind of customer that uses professional services by Xautomata to create tailor-made applications in analytics and the automation process.

    We use Apache Kafka for our digital cloud.

    What needs improvement?

    To store a large set of analytical data we are using SQL repository. This type of repository works very well but we need specific and high maintenance. The user experience is friendly.

    We are looking for alternative solutions, we tried with noSQL solutions and Confluent specific features but the results were not satisfactory both in terms of performance and usability.

    We are working on automated SQL repository management and maintenance tools in order to increase the democratization of our platform.

    For how long have I used the solution?

    We've been using this solution for a year and a half.

    What do I think about the stability of the solution?

    The stability is very nice. We currently manage 50 million events daily.

    What do I think about the scalability of the solution?

    It's scalable.

    How are customer service and support?

    Support is good. It's typical for an open source application. You can have all the information in a public portal. If you want specific consulting, there is a company that promotes this consulting worldwide called Conduent. Their consulting is quick and they have a lot of know-how.

    How was the initial setup?

    It's very complex, like Spark. 

    Deployment took 50 minutes for all the Kubernetes ports, Spark, Kafka, and other components based on Sherlogic. In 30 minutes, we created an environment using this program to make installation easier.

    What about the implementation team?

    Deployment was done in-house, but we're starting a collaboration with another company and we introduced this company to running this solution. Specifically, we started a collaboration with AWS to promote our platform in a Western marketplace. In this way, it's very easy to use our solution because it is a part of an AWS service, certificated by an engineer.

    What was our ROI?

    The return on investment has been having people dedicated to this solution because it's open source so it hasn't been necessary to invest in licensing or pay a fee. So, internal know-how has been the ROI.

    What's my experience with pricing, setup cost, and licensing?

    It's a bit cheaper compared to other Q applications.

    What other advice do I have?

    I would rate this solution 7 out of 10.

    I would recommend this solution because the queue manager is very fast and stable.

    Which deployment model are you using for this solution?

    Public Cloud
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    Flag as inappropriate
    PeerSpot user
    Vice President at a consultancy with 51-200 employees
    Real User
    Top 20
    Open - Source, integrates well with external systems, and has a built-in failover
    Pros and Cons
    • "It is the performance that is really meaningful."
    • "More Windows support, I believe, is one area where it can improve."

    What is our primary use case?

    We use Apache Kafka for our messaging. 

    We publish a message and ask the subscriber to listen to it. We use it to save events generated by integration with external systems. There are external events, that are first published to our Kafka queue, and then to a topic, and then we save it to our own data storage system.

    What is most valuable?

    I believe that the speed, and especially the performance, are very good features.

    Also included is a cluster with built-in failover. 

    It is the performance that is really meaningful.

    In terms of features, we are satisfied. I don't require any additional features. I don't believe we require any additional features at this time.

    What needs improvement?

    More Windows support, I believe, is one area where it can improve. We need to wrap it as a service, but there isn't one built into Windows. So that's something they could improve.

    I believe Windows Server is primarily aimed at the Windows shop or those who use Windows.

    For how long have I used the solution?

    I don't recall the specific version that we are using, it may be Kafka 2.11, but it is not the latest one.

    What do I think about the stability of the solution?

    It's stable. However, the Windows Service is not very stable because it is a wrapper.

    What do I think about the scalability of the solution?

    We are a small team with a few people.

    We might increase our usage in the future.

    How are customer service and support?

    We don't get in touch with technical support. We rely on open-source software. We haven't used the help of technical support. We did not seek assistance. As a result, I have no opinion on the subject.

    Which solution did I use previously and why did I switch?

    This is the first product we have used. We didn't have anything prior to that.

    How was the initial setup?

    The initial setup is straightforward. It's easy to set up.

    It took a few days to get it up and running.

    We only need one or two engineers to keep this solution running. We basically let it run and monitor what's going on. We usually don't touch it unless something goes wrong.

    What about the implementation team?

    We deployed it ourselves.

    What's my experience with pricing, setup cost, and licensing?

    It's free. We use the free version.

    Which other solutions did I evaluate?

    VMware RabbitMQ and ActiveMQ are products that are not being used by us. I wanted to look into it. But we use different things.

    We compared our findings to those of other researchers. We are primarily concerned with performance. Kafka is unquestionably the performance leader.

    What other advice do I have?

    I would recommend trying this solution, but you should probably run it on Linux.

    I like this product, I would rate Apache Kafka a nine out of ten.

    Which deployment model are you using for this solution?

    On-premises
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    Flag as inappropriate
    PeerSpot user
    Buyer's Guide
    Download our free Apache Kafka Report and get advice and tips from experienced pros sharing their opinions.
    Updated: September 2022
    Product Categories
    Message Queue (MQ) Software
    Buyer's Guide
    Download our free Apache Kafka Report and get advice and tips from experienced pros sharing their opinions.