Apache Kafka on Confluent Cloud Reviews

We use Apache Kafka on Confluent Cloud for streaming large volumes of data in real-time. It's employed in scenarios such as handling events from various countries and streaming them efficiently for our clients. 

We also utilize it for data analytics and in client versions for topic creation, consumer consumption, and ACL provisioning.

Apache Kafka on Confluent Cloud provides an enhanced level of reliability and resources compared to Apache Kafka alone. It offers more features which are beneficial for our clients, including cluster linking, schema registry, error handling, and dead-letter queues. It significantly improves customer and publisher satisfaction, especially with topic integration and data streaming.

In my company, we are not using the tool for analytics and it is more for CDC processes, so we change the capture processes. It is used to extract data from a database and make it available in other parts of our systems or produce events that inform us of data updates.

There are some premium connectors, for example, available in Confluent, which you cannot access in the marketplace, so there are some limitations. From Confluent's point of view, I understand where they come from, but I believe its deployment model is a little strict. Confluent Platform can be installed in your own infrastructure. Even if you install Confluent Platform on your own platform, you need to use the components that Confluent offers. Otherwise, the support is very limited, and I think this is an idea of improvement for Confluent. Confluent is pretty solid, so I don't have much in terms of improvement.

Whenever you need to handle a huge load of real-time data processing, Kafka is useful. We currently use it for an output management system for insurance, where the system receives data in a fixed amount and has to process it in several steps. 

We manage these steps with Kafka because the load can be quite big, with millions of XMLs coming into the system that need to be processed in near real-time.

The most valuable feature to me is it scales really well when there is a data influx, irrespective of the peak time. It can handle both large and small amounts of data. 

Also, the state-saving feature is very much appreciated. It allows me to rewind a certain process if I see an error and then reprocess it.

We find that the best features include using the CDC functionality with the connector to take the data from our SQL database and publish it to many consumers. Any changes enable us to easily publish changes about their domain business objects without too much code and work from domain teams. In this way, we can more easily provide a very robust layer of API and events.

The second use case is easier projection of data. We found that many teams were struggling to create projections and read stores with regular event buses, and Apache Kafka on Confluent Cloud helped us because of all sorts of features, such as the log architecture they have, and other features. KSQL also helped us there.

When order is more important, we rely on Apache Kafka on Confluent Cloud.

The benefits that I have seen from having a real-time architecture include better velocity for developers. That is the main one. Instead of developing many of those capabilities in each team, we can rely on Apache Kafka on Confluent Cloud to provide those functionalities we want, and the teams can focus on their own business instead of providing all sorts of APIs and dependencies to other domains, allowing everyone to run faster.

We find that the best features include using the CDC functionality with the connector to take the data from our SQL database and publish it to many consumers. Any changes enable us to easily publish changes about their domain business objects without too much code and work from domain teams. In this way, we can more easily provide a very robust layer of API and events.

The second use case is easier projection of data. We found that many teams were struggling to create projections and read stores with regular event buses, and Apache Kafka on Confluent Cloud helped us because of all sorts of features, such as the log architecture they have. KSQL also helped us there.

I use the solution in my company mostly for log-related purposes.

The solution's most valuable feature is that we used it during the building of data pipelines. From that, we read, wrote, and managed all the processes if there was any event happening. So, in real-time, we are just tracking how many users or logs are coming into our company.

We use Apache Kafka with Confluent Cloud for specific real-time transaction use cases, both on-premise and in the cloud. We have been using Confluent Cloud for about five years.

We initially used it for data reputation, then expanded to microservices integration and Kubernetes, focusing on improving data quality and enabling real-time location tracking.

We configure it for data transactions across various topics and partitions, depending on the specific use case and required throughput.

From an IT perspective, I've used this product across all domains: system development, operations, data management, and system quality.

We have experience using Kafka on Confluent Cloud for data pipelines. We've implemented several techniques to optimize topic usage, integrated it with microservices, and even utilized change data capture (CDC) components.