Apache Flink Reviews

I am working with Apache Flink, which is the tool we use for data integration. Apache Flink is for data, and we are working on the data integration project, not big data, using Apache Flink and Apache Spark on both sides.

It's for data integration; in terms of big data, we are taking the big data from Kafka and Elasticsearch, and we are moving the data pipeline.

We are not using Apache Flink stateful computations. We are not doing event processing; we are basically in the tool and not in the application. We are writing different database connectors to move the data. The application team will be writing the events and application events.

I have not used Apache Flink DataStream API. We are just using the Apache Flink pipeline in Apache SeaTunnel to move the streaming data.

Streaming functionality is the most useful function in Apache Flink for me. Data streaming is what we prioritize.

Apache Flink provides faster and low-cost investment for me; I find it to have low hardware requirements, and it's faster with low code, meaning it's easy to understand for moving the streaming data. Those two use cases are there.

Apache could improve Apache Flink by providing more functionality, as they need to fully support data integration. The connectors are still very few for Apache Flink.

There is a lack of functionality concerning data connectors.

The technical support from Apache is not good; support needs to be improved. I would rate them from one to ten as not good.

The points for improvement in technical support are in quality and also responsibility; everything they have to work on it.

I have been working with Apache Flink for one to one and a half years or more.

Its ability to scale and expand is also rated an eight or nine. It's quite scalable for the big data solution.

The technical support from Apache is not good; support needs to be improved. I would rate them from one to ten as not good.

The points for improvement in technical support are in quality and also responsibility; everything they have to work on it.

I have compared Apache Flink with other vendors; AWS is a bigger competitor, such as AWS Glue.

I think Apache Flink is better than AWS Glue because of the integration with Apache SeaTunnel.

The initial setup is very easy.

I have compared Apache Flink with other vendors; AWS is a bigger competitor, such as AWS Glue.

I think Apache Flink is better than AWS Glue because of the integration with Apache SeaTunnel.

We are not using Apache Flink in its advanced window capabilities. We are using the Apache Flink job in Apache SeaTunnel, meaning we can write the code inside Apache SeaTunnel.

Currently, we are moving; both solutions are there. We are doing it on-premises with the help of Kubernetes and OpenShift.

The main reason why Apache Flink is better is that it has more functions, and being open source with easy code in Apache SeaTunnel helps us achieve that. Cost is a major issue.

I would rate the stability of the product as an eight.

For Apache Flink, the final point can be rated an eight.

I can recommend Apache Flink to other users for streaming support, and I am recommending it. I would rate this review an eight overall.

Hybrid Cloud

Other

We use Apache Flink for enterprise orchestration, mostly for Walmart, which has many use cases for retail. We use it for our transportation, for the merchant use case related to suppliers, and for customer orchestrations.

What I appreciate best about Apache Flink is that it's open source and geared towards a distributed stream processing framework. We use it on the transportation side for real-time or near real-time data with low latency and high output, ensuring eventual consistency. Most of our event-driven applications utilize Apache Flink, alongside Spark and Kafka streaming. From the core components, we also use data streaming APIs, tables or SQL APIs, and state management, along with Flink connectors for integrations with Kafka, JDBC connections, file systems, and more.

We have numerous use cases for complex event processing, particularly concerning suppliers needing real-time data on store performance for predictive pattern detection, which involves complex space management and model creation for dispatching. We are also exploring Apache Flink ML focused on machine learning libraries, though this is currently in the early stages.

Apache Flink is very powerful, but it can be challenging for beginners because it requires prior experience with similar tools and technologies, such as Kafka and batch processing. It's essential to have a clear foundation; hence, it can be tough for beginners. However, once they grasp the concepts and have examples or references, it becomes easier. Intermediate users who are integrating with Kafka or other sources may find it smoother. After setting up and understanding the concepts, it becomes quite stable and scalable, allowing for customization of jobs.

Every software, including Apache Flink, has room for improvement as it evolves. One key area for enhancement is user-friendliness and the developer experience; improving documentation and API specifications is essential, as they can currently be verbose and complex. Debugging and local testing pose challenges for newcomers, particularly when learning about concepts such as time semantics and state handling. Although the APIs exist, they aren't intuitive enough. We also need to simplify operational procedures, such as developing tools and tuning Flink clusters, as these processes can be quite complex. Additionally, implementing one-click rollback for failures and improving state management during dynamic scaling while retaining the last states is vital, as the current large states pose scaling challenges.

I have been using Apache Flink for the last five years, and we have multiple enterprise products that we use with it.

In terms of stability, every system exhibits some challenges, particularly when customized through lift and shift. In general, Apache Flink is very stable, but it can face failures during customization. Stream processing at scale shows stability, yet I sometimes encounter timeouts with long-running jobs, indicating a need for optimizations in failure recovery. Cluster upgrades and compatibility with saved points need improvement. Despite these challenges, Apache Flink remains quite stable; every product evolves and introduces new features, which can lead to optimization needs.

I rate the scalability of Apache Flink an eight because it is quite scalable, but we need improvements in API documentation and state management.

Apache Flink does not offer enterprise software support, so companies rely on community support, such as mailing lists and forums, along with Slack channels. There is also the option to create issues on GitHub or check Stack Overflow for help. While some may have customizations or use cloud services offering specific support, it's essential to understand that community support is strong for Apache Flink, making it a viable option for many users despite lacking dedicated enterprise support.

Positive

The integration process for Apache Flink can be approached in two ways. The first method involves running it as an independent standalone service, where you create your own clusters and systems, and then submit jobs that connect to it. This approach is more reliable for stream analytics, ETL, or continuous data processing, as it handles scaling and fault tolerance. You can connect with REST APIs, SQL gateways, or JDBC, which is a standard and easier approach. The second option is embedding Apache Flink directly into your application, which is rarely used and mainly suited for simpler tasks. Nonetheless, I recommend running Apache Flink as a standalone service for distributed real-time processing.

I recommend Apache Flink to others, particularly for new teams assessing whether it is the right solution. Despite competition from Kafka streams and Apache Beam, Apache Flink stands out for its scalability, stability, and persistence capabilities. Personally, I have promoted its integration within Walmart, advocating for its use across eight to nine teams, as I am a big fan of data streaming and processing tools. Given its proven track record and stability in production, I confidently recommend it to anyone exploring unified batch processing or streaming databases with SQL or the table API.

On a scale of one to ten, I rate Apache Flink an eight overall.

On-premises

Other

I worked on a project where I provided architectural patterns for a client in the insurance industry, similar to companies like State Farm or Travelers Insurance. They needed a solution for streaming analytics with near real-time data processing, combining streaming data with existing data in S3.

Apache Flink makes it easy to write on-the-fly transformations on streaming data, which is especially beneficial for users familiar with SQL. It allows me to join different data streams and perform aggregations, offering a smooth user experience. The tool is highly effective for use cases where real-time data processing is required. The ease of usage, even for complex tasks, stands out. It is a solution that allows quick validation of architectural patterns within a few days. It is scalable and stable, enhancing its attractiveness for enterprise use.

Apache should provide more examples and sample code related to streaming to help me better adapt and utilize the tool. There is a need for increased awareness and education, especially around best practices, to help people avoid unnecessary costs.

I have been using Apache Flink as part of a project-focused exploration.

Apache Flink is stable for the tasks I have explored. It performed well within the limits of my use case, although I have not implemented it in a full-scale, end-to-end data engineering project.

Apache Flink is scalable. Solutions can always be scaled beyond initial boundaries.

This was my first experience with streaming analytics tools using AWS.

Setting up Apache Flink is not complex. Services like Kinesis and Kinesis Firehose are easy to set up, which subsequently makes integrating Flink straightforward.

I did not use or evaluate other streaming analytics tools before using Apache Flink.

I would rate this solution an eight out of ten. The ease of usage and technology capability are impressive. Increased awareness and more comprehensive documentation would benefit users. Using Apache Flink can help with more efficient streaming analytics processes.

Amazon Web Services (AWS)

We use it for batch processing, specifically for hiding certain data, including location information and other entity-specific attributes.

The significant advantage is the learning curve as it is easy. It's not associated with any specific cloud entity. It provides us the flexibility to deploy it on any cluster without being constrained by cloud-based limitations. It enables freedom for making optimizations, allowing for specific customizations as needed.

There is room for improvement in the initial setup process. I found myself spending a significant amount of time navigating through documentation to configure it. It would be beneficial to have streamlined commands, where the environment could be quickly initialized, including database setup, providing a more efficient and convenient starting point for projects. This would be particularly advantageous for development and experimentation, allowing more focus on feature testing rather than spending time on the setup process.

I have been working with it for one year.

In the tech industry, there's always room for enhancements. It's a dynamic environment where products evolve over time. I would rate it eight out of ten.

I would rate its scalability capabilities eight out of ten. Currently, a team of fifteen individuals is involved in everyday processes.

Regarding tech support, the challenge lies in the open-source realm where there's no inherent accountability. Modules are developed, and users often rely on forums for assistance. However, for more critical or specific problems, especially those related to compliance or complaints, a paid service is available with a dedicated support team. I would rate it seven out of ten.

Neutral

We initially utilized Databricks for batch processing, which also supports real-time capabilities. However, due to limited computing power usage, we transitioned to a more focused approach. We now employ smaller containers for handling about five to six attributes specific to each entity in real time which allows us to experiment with alerts, actions, and responses for various purposes.

The initial setup was complex.

Deployment takes a total of two days: one day for learning and another day for the actual deployment process. We have a small dedicated team that manages updates and handles the networking aspects of the development website. I would rate it eight out of ten.

It's an open source.

I would give it a rating of seven out of ten due to the need for improvement in command automation and perhaps some stability concerns. However, overall, I would still recommend it.

Private Cloud

Amazon Web Services (AWS)

We use the solution to handle million of events per second in telecom. We use the mobile AT&T. It is very simple.

We have a low latency. We can't process more data pipelines in real time. We could aggregate and make operations with less resource. It is very flexible for an optimization for reduced cost.

Apache Flink offers a range of powerful configurations and experiences for development teams. Its strength lies in its development experience and capabilities.

There are more libraries that are missing and also maybe more capabilities for machine learning. It could have a friendly user interface for pipeline configuration, deployment, and monitoring.

I have been using Apache Flink for five years.

I rate the solution’s stability an eight out of ten.

The solution is open-source.

The initial setup is straightforward and fast. It take couple of hours to complete. My team has recently made some optimizations and changed the way we deploy Apache Flink solutions.

I rate the initial setup an eight out of ten, where one is difficult, and ten is easy.

The solution is expensive. I rate the product’s pricing a nine out of ten, where one is cheap and ten is expensive.

The solution is is difficult to manage and handle. 

Apache Flink is one of the main solutions for real-time decision-making and rapid site provision on Azure. Currently, it's predominantly utilized for data engineering projects rather than AI initiatives, although it can indirectly influence analytics, machine learning, and other products in the stack.

The solution is resilient and store more capabilities and features.

Overall, I rate the solution an eight out of ten.

Apache Flink is not a solution but a framework. Spark is a framework, not a tool.

So, when dealing with real-time data processing and ETL use cases that require on-the-fly transformations, Apache Flink seems like a suitable choice as a framework.

Apache Flink allows you to reduce latency and process data in real-time, making it ideal for such scenarios. 

I've worked on three large-scale platform use cases involving Apache Flink. One of those use cases handled a volume of approximately two hundred to 300,000,000  records per day. It translates to approximately 900 to 1000 records per second.

If you have data that is streaming from different kinds of sources, Flink has many advantages. 

One is that Flink basically gives you stateful transformations. So, if you want to transform your streaming data that is coming in, and you want to apply stateful transformations, one transformation after another, depending on the result of the first transformation, Flink allows you to do that.

There are many other positive points. The reason why the Apache organization actually came up with Flink in the first place is that it offers stateful transformations and complete offset management between transformations.

One of the ways to interact with Flink is through a tool called PipeLINK for writing Flink code, and it doesn't require you to use Python directly.

While it does offer a Python-like syntax called PyFlink. PyFlink is a subset of Python that is specifically designed for writing Flink code. It provides a simpler and more accessible way to write Flink code compared to using the Java or Scala APIs.

PyFlink is not as fully featured as Python itself, so there are some limitations to what you can do with it. So, this is an area for improvement. 

However, it is a good choice for users who are not familiar with Java or Scala.

I've been working with Apache Flink for about a year and a half.

My previous organization, where I was employed as a CTO, had about 120 people on my team, and all of them were engineers.

It's a very engineering-specific tool. Many big organizations like Alibaba and Amazon use it. Actually, Amazon has Apache Flink as one of its offerings in the AWS ecosystem.

Flink is a framework, not a tool. You can deploy it on-premise servers, your own machines, or in Kubernetes on the cloud. It doesn't matter.

My company has a technical team that does deployment, maintenance, and everything for Apache Flink. 

Flink is free, it's open source. Flink is open source.

Depending on the use case, we use the appropriate framework. For certain use cases, it's an excellent choice. However, other use cases might require a different framework, such as Lambda or Spark Streaming. So, the choice of framework depends on the specific requirements of the task.

When it comes to real-time ETL and real-time transformation, I would rate Flink very highly, an eight out of ten.

Public Cloud

Amazon Web Services (AWS)

We used this solution for real-time analytics, and for identifying the outliers. We received live industrial data through the Kinesis data streams and Flink processed that data. We had two uses; through Lambda, Flink was putting into the S3 bucket and it was also able to put data into another S3 bucket, which was useful for our business analytics.

Amazon AWS Kinesis has two flavors; SQL and Flink. With SQL, you can file your statement, apply the schema and get what you need. With Flink, you can run it as a job. If your analytics are on the runtime streams, Flink is beneficial.

Apache Flink was easy to deploy and manage.

The issue we had with Flink was that when you had to refer the schema into the input data stream, it had to be done directly into code. The XLS format where the schema is stored, had to be stored in Python. If the schema changes, you have to redeploy Flink because the basic tasks and jobs are already running. That's one disadvantage. Another was a restriction with Amazon's CloudFormation templates which don't allow for direct deployment in the private subnet. You have to deploy into the public subnet and then from the Amazon console, specify a different private subnet that requires a lot of settings. In general, the integration with Amazon products was not good and was very time-consuming. I'd like to think that has changed.

I've used this solution for six months. 

Stability was not an issue. 

We did not use Apache technical support. We were able to manage with whatever documentation was available from Apache for Flink.

I wasn't using anything else for analytics purposes specifically, but I was working with Kafka, which basically works on the same concept as Amazon's Kinesis data streams with brokers. The limitation of Kafka is that it's less flexible than Kinesis. But I have not applied analytics over this on that front. So only data input source, because I am from an IoT background. So integration with the input sources was my core responsibility. I've used Kafka for a postal service company that provided delivery services and required different conditions such as temperature control.

I implemented this solution with two colleagues and the remainder of the team dealt with the CI/CD pipeline, automated scripts and configuring the different deployment environments like development, pre-product and production. For development and integration with the actual input source we used two engineers. If you're experienced, deployment can be completed within two to three weeks. The solution is used extensively. The company we deployed in had multiple factories in different locations. 

I'm unaware of ROI because once we deploy to the customer, our job is done and we don't have any follow-up with them. 

The company looked at Talend which is the complete analytics platform. But they have certain limitations and the customer was not using it to its full capacity. We wanted to replace it because of its limitations. 

I find this solution very handy. Prior to using Flink I had experience on audio and video data streaming. so I don't know how useful Flink is when you want to do real-time analytics for audio and video data. I think if real-time analytics could be supported by Flink, that would be good.

I rate this solution seven out of 10 based on the fact that I haven't used all of Flink's features.

Public Cloud

Amazon Web Services (AWS)

We predominantly use this solution on-premises but intend to migrate to some management services. Our primary use case for this solution is maintaining some pipelines which process data. We are currently integrating some pipelines from Pig to Spark and Pig to Flink.

We value this solution's intricate system because it comes with a state inside the mechanism and product. The system allows us to process batch data, stream to real-time and build pipelines. Additionally, we do not need to process data from the beginning when we pause, and we can continue from the same point where we stopped. It helps us save time as 95% of our pipelines will now be on Amazon, and we'll save money by saving time.

The solution could be more user-friendly. The debugging system could be more suitable in the new release.

We have been using this solution for approximately half a year and we are currently using version 1.15.1.

The solution is currently stable.

We don't have users operating the system because it is an online system which consumes the data we process. For example, our most extensive pipeline has processed 300 billion events daily, and the configuration works on Spark. The configuration keeps about 25 terabytes of RAM.

Regarding the deployment for users, our team operates internationally with employees in Israel, Dublin and the United States, so about 10% of our employees work with this solution, including DevOps. We intend to increase the usage of this solution, but we are currently completing integrations within our organization's pipeline centres.

We have an Amazon support team working with us, so we reach out to our technical support personnel at Amazon when we have issues.

The initial setup was not complex and we installed it on our own. However, we have engineers available to assist us if we need some configuration or installation. It took a few days to install and configure the solution.

I cannot comment on licensing costs because we only assessed prices at the beginning of this process. 

We previously used Apache Spark before using Apache Flink.

I rate this solution a nine out of ten. I would recommend Apache Flink to new users. In my opinion, it is possible to move from Spark to Apache Flink. Apache Flink's functionality overlaps Spark's functionality. The solution is good, but the debugging process could be improved.

We utilize IoT devices to gather data for our clients. This data is analyzed to produce reports and insights, and we leverage machine learning and artificial intelligence models.

The product helps us to create both simple and complex data processing tasks. Over time, it has facilitated integration and navigation across multiple data sources tailored to each client's needs. We use Apache Flink to control our clients' installations. 

Apache Flink should improve its data capability and data migration. 

I have been using the product for five years. 

The solution is stable. 

Apache Flink is scalable. 

The product's deployment can be completed in minutes, and we have a special team. It is straightforward. We initiate our requirements within our secure software and utilize Jenkins and our pipelines to carry out the deployment process, whether for expanding services on the cloud or on-premise servers.

Apache Flink can be deployed in-house. 

I rate the product an eight out of ten. 

On-premises

Apache Flink's best feature is its data streaming tool.

Apache Flink's documentation should be available in more languages.

I've been using Apache Flink for four to five years.

Apache Flink is quite stable.

Apache Flink is scalable.

The initial setup was easy, and the cloud-based version takes no more than a minute to deploy.

I would recommend Apache Flink to other users and rate it seven out of ten.

Hybrid Cloud