Apache Flink Reviews

We use Apache Flink for enterprise orchestration, mostly for Walmart, which has many use cases for retail. We use it for our transportation, for the merchant use case related to suppliers, and for customer orchestrations.

What I appreciate best about Apache Flink is that it's open source and geared towards a distributed stream processing framework. We use it on the transportation side for real-time or near real-time data with low latency and high output, ensuring eventual consistency. Most of our event-driven applications utilize Apache Flink, alongside Spark and Kafka streaming. From the core components, we also use data streaming APIs, tables or SQL APIs, and state management, along with Flink connectors for integrations with Kafka, JDBC connections, file systems, and more.

We have numerous use cases for complex event processing, particularly concerning suppliers needing real-time data on store performance for predictive pattern detection, which involves complex space management and model creation for dispatching. We are also exploring Apache Flink ML focused on machine learning libraries, though this is currently in the early stages.

I worked on a project where I provided architectural patterns for a client in the insurance industry, similar to companies like State Farm or Travelers Insurance. They needed a solution for streaming analytics with near real-time data processing, combining streaming data with existing data in S3.

Apache Flink makes it easy to write on-the-fly transformations on streaming data, which is especially beneficial for users familiar with SQL. It allows me to join different data streams and perform aggregations, offering a smooth user experience. The tool is highly effective for use cases where real-time data processing is required. The ease of usage, even for complex tasks, stands out. It is a solution that allows quick validation of architectural patterns within a few days. It is scalable and stable, enhancing its attractiveness for enterprise use.

Initially, we created our own servers and then eBay created their infrastructure. Now it's deployed on the eBay cloud.

Our primary use case is trying to do real time aggregations/near-real time aggregations. Let's say for example that we are trying to do some count, sum,min,max distinct counts for different metrics that we care about, but we do this in real time. So let's say, you have an e-commerce company and you want to measure different metrics. If I take the example of risk, let's say you want to check if one particular seller on your site is doing something fishy or not. What is the behavior? How many listings do they have? In the past five minutes, one hour or one day or one year? You want to measure this over time.

This data is very important to you from the business metric point of view. Often this data  data is delayed by 1 day via offline analytics. You do ETL for these aggregations ,it's okay for offline business metrics. But when you want to do risk detection for online businesses, it needs to be right away in real time, and that's where those systems fail and where Apache Flink helps. And if combined with Lambda architecture, you can get them real time with the help of a parallel system that captures very latest data.

The mighty work on risk, right? So we have to provide risk insights. When we have machine learning models or deep learning models or route based systems and when we want to evaluate some user or somebody's behavior on the site, we want to evaluate it right away. If we don't evaluate it right away, it's of no use to us. Let's say that a fraudulent buyer comes in and is trying to buy an item. In the process of buying, when the person clicks on the button and purchases the item, if you're not able to detect the fraud right then, it's of no use to us.

At the same time, we also have to be able to make sure that we are not pissing off the real people if they are good. So we have to create the very hard balance between whether it's a good person or a bad person without seeing what they're doing. Then you need to do it in real time. That's why with offline systems the aggregation of your data comes after one day, which is a typical case of ETL and is of no use to us. We want to do it right away. But at the same time, we don't want to provide too much friction either. This aggregation includes: how many transactions have you done in the past, in the last year? How many transactions did you do in the last minute? How many coupons did you use in last five years or how many coupons did you use in five minutes? All these kinds of metrics can be useful and can feed into a machine learning or deep learning model or a rule based system to do something with it.

Let's say if we feel that the seller has crossed a limit, or our seller is doing something fishy or buying something fishy, we can throw a capture to the user or we can provide some friction -"can you do any multifactor authentication?" These are just examples. So our use case is the idea that we want to have real time aggregations. One thing to note is that Flink will not help you to do real time navigation very well. It helps you in near-real time navigation. But with the lack that you get from near-real time vs. real time is that you can have it from another system. So let's say that Flink is able to give you all the aggregations accurately either daily or in five minutes. I'm just taking an example for our use here. That's how it works. Five minutes. And it depends on the complexity of the problem that you are trying to solve or how much infrastructure you have. So if it is delayed by five minutes, what you can do is have a panel system that only takes care of the latest data. And then you can take this to the system and combine it with the Lambda architecture. Where you combine a historical system with the real time system and then give the data aggregation. What we do is we apply the same concept in a near-real time integrated fashion and then we are able to give real time analytics on anything that happens on us.

We predominantly use this solution on-premises but intend to migrate to some management services. Our primary use case for this solution is maintaining some pipelines which process data. We are currently integrating some pipelines from Pig to Spark and Pig to Flink.

We value this solution's intricate system because it comes with a state inside the mechanism and product. The system allows us to process batch data, stream to real-time and build pipelines. Additionally, we do not need to process data from the beginning when we pause, and we can continue from the same point where we stopped. It helps us save time as 95% of our pipelines will now be on Amazon, and we'll save money by saving time.

We used this solution for real-time analytics, and for identifying the outliers. We received live industrial data through the Kinesis data streams and Flink processed that data. We had two uses; through Lambda, Flink was putting into the S3 bucket and it was also able to put data into another S3 bucket, which was useful for our business analytics.

Amazon AWS Kinesis has two flavors; SQL and Flink. With SQL, you can file your statement, apply the schema and get what you need. With Flink, you can run it as a job. If your analytics are on the runtime streams, Flink is beneficial.

We use the solution to handle million of events per second in telecom. We use the mobile AT&T. It is very simple.

We have a low latency. We can't process more data pipelines in real time. We could aggregate and make operations with less resource. It is very flexible for an optimization for reduced cost.

We use it for batch processing, specifically for hiding certain data, including location information and other entity-specific attributes.

The significant advantage is the learning curve as it is easy. It's not associated with any specific cloud entity. It provides us the flexibility to deploy it on any cluster without being constrained by cloud-based limitations. It enables freedom for making optimizations, allowing for specific customizations as needed.

We use Apache Flink to monitor the network consumption for mobile data in fast, real-time data architectures in Mexico. The projects we get from clients are typically quite large, and there are around 100 users using Apache Flink currently.

For maintenance and deployment, we split our team into two squads, with one squad that takes care of the data architecture and the other squad that handles the data analysis technology. Each squad is three members each.

The top feature of Apache Flink is its low latency for fast, real-time data. Another great feature is the real-time indicators and alerts which make a big difference when it comes to data processing and analysis.