Apache Airflow Reviews

Apache Airflow's best feature is its flexibility.

Apache Airflow could be improved with the addition of more frameworks.

I've been using Apache Airflow for four years.

Apache Airflow is stable.

Apache Airflow is scalable.

The initial setup was very easy.

We used an in-house team.

Apache Airflow is open-source and free of charge.

I would rate Apache Airflow eight out of ten.

Currently, I am a lead data scientist. Our primary use cases for Apache Airflow are for all orchestrations, from the basic big data lake to machine learning predictions. It is used for all the MLS processes. It is also used for some ELT, to transform, load, and export all big data from restricted, unrestricted, and all phase processes.

The user experience of Apache Airflow is good. The solution is flexible for all programming languages for all frameworks. I also value that it is used for monitoring. Apache Airflow helps to easily integrate data sources with other products.

Apache Airflow could be improved by integrating some versioning principles. Currently, we have to swap some tags in our flow. It would be interesting if we can check the product and version all of the product at the same time comparing what scripts have changed from last year to this year, or last month to this month.

For example, we have a flow for one project, to version it we need to check it one by one to identify which tags changed and which scripts changed. All of these need to be done manually.

I have been using Apache Airflow for four months.

We have experienced some bugs in Airflow. For example, the solution did not mention all the errors regarding why a process did not work. We had to investigate to try and understand why it was not working.

The solution is easy to scale. We have four people in our organization that use Airflow. One is dedicated to the solution, while the others can use it to adjust the flow of their jobs on their own.

We do not use technical support. We are trained to resolve concerns on our own. If a problem is significant we could call support, however, there is a good developer community that uses Airflow that can help resolve the issue with us.

Positive

Prior to using Airflow, I used Windows SSIS for three years. We made the switch because Windows SSIS uses the drag-and-drop concept, where Airflow requires coding. Also, Windows is orientated to Microsoft products and is not very flexible.

I am a technician, so the initial setup is instinctive. Without experience, it would not be as simple. Experience with configurations with parameters is required. The documentation is good, however, it does not mention some features explicitly requiring some research. 

I would rate the ease of implementation a three out of five.

We have dedicated machine learning ops, so we manage all product deployment ourselves. The deployment takes about four days, including two days of administration. 

Apache Airflow requires maintenance. It is very important to maintain all the source codes and all the data. We are looking for a platform that would facilitate the maintenance of the project.

We use a community edition of Apache Airflow. It is open-source and free. 

Anyone considering Apache Airflow should make sure that they have a good team with experience, including some administration. A strong background will help to understand and exploit the strengths of the platform.

I would rate this solution a nine out of 10 overall.

We use this solution to monitor BD tasks.

The solution is quite configurable so it is easy to code within a configuration kind of environment. 

The ease of learning and using the solution is quite good. The learning curve is low so new users can learn in a short period of time in comparison to other products. 

The following should be improved:

• Dashboards

• Security

• Airflow web UI

• Telemetry for logging, monitoring, and alerting purposes

• Documentation 
  

I have used the solution for six months. 

The solution is 99% stable. We have a few glitches here and there but have been able to fix them. 

The solution is quite scalable. You can grow in terms of users and environment. You can grow to multi-server applications. You can use the solution on desktops, mobile, or other devices. 

We have an internal tech support team so have not needed support from the vendor. 

The setup is straightforward. The time for deployment depends on the environment and user base.

We implement the solution in-house. We have one implementation with 60 users and another with 75 users. 

We have a tech support team that consists of ten engineers who support implementations. They follow up on issues that might arise during the process automation or implementation of the workflow itself. 

For example, our tech support team will resolve a workflow that gets stuck during the MDM workflow engine. The tech team has the knowledge base to resolve any of these issues. 

The solution is open source.

I do not have exposure to use cases for large organizations with a huge user environment, so I cannot speak to the solution's effectiveness in these scenarios. 

I rate the solution an eight out of ten. 

I use this solution for scheduling purposes. We have our own Python framework to run jobs, do the extractions, and for transformation loading.

We have 20 people who are using Airflow. It's being used on a daily basis. We don't have any plans to increase usage because we have low data sets.

The solution is deployed on cloud. The cloud provider is Azure.

Everything is in the Python framework now. I would like to see some no-code capabilities and drag and drop abilities in Airflow.

We're expecting a few more improvements in the log generator. Currently, it's very clumsy.

I have used Apache Airflow for three years.

It's stable.

It's scalable. So far, we haven't needed more scalability because it's totally controlled by administrators.

The only difference between Apache Airflow and BPM software is the pricing.

Setup is about medium difficulty. You need to have some prior knowledge and experience with docker containers and AKS.

It's open-source.

I would rate this solution as seven out of ten.

Our primary use case for this solution is scheduling task rates. We capture the data from the SQL Server location and migrate it to the central data warehouse.

The best feature is the customization that can be done using Python. For example, there are use cases where we have to tweak the algorithm and with Apache Script Rate, we have extra functionality that helps to change the underlying process. We can define our algorithms and processes using Python.

The solution could be improved by simplifying the integration process and providing access to its support team to guide integration.

We have been using this solution for two months and it is deployed on-premises.

The solution is stable but primarily depends on the support team and how they manage it.

Apache Airflow is scalable. Approximately 20 people use this solution on my team.

We haven't had any experience with customer service and support.

Previously, we were using SQL server integration tools and integration service SSIS packages. We had project orders and wanted to migrate everything as it was an open source rate and no license was required. We switched to Apache Flow because we are trying to migrate all the projects developed in SSIS using Python.

The initial setup was straightforward. However, if a script is written, it takes four to five minutes to set up.

Apache Airflow is open source, so I cannot comment on licensing costs.

We chose this solution because it was suitable for our business needs.

I rate this solution a seven out of ten. My advice to new users is to have good proficiency with Python language. The solution is good but can be improved by simplifying its integration process.

There are a few use cases we have for Apache Airflow, one being government projects where we perform data operations on a monthly basis. For example, we'll collect data from various agencies, harmonize the data, and then produce a dashboard. In general, it's a BI use case, but focusing on social economy.

We concentrate mainly on BI, and because my team members have strong technical backgrounds we often fall back to using open source tools like Airflow and our own coded solutions. 

For a single project, we will typically have three of us working on Airflow at a time. This includes two data engineers and a system administrator. Our infrastructure model is hybrid, based both in the cloud and on-premises. 

The best part of Airflow is its direct support for Python, especially because Python is so important for data science, engineering, and design. This makes the programmatic aspect of our work easy for us, and it means we can automate a lot.

It's such a natural fit because our engineers are also Python-based, and I think we also quite like that we don't have to learn different kinds of UIs. Airflow is based on standard software packages, so we don't have to learn anything new in the way of opinionated UIs from different vendors.

We're currently using version 1.10, but I understand that there's a lot of improvements in version 2. In the earlier version that we're using, we sometimes have problems with maintenance complexity. Actually using Airflow is okay, but maintaining it has been difficult.

When something fails, it's not that easy to troubleshoot what went wrong. Sometimes the UI becomes really slow and there's no easy way to diagnose the problem. For the most part, we have had to learn through trial and error how to operate it properly. 

The UI is also not that attractive, and I feel that the user experience isn't that nice. Version 2 is supposedly better, but without having tried it, I could suggest more improvements in the visual UI. We want to do the ETL as code, but having a nice visual UI to facilitate this process would be great. Because that means we can also rely on non-technical staff, rather than just the three solid technical staff we have here. If there were better features for the UI, like drag-and-drop, then we could expand its use to more of our team.

I've been using Apache Airflow for about two and a half years. 

I think how Apache Airflow works is great. We like the paradigm of ETL as code, which means you define your pipeline as code. All the while, people talk about infrastructure as code, so the practice of ETL as code really fits into that philosophy.

We can scale it well, and it runs on cloud, too. It's compatible with cloud-native technologies like Kubernetes so it has no issues regarding elasticity.

We contacted an Airflow developer for assistance once and it was a good experience.

We like to explore different tools, mixing and matching them to our needs, but we have never really found any like Airflow that are to our liking. We tried looking into Talend and Alteryx but we didn't find them suitable to our style or approach.

As a first-time user, it was complex and somewhat difficult to set up as there are many components to put together. You've got your data portion, your scheduler portion, your web server portion, etc., and you've got all these parts to set up at first.

The next project that you get to, it gets easier. You really need to acquire a feel for what you're doing, and once you get over that, it's not too bad.

We implemented Airflow ourselves, with the help of our two in-house data engineers and system administrator. It took around three months to get it deployed initially, from concept into production. Then after that, the goal is just to operate it and keep it running.

Although Airflow is open source software, there's also commercial support for it by Astronomer. We personally don't use the commercial support, but it's always an option if you don't mind the extra cost.

I can recommend Apache Airflow, especially if there are serious data engineers on your team. If, on the other hand, you're looking to enable business users, then it's not suitable.

I would rate Apache Airflow an eight out of ten.

We use Apache Airflow for data orchestration.

Apache Airflow is a pretty useful tool for collecting information. Apache Airflow is a pretty easy solution that can be used with Python. The solution's UI allows me to collect all the information and see the code lines.

I have some issues with the solution's communication. The solution uses the same database or data set. Sometimes, we consume the same data and send it to a different place when doing a different DAG. When using the UI, I want to see that we use the same data set more than once.

I have been using Apache Airflow for five years.

I rate Apache Airflow a seven out of ten for stability.

I rate Apache Airflow an eight out of ten for scalability. Around 400 users are using the solution in our organization.

I previously used Control-M and some AWS and Google Cloud Platform tools.

Apache Airflow's initial setup is pretty straightforward. Apache Airflow is quite intuitive to set up and create DAGs.

It takes around two days to deploy Apache Airflow. A DAG can be created in just a few hours.

Apache Airflow is deployed on-cloud in our organization.

Overall, I rate Apache Airflow a nine out of ten.

Our primary use case for the solution is setting up workflows and processes applied everywhere because most industries are based on workflows and processes. We've deployed it for all kinds of workflows within the organization.

The ability to easily set up and deploy workflows with Airflows is valuable. Additionally, designing processes and workflows is easier, and it assists in coordinating all of the different processes.

The solution can be improved by creating a tool that allows us to do these kinds of things graphically instead of just writing scripts. Hence, the graphical user interface can be improved.

We have been using the solution for approximately one year and are currently using the latest version.

The solution is stable.

The solution is scalable. Approximately hundreds of thousands of people are utilizing it.

We have not had any issues that require customer service and support.

The initial setup is intermediate, and two people are required for deployment.

There is a return on investment because it's free, open source and very useful, so there is a significant return on investment.

I rate the solution an eight out of ten.