Apache NiFi Reviews

One example is how Apache NiFi has helped us to create data pipelines to migrate data from Oracle to Postgres, Oracle to Oracle, Oracle to Minio, or other databases, such as relational databases, NoSQL databases, or object storage. We create templates for these pipelines so that we can easily reuse them for different data migration projects.

For example, we have a template for migrating data from Oracle to Postgres. This template uses an incremental load process. The template also checks the source and destination databases for compatibility and makes any necessary data transformations.

If our data is not more than ten terabytes, then NiFi is mostly used. But for a heavy table setup, I don't use NiFi for customers or enterprise solutions.

I use custom functions for specific features in Apache NiFi. I also use the processes available in NiFi. I can write custom functions to achieve the desired functionality, even if it is not explicitly available as a built-in NiFi feature.

Apache NiFi is slow to control and needs to be improved. I have to run many jobs and there are already large tables, which can make it difficult to control NiFi on time.

There is no one to tell me when there is an incident and my server is down. When we manually start the NiFi process, it is not always started correctly. We can write scripts to run when a message is received from Airflow saying that the firewall is not running. This script will automatically start all servers, including the application servers. It will also check the status of all my NiFi processes and send a callback message with the results. I have written down all the processes that are monitored.

We run many jobs, and there are already large tables. When we do not control NiFi on time, all reports fail for the day. So it's pretty slow to control, and it has to be improved.

In future releases, there are extra features I’d like to add. For example, NiFi is not suitable for migration, and the replication in NiFi is really not good. Because when you process ten years of data, you can't manage all the transactions; it is not enough. Moreover, the handling of monthly transactions is not enough due to a lack of partitioning for dates. And, when we grade a monthly ticket, we must process all data then rerun our ETL jobs. If it's possible, enhancing the partitioning in NiFi for features would be beneficial.

I have been working with Apache NiFi for one year. 

I would rate the stability an eight out of ten. 

I would rate the scalability a five out of ten because, in our experience, it doesn't scale correctly, especially if you don't use a Kubernetes system. 

If you want it to be scalable, you must use Kubernetes, but in our system, it's in VM and VM disc—external and not external. Increasing disc space is a very hard process. NiFi is not easily scalable. You can increase, but decreasing is not possible. So, it is easy to scale up, but scaling down is difficult.

There are around ten end users in our company. We plan to increase the further usage. 

The initial setup is very easy. I would rate my experience with the initial setup a ten out of ten, where one point is difficult, and ten points are easy.

But if you want its custom mode and control, it's five out of ten. 

For the initial setup, if you configure to custom mode, it's five points. But if you use its single-mode configuration and installation, it's ten.

The deployment takes one week due to network access and some VM installation. Then, we install NiFi and deploy it. But, if you have all the scripts written automatically, it’s five minutes for us.

One person is enough for the deployment process. It's all about script writing in CAC, and it's one-button quick for deployment.

I am using it open source, so it means it's free for me to use. 

If the volume is manageable, I would recommend it. Overall, I would rate the solution a six out of ten. 

On-premises

I am implementing the ETL workflow using Apache NiFi to prepare data and upload it to the cloud. Our use case involves importing data from on-premise and private servers to build a data hub and data mart. The data mart is then published on the cloud.

We primarily use Apache NiFi for data preparation tasks.

The visual workflow aspect of Apache NiFi is an invaluable feature as it operates on a no-code platform that allows for easy drag-and-drop pipeline construction. Compared to Airflow, which requires programming before visual representation, Apache NiFi offers clarity in pipeline activities. This feature greatly aids in understanding what the pipeline is doing.

The logging system of Apache NiFi needs improvement. It is difficult to debug compared to Airflow, where task details and issues are clear. With Apache NiFi, I have encountered processes that die without any traceable error, which might relate to the inadequate logging system.

I have been working with Apache NiFi for about six months.

Sometimes, when I run Apache NiFi, processes crash without any clue, which might relate to the logging system. The process can die, and the logs do not show any detail to identify the problem, impacting stability.

For scalability, I would rate it an eight. We can run parallel pipelines simultaneously without issues unless memory is full. Scarcity of memory is the only constraint, but processing capabilities allow us to handle much simultaneously.

The technical support from the official Apache team is rated a three out of ten. Issues often require self-resolution or community help, as the support isn't effectively managed.

Negative

I have used Airflow before, which required programming first and then visual representation of the workflow.

There is another team responsible for setting up Apache NiFi, so I'm not involved in the deployment process.

Apache NiFi is open-source and free. Its integration with systems like Cloudera can be expensive, but Apache NiFi itself presents the best pricing as a standalone tool.

Prior to Apache NiFi, I used Airflow, which differed mainly in its approach to programming and workflow visualization.

Overall, I rate Apache NiFi an eight out of ten. I am quite happy with it.

On-premises

We use the solution for data streaming.

We use the tool to stream live data. The end users can see the real-time data.

We can integrate the tool with other applications easily.

More features must be added to the product. As compared to Kafka, the tool must be improved.

I have been using the solution for two years. I am using the version that was released before the latest version.

The solution’s stability is good. I rate the stability a seven out of ten.

More than 100 people are using the solution in our organization. We can scale the tool easily. I rate the scalability a ten out of ten.

We find most of the solutions to our issues on the internet. We didn’t have to approach the technical support team.

The initial setup was straightforward. We can deploy the tool easily on a single node. It won’t take much time. If it is a multi-node cluster, it will take two to three hours.

We need two engineers to maintain the solution. These engineers maintain other solutions in our organization, too.

The solution is open-source.

The solution must be improved to compete with Kafka. As it is an open-source tool, it will take time to get all the functions. I would recommend the product to others. Overall, I rate the product a seven out of ten.

On-premises

Our company uses the solution to ingest raw data. We have five repositories with a huge amount of data. We normalize the data to previously structured files, prepare it, and ingest it to devices. 

The size of any project team depends on the workflow or management activities but typically includes two to five users.

The user interface is good and makes it easy to design very popular workflows. 

There are nice parameters for migration.

It is very easy to schedule jobs that realize improvements and monetize.

The use case templates could be more precise to typical business needs. Available templates and model workflows are very high-level so don't really match real needs. It would help to have templates that allow us to see business opportunities. 

It would help to be able to copy workflow to another device rather than having to ingest it. 

I have been using the solution for five years. 

The stability is good so is rated an eight out of ten. 

The scalability can be a little bit limited in the 2020 version so is rated a seven out of ten. 

We contacted technical support by phone once and they were helpful, but there was some delay with their response. We would have preferred a quicker response time. 

After a bit of study, we can support most of our needs so don't really need technical support. 

The setup is very simple. We use a VM on our private cloud and install the solution on that VM. 

We implemented the solution in-house. We are a simpler, nonproductive environment and our implementation took about two hours. 

Implementation complexity and time depend on the use case. A production environment might take a bit longer to implement because there might be cluster situations. Environments with many rules or policies could also take longer to implement. 

One person with the right skills and knowledge can handle ongoing maintenance. 

I rate the solution an eight out of ten.

Private Cloud

Amazon Web Services (AWS)

Apache NiFi is used to orchestrate ingestion processes. For example, Apache NiFi ingests data from external sources such as external databases or external APIs. Custom transformation is then applied, and data is written inside the data lake.

Apache NiFi speeds up ingestion pipelines development. Ingestion pipelines that usually took a week to develop can now be developed in a couple of days.

Apache NiFi has extensive integration capabilities and integrates with many sources. It supports custom transformations, making it a very flexible tool that can be leveraged to perform most computation needs.

For transformation with Apache NiFi, JSONs are processed and denormalized to map information onto different tables. For source integration, the most valuable aspect was the ingestion from external APIs.

Apache NiFi is a very good tool, but there is room for improvement.

Apache NiFi has been used on different projects for a couple of years.

Apache NiFi should be considered if a scalable and flexible tool is needed for building ETL pipelines and reducing time to production. This review has a rating of 8.

We use the tool to transfer data from one service to another. It helps us to migrate data from one department to another. 

Apache NiFi is user-friendly. Its most valuable features for handling large volumes of data include its multitude of integrated endpoints and clients and the ability to create cron jobs to run tasks at regular intervals.

The tool should incorporate more tutorials for advanced use cases. It has tutorials for simple use cases. 

I rate the tool's stability an eight out of ten.

I have relied on the documentation available on Apache NiFi's website for support. 

I tried to install the tool on my work laptop, and while it worked initially, it started to run slowly after some time. The department that handles the company's databases uses Apache NiFi on proper servers. I tried using it on my laptop to see if it worked, but it ran very slowly and consumed many resources from my machine.

I used the tool's free version.

I rate Apache NiFi an eight out of ten. 

I use Apache NiFi to build workflows.

It's an event that is used for distributed messaging. You need to transfer the message that comes into Kafka Broker Topic.

You get the messages in the Kafka queue topic, you transform it and send it to other entities for storage, or you can return it back to Kafka to send to the consumer.

It's very handy.

I like building the independent flow. 

It's an automated flow, where you can build a flow from source to destination, then do the transformation in between.

Integration with Kafka and different data formats would be helpful, not only in JSON but in Avro, which is the data platform that is in a universal format.

I think the UI interface needs to be more user-friendly. The UI portal needs a lot of improvement for people who have less experience.

I have been using this solution for a year.

It's very stable, but I have not used it in production. I have used it in development and QA and I have tested it on various platforms with Kafka distributed messaging.

I have not needed Apache support. With my seniority, I have not used anything that requires their support.

It's a good thing to be able to go back in the documentation and the information is enough to get you going. 

Juniors or people who have less experience may have a lot of different questions, but I have not tested their support.

You have to install it on your platform using the Windows and Linux Operating System, or any Mac OS.

The Apache documentation is very straightforward. You can do it independently of any other guidance. 

It's self-explanatory. 

You can do it yourself without any hiccups, It's straightforward.

I am not involved in this area. I don't have any visibility in the costing and licensing.

We have a department that goes through the licensing and once we approve the functions and the performance of Apache NiFi, they deal with it.

You have to independently develop a use case for using NiFi, then test it and do the integration to have a stable workflow. After this, you jump in and integrate with Kafka distributed messaging and also connecting to your MDM database.

I would rate Apache NiFi an eight out of ten.

The most valuable feature has been the range of clients and the range of connectors that we could use. The processing ability of the solution is also really good because it allows high volumes of data to flow in real time. 

Overall, this solution is reliable. 

The overall stability of this solution could be improved. In a future release, we would like to have access to more features that could be used in a parallel way. This would provide more freedom with processing. 

I have been using this solution for one year. 

This is a stable solution although the stability can depend on the process selection and how well we apply for it. 

I have not needed to contact their customer service and support. 

The initial setup is not very complex.

We use the free version of Apache NiFi.

I would rate this solution an eight out of ten.

On-premises

It is really good when it comes to dealing with pipelines.

Visually, this is a good product.

The challenge with Apache NiFi is that it's not cloud-native. This makes it different from our workflow.

The operations are over-complicated and when you build your pipeline, it's a nightmare to follow them. Then, as your pipeline or workflow becomes more complex, the operation of it gets worse. It is not easy to use and it requires a bigger ramp-up than any other solution that we have seen.

We have been using Apache NiFi for between five and six years.

This is a very solid product.

Scaling this product means using a bigger and more powerful machine. If it were cloud-native then it would be different.

We have used similar solutions like Apache Airflow, StreamSets, and others. The ramp-up for NiFi is a lot bigger and you have to spend a lot of time to understand how to use it.

There are some claims that NiFi is cloud-native but we have tested it, and it's not.

I would rate this solution a seven out of ten.

On-premises

The primary use case is to collect data from different source systems. This includes different soft types of files, such as text files, bin files, and CSV files, to name a few. It is also used for API training.

It works for a large amount of data. It is oriented for endpoint solutions and high and low frequency in small packets of data, for example, files. It can also work well when integrated with Spark, they are complementary in some use cases. At times we work only with Nifi, at times only with Spark and other times when they are integrated.

The most valuable features of this solution are ease of use and implementation. It is very visual. We can orchestrate all of the data flow right on the user interface.

We can connect each processor that we design and we can make it quicker with faster development. This is an advantage in terms of development.

Regarding the use, it is very powerful because it scales easily based on the number of workers and nodes that you have. Also, the expiration based on the computation is very easy to configure.

There should be a better way to integrate a development environment with local tools. Most of the development is done on the console. For example, in Spark, we can develop on our local desktop and then deploy it to another environment. The integration using this solution is not simple.

Eventually, we can implement a local environment on our machines that is web-oriented and we have a browsing console to do it.

At times, it is not easy to integrate with other components that could be a part of the entire solution in terms of development.

There are issues with stability due to memory. It would be good to include a lock or an alarm to detect or alert you that it needs more resources.

If the solution could be integrated more it would repair every part of the flow. Eventually, if we are integrated with other technology, we need to have a notion that we need to plan and have the correct sizing. We could implement an ecosystem that could scale with the requirements.

In the next release, I would like to see the support of monitorization with the interface. Also an integrated development environment.

I have been working with Apache NiFi for a year and a half.

It's very stable if used properly. If we use it the way it is intended then it should work fine, it's a good solution.

There can be some issues with memory. If there were mechanisms in place to detect issues beforehand, that there was a need for available resources, it would be much easier.

This solution is very scalable, depending on how the solution is implemented.

We have four or five administrators who manage this solution.

It depends on the frequency of the data source and the use case on often it is being used, but generally, it is used intensively.

Because it is an open-source solution, there are forums and a community that will exchange information.

We used other solutions previously, but this is the best one. It was more stable, it is easier to use and it deploys quickly.

In terms of the setup, the most complex part is if you want to put it on the cloud. On a private cloud, it's tricky. If you set up on-premises, on the other hand, it's easier.

It can take a week or less to deploy the environment. 

We are part of the multinational department and they have their team in place. In my company, we do end to end. If you understand the plan it is easy to install a local environment to develop.

On the other end, if you are a customer then you need other departments to install it. Mainly departments for the infrastructure are required because there are certain requirements that the development team may not have. For example, contacts, security, and setting up hardware.

It's an open-source solution.

The architect needs to evaluate the entire architecture with this platform so eventually, we are left with our architects and we need to get approval from them to do that.

I would rate this solution an eight out of ten.

Hybrid Cloud