Azure Data Factory Reviews

We use Azure Data Factory to build data analytics products.

Azure Data Factory helps in data integration and data orchestration in a self-service way, and it is a native component to the Azure platform.

The valuable feature of Azure Data Factory is its integration capability, as it goes well with other components of Microsoft Azure.

I'm not confident in highlighting any potential room for improvement with Azure Data Factory at this time. To the best of my knowledge, it is satisfactory as it is.

I have been using Azure Data Factory for the past six years.

I haven't encountered any stability issues with Azure Data Factory. However, I am not deeply technical and cannot comment on specifics.

Azure Data Factory is scalable enough to deal with medium to large-size projects.

Customer service is not satisfactory. Third-party personnel handle support and rely on a knowledge repository. Resolution times are long, and their ability to resolve issues could be improved.

Positive

In the past, Talend Data Integration Studio was used, however, Azure was chosen for better integration with other Microsoft Azure components.

Azure Data Factory does not require an initial setup since it's a cloud-based service.

We previously considered Talend for the same use case.

Azure Data Factory is specifically meant for data integration and nothing more. For reporting and other capabilities, different Microsoft tools should be used.

I'd rate the solution nine out of ten.

My main use case for Azure Data Factory is to pull data from on-premises systems. Most data transformation is done through Databricks, but Data Factory mainly pulls data into different services.

The solution handles large volumes of data very well. One of its best features is its ability to integrate data end-to-end, from pulling data from the source to accessing Databricks. This makes it quite useful for our needs.

The main challenge with implementing Azure Data Factory is that it processes data in batches, not near real-time. To achieve near real-time processing, we need to schedule updates more frequently, which can be an issue. Its interface needs to be lighter. 

One specific issue is with parallel executions. When running parallel executions for multiple tables, I noticed a performance slowdown.

I have been working with the product for five years. 

We haven't faced any issues with the tool's stability. 

The solution can handle large datasets. 

I am satisfied with Microsoft's support. They provide solutions to our challenges. 

Positive

The solution is cheap. 

I rate the overall product an eight out of ten. 

We use the solution for migration. We collect data from SAP and various other sources, including multiple ERP systems. These ERP systems encompassed different versions of SAP, Dynamics, Navision, and Oracle, presenting a considerable challenge for data integration. The objective was to consolidate all data into Azure Data Factory and Data Warehouse, establishing a structured framework for reporting and analytics. The main hurdle encountered was data ingestion, particularly with SAP data, due to its significant volume. Alternative tools such as PolyBase were utilized to expedite the process, as standard SAP APIs were insufficient for loading data into Azure Data Services. Collaboration with an Azure data engineer facilitated the exploration of alternative ingestion methods. 

The most important feature is the Data Catalog. We need to define all the data fields we test. It has technical information in the Data Catalog. The main feature is data ingestion in ADF. We also extended it to PurView because PurView is an extension of the Azure data catalog. It can scan metadata. There is a limitation in ADF when setting up the data catalog.

Integration with other tools, such as SAP, could be enhanced. It still has challenges when we talk about different types of structured and non-structured datages. Azure Data Factory has data ingestion issues. There are no delays out of the box. We needed a lot of tools to make the ingestion happen because of the data structure and size of the data.

The transformation we needed to do on data was also not so easy. It was also a long process. We had a bit more capabilities for setting up the Data Catalog, but it still didn't solve the problem from the data ingestion.

I have been using Azure Data Factory as a consultant for five years.

Sometimes, we experienced some instability, mainly on injection.

I rate the solution’s stability as seven out of ten.

I rate the solution’s scalability an eight out of ten.

The support is very good.

Positive

You need to be experienced in deploying the solution. It's not so easy for a business user. Depending on the use case, it takes around six months to get a proof of concept done.

I rate the initial setup a seven out of ten, where one is easy and ten is difficult.

The pricing is visible because you pay for what you do.

The product looks quite expensive because it charges based on the size of the data. If you're not aware, your cost can be very high. If you are experienced, you know that.

I rate the product’s pricing a seven out of ten, where one is cheap and ten is expensive.

I was mainly focusing on ingestion and cataloging. Data engineers were handling data orchestration.

The tool’s maintenance is easy.

There could be a bit more clarity in the pricing structure. It should be understandable for business users. The cost is is becoming too high because users are unaware of the pricing structure. Secondly, the tool should integrate better with other tools like ERP systems.

Overall, I rate the solution a seven out of ten.

We use Azure Data Factory for data integration.

The most valuable features of the solution are its ease of use and the readily available adapters for connecting with various sources.

Some known bugs and issues with Azure Data Factory could be rectified.

I have been using Azure Data Factory for about two years.

I rate the solution an eight out of ten for stability.

Azure Data Factory is a scalable solution. A team of 16 people from the data analytics team use the solution in our organization.

I rate the solution an eight out of ten for scalability.

On a scale from one to ten, where one is difficult and ten is easy, I rate the solution's initial setup a seven out of ten.

A team of three people deployed Azure Data Factory in three to four days.

The solution's pricing is competitive.

We build data pipelines primarily for integration. Few of them are real-time data transfers, and few of them would be a batch-free file. These would direct the data from various sources to our data warehouse. Azure Data Factory helps build the data pipelines and adaptors.

The solution has built-in features and a control center for us to monitor the status of the pipelines. The solution's email notification also helps us in monitoring. We didn't face any challenges to set up the data pipelines. We know there are some controls, but governance is customized for the organization's requirements. We have our own policies.

Azure Data Factory is deployed on the cloud in our organization. I would recommend Azure Data Factory to other users.

Overall, I rate the solution a nine out of ten.

We use the solution for building a few warehouses using Microsoft services.

We worked on a project for the textile industry where we needed to build a data warehouse from scratch. We provided a solution using Azure Data Factory to pull data from multiple files containing certification information, such as CSV and JSON. This data was then stored in a SQL Server-based data warehouse. We built around 30 pipelines in Azure Data Factory, one for each table, to load the data into the warehouse. The Power BI team then used this data for their analysis.

For the integration task, we used Azure Data Factory for orchestration and data flows for transformation. The data flows were beneficial, allowing us to perform multiple transformations. Additionally, we utilized web API activities to log data from third-party API tools, which greatly assisted in loading the necessary data into our warehouse.

When we initiated the cluster, it took some time to start the process. Most of our time was spent ensuring the cluster was adequately set up. We transitioned from using the auto integration runtime to a custom integration runtime, which showed some improvement.

I have been using Azure Data Factory for four years.

When running the process server, we encountered frequent connection disconnect issues. These issues often stemmed from internal problems that we couldn’t resolve then, leading to repeated disruptions.

I rate the stability as seven out of ten.

20 people are using this solution daily. I rate the scalability around eight out of ten.

Customer service supported us whenever we needed it.

Positive

We have used SQL Server.

The initial setup is easy and takes four to five hours to complete.

They have reduced the infrastructure burden by 60 percent.

Pricing is reasonable when compared with other cloud providers.

We have used the Key value pair for authentication with the adoption. I can rate it around eight out of ten.

I recommend the solution.

Overall, I rate the solution a nine out of ten.

We are currently migrating from on-prem to the cloud, and our on-prem tables are getting data from upstream. We used ADF to build a pipeline to facilitate this migration. A team of 15-20 people currently uses ADF, and more will join once it goes live.

ADF is another ETL tool similar to Informatica that can transform data or copy it from on-prem to the cloud or vice versa. Once we have the data, we can apply various transformations to it and schedule our pipeline according to our business needs. ADF integrates with Databricks. We can call our Databricks notebooks and schedule them via ADF. 

I have used Azure Data Factory for about six months.

I rate Azure Data Factory six out of 10 for stability. ADF is stable now, but we had problems recently with indexing on an SQL database. It's slow when dealing with a huge volume of data. It depends on whether the database is configured as general purpose or hyperscale. 

I rate Azure Data Factory eight out of 10 for ease of setup. The deployment time depends on the data volume. Four million records will take longer than four thousand. Migrating our full load from on-prem to the cloud took around 16-18 hours because the volume was 17 million. 

I rate ADF six out of 10 for affordability. The cost depends on the services we use. It's usage-based. 

I rate Azure Data Factory seven out of 10. Companies that want to migrate from on-prem to the cloud have lots of options. I haven't explored them all, but Azure, GCP, and AWS are essentially all the same.

Azure Data Factory is primarily used to orchestrate workflows and move data between various sources. It supports both ETL and ELT  processes. For instance, if you have an ERP system and want to make the data available for reporting in a data lake or data warehouse, you can use Data Factory to extract data from the ERP system as well as from other sources, like CRM systems.

Data Factory allows you to pull data from multiple systems, transform it according to your business needs, and load it into a data warehouse or data lake. It also supports complex data transformations and aggregations, enabling you to generate summary and aggregate reports from the combined data. Data Factory helps you ingest data from diverse sources, perform necessary transformations, and prepare it for reporting and analysis.

I have extensive experience building things independently, with over twenty years of experience in SQL, ETL, and data-related projects. Recently, I have been using Azure Data Factory for the past two years. It has proven to be quite effective in handling large volumes of data and performing complex calculations. It allows for the creation of intricate data workflows and processes faster. Azure Data Factory is particularly useful for enterprise-level data integration activities, where you might deal with millions of records, such as in SAP environments. For example, SAP tables can contain tens or hundreds of millions of records. Managing and maintaining the quality of this data can be challenging, but Azure Data Factory simplifies these tasks significantly.

It is a powerful tool and is considered one of the leading solutions in the market, especially for handling large volumes of data. It is popular among large enterprises.

While it has a range of connectors for various systems, such as ERP systems, the support for these connectors can be lacking. Take the SAP connector, for example. When issues arise, it can be challenging to determine whether the problem is on Microsoft's side or SAP's side. This often requires working with both teams individually, which can lead to coordination issues and delays. It would be beneficial if Azure Data Factory provided better support and troubleshooting resources for these connectors, ensuring a smoother resolution of such issues.

I have been using Azure Data Factory for two years.

I rate the solution's stability a nine out of ten.

It's pretty good. There are no issues with scalability.

The support has been good.

Positive

It is straightforward to set up. However, ensuring its security requires careful configuration, which can vary depending on the organization's requirements. While the basic setup is user-friendly and doesn’t necessarily require advanced technical skills, securing the environment involves additional steps to prevent unauthorized access and ensure that data is only accessible from permitted locations. This can be more complex depending on the specific setup and organizational needs.

Setting up the infrastructure typically takes about two to three weeks and usually requires the effort of two people.

Azure Data Factory serves several important purposes. One key reason for using it is to build an enterprise data warehouse. This is crucial for centralizing data from various sources. Another reason is to gain insights from that data. By consolidating data in a unified location, you enable data scientists and engineers to analyze it and generate valuable insights.

Customers use Azure Data Factory to bring their data together, creating opportunities to understand their data better and extract actionable insights. However, simply consolidating data is not enough; the actual value comes from how you analyze and utilize it. This involves deriving insights, creating opportunities, and understanding customers better, which can significantly benefit the organization.

Pricing is fine. It's a pay-as-you-go option.

It is in the same price range as other major providers. However, costs can vary depending on enterprise agreements and relationships.

Overall, I rate the solution a nine out of ten.

We use the product for data warehouses. It helps us to load data to warehouses. 

The tool's most valuable features are its connectors. It has many out-of-the-box connectors. We use ADF for ETL processes. Our main use case involves integrating data from various databases, processing it, and loading it into the target database. ADF plays a crucial role in orchestrating these ETL workflows. 

The tool's visual interface is good. The ADS scheduling feature impacts data management by determining when jobs must be run and setting up dependencies. This capability eliminates the need to rely on enterprise data scheduling tools. 

When working with AWS, we have noticed that the difference between ADF and AWS is that AWS is more customer-focused. They're more responsive compared to any other company. ADF is not as good as AWS, but it should be. If AWS is ten out of ten, ADF is around eight out of ten. I think AWS is easier to understand from the GUI perspective compared to ADF.

I have been using the product for 6 months. 

ADF is stable. 

I rate the tool's scalability an eight out of ten. 

The tool's deployment is easy. The deployment typically takes around two to three days to set up. However, the duration may vary depending on factors such as the number of integrated endpoints. In our company, the deployment team had three to four people. This team consisted of an IT engineer, a network engineer, and an ETL admin.

We still haven't required much maintenance since we're still in the development phase. However, as time progresses and we move into production, we'll better understand the maintenance requirements.

ADF is cheaper compared to AWS. 

The tool has met our projects' growing data needs effectively so far. I rate it an eight out of ten.