Azure Data Factory Reviews

We are currently migrating from on-prem to the cloud, and our on-prem tables are getting data from upstream. We used ADF to build a pipeline to facilitate this migration. A team of 15-20 people currently uses ADF, and more will join once it goes live.

ADF is another ETL tool similar to Informatica that can transform data or copy it from on-prem to the cloud or vice versa. Once we have the data, we can apply various transformations to it and schedule our pipeline according to our business needs. ADF integrates with Databricks. We can call our Databricks notebooks and schedule them via ADF. 

I have used Azure Data Factory for about six months.

I rate Azure Data Factory six out of 10 for stability. ADF is stable now, but we had problems recently with indexing on an SQL database. It's slow when dealing with a huge volume of data. It depends on whether the database is configured as general purpose or hyperscale. 

I rate Azure Data Factory eight out of 10 for ease of setup. The deployment time depends on the data volume. Four million records will take longer than four thousand. Migrating our full load from on-prem to the cloud took around 16-18 hours because the volume was 17 million. 

I rate ADF six out of 10 for affordability. The cost depends on the services we use. It's usage-based. 

I rate Azure Data Factory seven out of 10. Companies that want to migrate from on-prem to the cloud have lots of options. I haven't explored them all, but Azure, GCP, and AWS are essentially all the same.

The current use is for extracting data from Google Analytics into Azure SQL Database as a source for our EDW.  Extracting from GA was problematic with SSIS. 

The larger use case is to assess the viability of the tool for larger use in our organization as a replacement for SSIS for our EDW and also as an orchestration agent to replace SQL Agent for firing SSIS packages using Azure SSIS-IR.

The initial rollout was to solve the immediate problem while assessing its ability to be used for other purposes within the organization. And also establish the development and administration pipeline process.  

ADF allowed us to extract Google Analytics data (via BigQuery) without purchasing an adapter.  

It has also helped with establishing how our team can operate within Azure using both PaaS and IaaS resources and how those can interact. Rolling out a small data factory has forced us to understand more about all of Azure and how ADF needs to rely upon and interact with other Azure resources.

It provides a learning ground for use of DevOps Git along with managing ARM templates as well as driving the need to establish best practices for CI.  

The most valuable aspect has been a large list of no-cost source and target adapters.

It is also providing a PaaS ELT solution that integrates with other Azure resources. 

Its graphical UI is very good and is even now improving significantly with the latest preview feature of displaying inner activities within other activities such as forEach and If conditions.   

Its built-in monitoring and ability to see each activity's JSON inputs/outputs provide an excellent audit trail.

The trigger scheduling options are decently robust.

The fact that it's continually evolving is hopeful that even if some feature is missing today, it may be soon resolved. For example, it lacked support for simple SQL activity until earlier this year, when that was resolved. They have now added a "debug until" option for all activities. The Copy Activity Upsert option did not perform well at all when I first started using the tool but now seems to have acceptable performance.  

The tool is designed to be metadata driven for large numbers of patterned ETL processes, similar to what BIML is commonly used for in SSIS but much simpler to use than BIML. BIML now supports generating ADF code although with ADF's capabilities I'm not sure BIML still holds its same value as it did for SSIS.

The list of issues and gaps in this tool is extensive, although as time goes on, it gets shorter. It currently includes:

1) Missing email/SMTP activity

2) Mapping data flows requires significant lag time to spin up spark clusters

3) Performance compared to SSIS. Expect copy activity to take ten times that of what SSIS takes for simple data flow between tables in the same database

4) It is missing the debug of a single activity. The workaround is setting a breakpoint on the task and doing a "rerun from activity" or setting debug on activity and running up to that point

5) OAuth 2.0 adapters lack automated support for refresh tokens

6) Copy activity errors provide no guidance as to which column is causing a failure

7) There's no built-in pipeline exit activity when encountering an error

8) Auto Resolve Integration runtime should never pick a region that you're not using (should be your default for your tenant)

9) IR (integration runtime) queue time lag. For example, a small table copy activity I just ran took 95 seconds of queuing and 12 seconds to actually copy the data. Often the queuing time greatly exceeds the actual runtime

10) Activity dependencies are always AND (OR not supported). This is a significant missing capability that forces unnecessary complex workarounds just to handle OR situations when they could just enhance the dependency to support OR like SSIS does. Did I just ask when ADF will be as good as SSIS?  

They need to fix bugs. For example:

1) The debug sometimes stops picking up saved changes for a period of time, rendering this essential tool useless during that time

2) Enable interactive authoring (a critical tool for development) often doesn't turn on when enabled without going into another part of the tool to enable it. Then, you have to wait several minutes before it's enabled which is time you're blocked from development until it's ready.  And then it only activates for up to 120 minutes before you have to go through this all over again. I think Microsoft is trying to torture developers

3) Exiting the inside of an activity that contains other activities always causes the screen to jump to the beginning of a pipeline requiring re-navigating where you were at (greatly slowing development productivity)

4) Auto Resolve Integration runtime (using default settings) often picks remote regions (not necessarily even paired regions!) to operate, which causes either an unnecessary slowdown or an error message saying it's unable to transfer the volume of data across regions

5) Copy activity often gets the error "mapping source is empty" for no apparent reason. If you play with the activity such as importing new metadata then it's happy again. This sort of thing makes you want to just change careers. Or tools. 

I have been using this product for six months.

Production operation seems to run reliably so far, however, the development environment seems very buggy where something works one day and not the next. 

So far, the performance of this solution is abysmal compared to SSIS. Especially with small tasks such as copying activity from one table to another within the same database. 

Customer support is non-existent. I logged multiple issues only to hear back from 1st level support weeks later asking questions and providing no help other than wasting my time. In one situation it was a bug where the debug function stopped working for a couple of days. By the time they got back to me, the problem went away. 

Negative

We have been and still rely on SSIS for our ETL. ADF seems to do ELT well but I would not consider it for use in ETL at this time.  Its mapping data flows are too slow (which is a large understatement) to be of practical use to us. Also, the ARM template situation is impractical for hundreds of pipelines like we would have if we converted all our SSIS packages into pipelines as a single ADF couldn't take on all our pipelines. 

Initial setup is the largest caveat for this tool. Once you've organized your Azure environment and set up DevOps pipelines, the rest is a breeze. But this is NOT a trivial step if you're the first one to establish the use of ADF at your organization or within your subscription(s). Instead of learning just an ETL tool, you have to get familiar with and establish best practices for the entire Azure and DevOps technologies. That's a lot to take on just to get some data movements operational. 

I did this in-house with the assistance of another team who uses DevOps with Azure for other purposes (non-ADF use). 

The setup cost is only the time it takes to organize Azure resources so you can operate effectively and figure out how to manage different environments (dev/test/sit/UAT/prod, etc.). Also, how to enable multiple developers to work on a single data factory without losing changes or conflicting with other changes.

We operate only with SSIS today, and it works very well for us. However, looking toward the future, we will need to eventually find a PaaS solution that will have longer sustainability.

Azure Data Factory is an all-in-one solution for ETL in our company.

My company doesn't use the product for development purposes.

I use the solution in my company as an ETL tool and for orchestration.

As a DevOps engineer, I feel that the CI/CD part and the tool's integration with GitHub are the product's best features. If you compare it with other tools, like Glue, AWS, and other solutions, I feel Azure Data Factory's deployment part is a lot easier to manage. The code promotions and the data pipeline promotions to higher environments are a lot easier with Azure Data Factory.

The product's technical support has certain shortcomings, making it an area where improvements are required. Instead of sending out documents, I think the tool's support team should focus on how to troubleshoot issues. I want the tool's support team to have real-time interaction with users.

The product's price can be problematic for small businesses, making it an area where improvements are required.

I have experience with Azure Data Factory. I am the end user of the tool. Azure Data Factory is a PaaS solution. I use the solution's latest version.

It is a stable solution since it is a PaaS product. Stability-wise, I rate the solution an eight out of ten.

The scalability of the product is impressive. Scalability-wise, I rate the solution an eight out of ten.

Most of the people in my company work on Azure, and those who want to use the native ETL capabilities provided by the product opt for Azure Data Factory.

The product is useful in medium to large-sized businesses. Smaller businesses can opt for other options other than Azure Data Factory, considering the amount of money they are ready to spend. There are better options available in the market than Azure Data Factory.

I rate the technical support a five to six out of ten.

Neutral

I rate the product's initial setup phase a seven or eight on a scale of one to ten, where one is difficult and ten is easy.

In my company, we take care of the product's deployment process and maintenance phase.

The solution is deployed using Azure's cloud services.

The solution can be deployed in ten to fifteen minutes.

For deployments, my company usually creates codes in Terraform so that we can have automated deployments, and it is connected to us with a CI/CD tool like Azure DevOps. Azure DevOps does the automated deployment for our company.

During the setup phase, users may face issues when it comes to infrastructure deployment and the configuration around it, especially if you consider the integration runtime, as it is something that is too complicated for a normal developer to understand. There is a need for a cloud expert with a good understanding to be able to take care of the deployment in the right manner and in a secure way. The networking setup and security part of the product are a bit complicated, which I might understand since I am a DevOps engineer, but a developer who is new to the product might not understand such parts of the tool. The deployment of the service in an infrastructure can be possible only if the person involved in the deployment has a basic level of understanding related to the product.

I rate the product price as six on a scale of one to ten, where one is low price and ten is high price.

I wanted to compare Azure Data Factory with Fivetran.

Users rely on Azure Data Factory's connectors to meet data integration and transformation needs. Users use connectors that are native to Azure Data Factory. The tool offers more than 90 connectors that can be used to ingest data from different sources.

The feature of the solution I find to be the most beneficial for data management tasks is its connectors, and it can even be used for hybrid scenarios. The tool can connect to a different cloud, like AWS. The product can connect to your on-premises systems. In general, users are able to ingest data from everywhere, and the best part is that all of the aforementioned areas can be managed through GUI. The tool is like a low code-no code solution.

The visual interface of the solution impacts workflow efficiency because I think it is easier to start with for any developer who wants to use the tool. It is easier to start with and also easier to troubleshoot or debug, especially at a time when you cannot expect all your developers to understand codes. It would be good to have an intuitive GUI. Azure Data Factory

does a pretty good job when you compare it with its competitors.

Most of the time, my company uses integration runtime, so we mostly use a self-hosted integration runtime. In short, my company has not seen my impact has not seen much impact on a project from the product's scalability capabilities on any projects because we use it according to the needs of our customers.

I rate the tool an eight out of ten.

I use Azure Data Factory for architecture creation, for example, loading data from Oracle DB to Azure Synapse Analytics, creating facts and dimensions using Azure Data Pipeline, and creating Azure Synapse notebooks for data transformation. 

Another use case for Azure Data Factory is dashboard creation to help customers make informed decisions.

Compared to the on-premise SSIS, Azure Data Factory has better infrastructure. It also benefits my company because you can scale the solution up or down with different resources.

Azure Data Factory is also on a pay-as-you-go or pay-as-you-use model, which is suitable for the company because my company only pays for its usage or requirement.

The solution is also very user-friendly, and the Azure Data Factory support team responds quickly whenever my team has a loading issue.

One advantage of Azure Data Factory is that it's fast, unlike SSIS and other on-premise tools.

It's also very convenient because Azure Data Factory has multiple connectors. It has sixty connectors which you can't find in SSIS. The availability of native connectors allows you to connect to several resources to analyze data streams.

I also like that you can set up your own VM and infrastructure on Azure Data Factory without any help from the IT team because it only requires a single click.

What's missing in Azure Data Factory is an Oracle connector. If you want to connect directly to the Oracle database, you must copy and transform the data. There's no Oracle connector if you want to do transformation using data flow activity, so Azure Data Factory needs more connectors for data flow transformation.

Sending out emails after a job is completed is another area for improvement in the tool.

I've been using Azure Data Factory for three years.

Azure Data Factory is a scalable tool.

We used SSIS, but its on-premise version is slower than Azure Data Factory, and Azure Data Factory, infrastructure-wise, is better, so we went with Azure Data Factory.

The initial setup for Azure Data Factory is an eight out of ten.

Manually deploying Azure Data Factory is easy and doesn't take much time, but I'm not sure how long it takes for an automated approach to deployment.

The licensing model for Azure Data Factory is good because you won't have to overpay. Pricing-wise, the solution is a five out of ten. It was not expensive, and it was not cheap. It's in the middle.

I have experience with both Azure Data Factory and SSIS.

I'm using the latest version of Azure Data Factory.

My rating for Azure Data Factory is eight out of ten.

My company is an Azure Data Factory user.

As a management consultancy company, we help our clients deploy Azure Data Factory or any other cloud-based solution depending on data integration needs. Regarding how we use Azure Data Factory within our company, we are on the Microsoft Stack, so we use the solution primarily for data warehousing and integration.

The feature I found most helpful in Azure Data Factory is the pipeline feature, including being able to connect to different sources.

I also found running Python codes whenever you need to valuable in Azure Data Factory, especially for certain features of the solution, such as data integrations, aggregations, and manipulations.

Azure Data Factory also has built-in security, which is another valuable feature.

I also like that you get access to the whole Azure suite through Azure Data Factory, so the overall architecture design, defining security and access, role-based access management, etc. It's helpful to have the whole suite when designing applications.

Areas for improvement in Azure Data Factory include connectivity and integration.

When you use integration runtime, whenever there's a failure, the backup process in Azure Data Factory takes time, so this is another area for improvement.

Database support in the solution also has room for improvement because Azure Data Factory only currently supports MS SQL and Postgres. I want to see it supporting other databases.

If you want to connect the solution from on-premises to the cloud, you will have to go with a VPN or a pretty expensive route connection. A VPN connection might not work most of the time because you have to download a client and install it, so an interim solution for secure access from on-premise locations to the cloud is what I want to see in Azure Data Factory.

I've been using Azure Data Factory for about a year now.

Azure Data Factory is very stable, so it's a four out of five for me. In some instances, the solution failed, but I wouldn't wholly blame Azure Data Factory because my company connected to some on-premise databases in some cases. Sometimes, you'll get errors from self-hosted integration, faulty connections, or the on-premise server is down, so my rating for stability is a four.

Scalability-wise, Azure Data Factory is a four out of five because Microsoft is still developing certain tiers, which means you can't upgrade an older skill or tier. In contrast, the more modern, newer tiers could be upgraded easily. Rarely will you get stuck in one platform where you have completely destroyed that container and then fit a new container. Most of the time, Azure Data Factory is pretty easy to scale.

We haven't used Microsoft support directly because whenever we have issues with Azure Data Factory, we can find resolutions through their online documentation.

We're using both Azure Data Factory and SSIS.

We had several in-house solutions, but we moved to Azure Data Factory because it was straightforward. From a deployment standpoint, the solution comes with different services, so we didn't have to worry about separate hardware or infrastructure for networking, security, etc.

The initial setup for Azure Data Factory was easy, so I'd rate the setup a four out of five.

The implementation strategy was looking into what my organization needed overall, then planning and direct deployment. My company first did a test, a dummy version, then a UAT with stakeholders before going into production.

It took about two months to complete the deployment for Azure Data Factory.

An in-house team, the digital data engineering team, deployed Azure Data Factory.

We're still computing the ROI from Azure Data Factory. It's too early to comment on that.

My company is on a monthly subscription for Azure Data Factory, but it's more of a pay-as-you-go model where your monthly invoice depends on how many resources you use.

On a scale of one to five, pricing for Azure Data Factory is a four.

It's just the usage fees my company pays monthly. No support fees because my company didn't need support from Microsoft.

If you're not using core Microsoft products, the cost could be slightly higher, for example, when using a Postgres database versus an MS SQL database.

My company uses Azure Data Factory, SSIS, and for a few other instances, Salesforce.

My company currently has about fifty Azure Data Factory users, but not directly exposed to the solution compared to the developers; for example, members of corporate management and other teams apart from the development team are exposed to Azure Data Factory.

Shortly, there could be about two hundred users of Azure Data Factory within the company.

The developer team working directly on Azure Data Factory comprises ten individuals.

For the maintenance of the solution, my company has two to three staff, but it could reach up to eight or ten for the entire product. It's a mix of engineers and business analysts who handle Azure Data Factory maintenance.

I'd rate Azure Data Factory as eight out of ten.

My company is an end user of Azure.

We use the solution for building a few warehouses using Microsoft services.

We worked on a project for the textile industry where we needed to build a data warehouse from scratch. We provided a solution using Azure Data Factory to pull data from multiple files containing certification information, such as CSV and JSON. This data was then stored in a SQL Server-based data warehouse. We built around 30 pipelines in Azure Data Factory, one for each table, to load the data into the warehouse. The Power BI team then used this data for their analysis.

For the integration task, we used Azure Data Factory for orchestration and data flows for transformation. The data flows were beneficial, allowing us to perform multiple transformations. Additionally, we utilized web API activities to log data from third-party API tools, which greatly assisted in loading the necessary data into our warehouse.

When we initiated the cluster, it took some time to start the process. Most of our time was spent ensuring the cluster was adequately set up. We transitioned from using the auto integration runtime to a custom integration runtime, which showed some improvement.

I have been using Azure Data Factory for four years.

When running the process server, we encountered frequent connection disconnect issues. These issues often stemmed from internal problems that we couldn’t resolve then, leading to repeated disruptions.

I rate the stability as seven out of ten.

20 people are using this solution daily. I rate the scalability around eight out of ten.

Customer service supported us whenever we needed it.

Positive

We have used SQL Server.

The initial setup is easy and takes four to five hours to complete.

They have reduced the infrastructure burden by 60 percent.

Pricing is reasonable when compared with other cloud providers.

We have used the Key value pair for authentication with the adoption. I can rate it around eight out of ten.

I recommend the solution.

Overall, I rate the solution a nine out of ten.

The primary use case is to connect to various different data sets and do an EAT into our data warehouse.

We have been using drivers to connect to various data sets and consume data. The solution gives everything under one roof, which is an important feature.

We require Azure Data Factory to be able to connect to Google Analytics.

I have been using the solution for two years.

The solution is stable.

The solution is scalable.

We had a few technical calls with the Microsoft technical support team for some issues that we were facing, which they helped us resolve.

Positive

The initial setup is straightforward and the team is able to deploy between six and seven days.

The implementation was completed in-house.

The cost is based on the amount of data sets that we are ingesting. The more data we ingest the more we pay.

I give the solution a nine out of ten. We have been happy with all the customer implementations, and the customers are satisfied with the ADF pipelines. We are also currently examining the Synapse pipelines, which are likely similar.

We have six developers using the solution in our organization.

People should use the solution for two reasons. Firstly, we can switch off any data pipelines we set up to save costs. Secondly, there are several connectors available in one place, including most standard connectors.

We use Azure Data Factory to set up a data lake information repository.

Most of our customers are Microsoft shops and prefer Azure Data Factory because they have good licensing options and a trust factor with Microsoft. Our core customers are traditional Microsoft shops who prefer to expand on Microsoft on-cloud.

Azure Data Factory's pricing in terms of utilization could be improved. Our customers complain that the solution's bills keep growing.

I have worked with Azure Data Factory for more than five years as a consultant.

Azure Data Factory is a pretty stable solution. I rate Azure Data Factory a nine out of ten for stability.

Azure Data Factory is a scalable solution. We have around 15 to 20 customers, of which about 8 to 10 use Azure Data Factory.

Azure Data Factory's initial setup needs some amount of professional training and qualification.

It seems very low initially, but as the data grows, the solution’s bills grow exponentially.

Azure Data Factory is deployed on-cloud for our clients.

Azure Data Factory is a pretty solid solution with all the factors and integration built in. It's as good as any product.

I recommend users compare with Snowflake before choosing Azure Data Factory. We've had customers who prefer Snowflake just for its ease of use. Since we're not a Microsoft official reseller, I give options to customers and then let them pick and choose some from Azure Data Factory, Snowflake, or anything on Amazon.

Overall, I rate Azure Data Factory an eight out of ten.