Azure Data Factory Reviews

My main use case for Azure Data Factory is to pull data from on-premises systems. Most data transformation is done through Databricks, but Data Factory mainly pulls data into different services.

The solution handles large volumes of data very well. One of its best features is its ability to integrate data end-to-end, from pulling data from the source to accessing Databricks. This makes it quite useful for our needs.

The main challenge with implementing Azure Data Factory is that it processes data in batches, not near real-time. To achieve near real-time processing, we need to schedule updates more frequently, which can be an issue. Its interface needs to be lighter. 

One specific issue is with parallel executions. When running parallel executions for multiple tables, I noticed a performance slowdown.

I have been working with the product for five years. 

We haven't faced any issues with the tool's stability. 

The solution can handle large datasets. 

I am satisfied with Microsoft's support. They provide solutions to our challenges. 

The solution is cheap. 

I rate the overall product an eight out of ten. 

We use the solution for building a few warehouses using Microsoft services.

We worked on a project for the textile industry where we needed to build a data warehouse from scratch. We provided a solution using Azure Data Factory to pull data from multiple files containing certification information, such as CSV and JSON. This data was then stored in a SQL Server-based data warehouse. We built around 30 pipelines in Azure Data Factory, one for each table, to load the data into the warehouse. The Power BI team then used this data for their analysis.

For the integration task, we used Azure Data Factory for orchestration and data flows for transformation. The data flows were beneficial, allowing us to perform multiple transformations. Additionally, we utilized web API activities to log data from third-party API tools, which greatly assisted in loading the necessary data into our warehouse.

When we initiated the cluster, it took some time to start the process. Most of our time was spent ensuring the cluster was adequately set up. We transitioned from using the auto integration runtime to a custom integration runtime, which showed some improvement.

I have been using Azure Data Factory for four years.

When running the process server, we encountered frequent connection disconnect issues. These issues often stemmed from internal problems that we couldn’t resolve then, leading to repeated disruptions.

I rate the stability as seven out of ten.

20 people are using this solution daily. I rate the scalability around eight out of ten.

Customer service supported us whenever we needed it.

Positive

We have used SQL Server.

The initial setup is easy and takes four to five hours to complete.

They have reduced the infrastructure burden by 60 percent.

Pricing is reasonable when compared with other cloud providers.

We have used the Key value pair for authentication with the adoption. I can rate it around eight out of ten.

I recommend the solution.

Overall, I rate the solution a nine out of ten.

We mainly use it to migrate data from on-premises sources to the cloud, such as Oracle and Cisco servers. It's a good solution for integrations within the Azure environment, and it connects well with other Azure data products. However, for external configurations, we use Informatica Cloud or Informatica Data Accelerator (IDA).

For automation, we primarily rely on Snowflake and Informatica. Our strategy is not to depend on a single tool. When it's strictly on-premises to cloud, we use ADF. 

Otherwise, Informatica is more mature and integrates well with various third-party products. We also use Snowflake copy commands to load data into Snowflake. Azure Data Factory doesn't fully meet your automation requirements.

We use Informatica for pipelines that were originally in SSIS, but for new pipelines and ETL processes, we choose either Informatica or Snowflake scripts.

We have multiple banking applications running on SSIS pipelines. We're in the process of upgrading them to a hybrid cloud architecture. For that, we use Azure Data Factory to move data from on-premises to the cloud – mainly for back-end database operations and ETL transformations.

We primarily use it to load data from an on-premises SQL Server to either Blob storage or an Azure SQL data warehouse. For other integrations, especially those outside of Azure, we tend to use Informatica Cloud Services (ICS).

For structured data loading, we use it However, we use Informatica for unstructured or semi-structured data. We also use Snowflake for ETL processes and sometimes for streaming. 

In my opinion, ADF isn't as suitable for streaming – for streaming, Snowflake streamlets or Informatica structured streaming are more reliable. ADF works well for batch processing, though.

I like its integration with SQL pools, its ability to work with Databricks, its pipelines, and the serverless architecture are the most effective features.

There is room for improvement primarily in its streaming capabilities. For structured streaming and machine learning model implementation within an ETL process, it lags behind tools like Informatica. 

Snowflake is also more efficient for loading data into Snowflake, whether from Blob storage or AWS. From our experience, ADF is mainly useful for batch processing. I'm not sure how its streaming capabilities compare to others for industry-wide use cases.

I've been using Azure Data Factory for the past two years.

I would rate the stability an eight out of ten.

When it comes to scalability and handling large datasets, it works well for datasets within the Azure environment because it's tightly integrated. 

However, for third-party integrations – things like SAP HANA, MongoDB, or handling semi-structured and unstructured data from logs – it's not as reliable. ADF excels with tight Azure cloud integration.

There are around seven end-users. Across the enterprise, Informatica is our main tool because it includes data governance (DG/DM), data quality (DQ), data cataloging, API integration, and streaming capabilities – like IDM and Informatica Cloud Services (ICS) as a SaaS platform hosted on either AWS or Azure. 

We are transitioning SSIS pipelines to ADF, but otherwise, Informatica is our central tool.

I would rate the scalability an eight out of ten.

The customer service and support are good. 

Positive

I primarily work with Informatica, Azure, and Snowflake for data pipeline tasks.

We've used Talend, SAP BODS, and AWS Glue. Oracle Data Integrator (ODI).

Now, we primarily use products within the Azure ecosystem, like Snowflake and Informatica Cloud Services (ICS). Specifically, we use Informatica Data Management Cloud (IDMC).

Our goal is to migrate from on-premises to IDMC, and we've been using ADF for the past two years to convert SSIS scripts.

That's likely handled by your cloud infrastructure team.

We have an internal cloud infrastructure team. Our data engineering team is quite large, around 30 people, since multiple projects are happening. Within my immediate team, there are seven people.

The maintenance is primarily handled by our database administrators (DBAs). Our involvement is mainly focused on building data pipelines and the ETL process as part of the data engineering team.

We sometimes need to look into performance issues due to it being in the cloud. However, Snowflake is much easier to maintain – that's all handled by Snowflake itself. Overall, Informatica and Snowflake are less demanding in terms of maintenance compared to ADF.

For our use case, it is not expensive. We take into the picture everything: resources, learning curve, and maintenance. 

I would rate the price a six out of ten. It is moderate pricing. 

I would definitely recommend using it It's a good tool. Because there isn't a comparable native Azure product for cloud-based integrations, using ADF is often necessary. If working with multiple clouds (Azure, AWS, etc.), we end up using tools like Informatica or Snowflake.

Overall, I would rate the solution a seven out of ten. It has some limitations, especially with streaming data.

Compared to Talend, Snowflake, or Informatica, which have rich screen-based GUIs, ADF's visual capabilities are weaker. There's a steeper learning curve, as you need to understand its technical UIs and data flows. 

However, with Microsoft's acquisition of Power BI, I suspect they might integrate these capabilities as 'Microsoft Fabric' in the future.

Azure Data Factory is primarily used to orchestrate workflows and move data between various sources. It supports both ETL and ELT  processes. For instance, if you have an ERP system and want to make the data available for reporting in a data lake or data warehouse, you can use Data Factory to extract data from the ERP system as well as from other sources, like CRM systems.

Data Factory allows you to pull data from multiple systems, transform it according to your business needs, and load it into a data warehouse or data lake. It also supports complex data transformations and aggregations, enabling you to generate summary and aggregate reports from the combined data. Data Factory helps you ingest data from diverse sources, perform necessary transformations, and prepare it for reporting and analysis.

I have extensive experience building things independently, with over twenty years of experience in SQL, ETL, and data-related projects. Recently, I have been using Azure Data Factory for the past two years. It has proven to be quite effective in handling large volumes of data and performing complex calculations. It allows for the creation of intricate data workflows and processes faster. Azure Data Factory is particularly useful for enterprise-level data integration activities, where you might deal with millions of records, such as in SAP environments. For example, SAP tables can contain tens or hundreds of millions of records. Managing and maintaining the quality of this data can be challenging, but Azure Data Factory simplifies these tasks significantly.

It is a powerful tool and is considered one of the leading solutions in the market, especially for handling large volumes of data. It is popular among large enterprises.

While it has a range of connectors for various systems, such as ERP systems, the support for these connectors can be lacking. Take the SAP connector, for example. When issues arise, it can be challenging to determine whether the problem is on Microsoft's side or SAP's side. This often requires working with both teams individually, which can lead to coordination issues and delays. It would be beneficial if Azure Data Factory provided better support and troubleshooting resources for these connectors, ensuring a smoother resolution of such issues.

I have been using Azure Data Factory for two years.

I rate the solution's stability a nine out of ten.

It's pretty good. There are no issues with scalability.

The support has been good.

Positive

It is straightforward to set up. However, ensuring its security requires careful configuration, which can vary depending on the organization's requirements. While the basic setup is user-friendly and doesn’t necessarily require advanced technical skills, securing the environment involves additional steps to prevent unauthorized access and ensure that data is only accessible from permitted locations. This can be more complex depending on the specific setup and organizational needs.

Setting up the infrastructure typically takes about two to three weeks and usually requires the effort of two people.

Azure Data Factory serves several important purposes. One key reason for using it is to build an enterprise data warehouse. This is crucial for centralizing data from various sources. Another reason is to gain insights from that data. By consolidating data in a unified location, you enable data scientists and engineers to analyze it and generate valuable insights.

Customers use Azure Data Factory to bring their data together, creating opportunities to understand their data better and extract actionable insights. However, simply consolidating data is not enough; the actual value comes from how you analyze and utilize it. This involves deriving insights, creating opportunities, and understanding customers better, which can significantly benefit the organization.

Pricing is fine. It's a pay-as-you-go option.

It is in the same price range as other major providers. However, costs can vary depending on enterprise agreements and relationships.

Overall, I rate the solution a nine out of ten.

I can describe a scenario where I was tasked with developing a beta data trace and integration system. GitLab served as the data integration platform responsible for creating pipelines to extract data from various sources, including on-premises and cloud-based systems. Alongside GitLab Data Factory, we utilized Azure Logic Apps for orchestration and Azure Key Vault for securely storing data-related information. This combination enabled us to manage data extraction, transformation, and loading efficiently. The solution also involved Azure Data Lake for further data transformations, culminating in a comprehensive data processing engine that I played a key role in implementing.

The workflow automation features in GitLab, particularly its low code/no code approach, are highly beneficial for accelerating development speed. This feature allows for quick creation of pipelines and offers customization options for integration needs, making it versatile for various use cases. GitLab supports a wide range of connectors, catering to a majority of integration needs. Azure Data Factory's virtual enterprise and monitoring capabilities, the visual interface of GitLab makes it user-friendly and easy to teach, facilitating adoption within teams. While the monitoring capabilities are sufficient out of the box, they may not be as comprehensive as dedicated enterprise monitoring tools. GitLab's monitoring features are manageable for production use, with the option to integrate log analytics or create custom dashboards if needed.

The data flow feature in Azure Data Factory within GitLab is valuable for data transformation tasks, especially for those who may not have expertise in writing complex code. It simplifies the process of data manipulation and is particularly useful for individuals unfamiliar with Spark coding. While there could be improvements for more flexibility, overall, the data flow feature effectively accomplishes its purpose within GitLab's ecosystem.

Azure Data Factory could benefit from improvements in its monitoring capabilities to provide a more robust feature set. Enhancing the ease of deployment to higher environments within Azure DevOps would be beneficial, as the current process often requires extensive scripting and pipeline development. It is also known for the flexibility of the data flow feature, particularly in supporting more dynamic data-driven architectures. These enhancements would contribute to a more seamless and efficient workflow within GitLab.

I have been using Azure Data Factory for the past 3 years. 

I find GitLab to be quite stable overall. I haven't encountered significant issues with its stability and consider it a reliable platform.

In terms of scalability, there are a few aspects of GitLab that I find disappointing. For instance, the limitation on self-hosted integration run time to just four VMs restricts scalability, especially for handling large volumes of data. Improvements are needed in this area to support more than four VMs for scalability. The documentation regarding bandwidth support is unclear, making it difficult to assess the full scalability potential. While GitLab performs well in cloud scalability in terms of compute power, the limitations on self-hosted integration run time are a concern for certain use cases. scalability in GitLab is highly dependent on specific use cases and could benefit from enhancements in self-hosted integration run time capabilities.

As for technical support from Microsoft the response times and solutions can sometimes be delayed. 

Neutral

Our organization did not switch from a previous solution to Azure Data Factory; rather, we implemented Azure Data Factory as a new solution to enable cloud data processing capabilities.

The deployment was handled in-house, and initially, we had a team of five working on the project in 2019. 

We have approximately six to seven data engineers from various departments utilizing the solution in our organization. As for the frequency of using GitLab, it varies depending on the workload and projects, but on average, there is someone working on the platform at least several times a week, as our data engineers are involved in various tasks beyond GitLab.

I am aware of the pricing of Azure Data Factory, but I prefer not to disclose specific details.

In terms of handling complex data transformations and cleansing, Azure Data Factory is capable for simple to medium tasks, but for more complex tasks, we resort to custom coding solutions. Overall, I would recommend Azure Data Factory for data integration and management, and I would rate it an eight out of ten for its flexibility and ability to support third-party integrations.

The current use is for extracting data from Google Analytics into Azure SQL Database as a source for our EDW.  Extracting from GA was problematic with SSIS. 

The larger use case is to assess the viability of the tool for larger use in our organization as a replacement for SSIS for our EDW and also as an orchestration agent to replace SQL Agent for firing SSIS packages using Azure SSIS-IR.

The initial rollout was to solve the immediate problem while assessing its ability to be used for other purposes within the organization. And also establish the development and administration pipeline process.  

ADF allowed us to extract Google Analytics data (via BigQuery) without purchasing an adapter.  

It has also helped with establishing how our team can operate within Azure using both PaaS and IaaS resources and how those can interact. Rolling out a small data factory has forced us to understand more about all of Azure and how ADF needs to rely upon and interact with other Azure resources.

It provides a learning ground for use of DevOps Git along with managing ARM templates as well as driving the need to establish best practices for CI.  

The most valuable aspect has been a large list of no-cost source and target adapters.

It is also providing a PaaS ELT solution that integrates with other Azure resources. 

Its graphical UI is very good and is even now improving significantly with the latest preview feature of displaying inner activities within other activities such as forEach and If conditions.   

Its built-in monitoring and ability to see each activity's JSON inputs/outputs provide an excellent audit trail.

The trigger scheduling options are decently robust.

The fact that it's continually evolving is hopeful that even if some feature is missing today, it may be soon resolved. For example, it lacked support for simple SQL activity until earlier this year, when that was resolved. They have now added a "debug until" option for all activities. The Copy Activity Upsert option did not perform well at all when I first started using the tool but now seems to have acceptable performance.  

The tool is designed to be metadata driven for large numbers of patterned ETL processes, similar to what BIML is commonly used for in SSIS but much simpler to use than BIML. BIML now supports generating ADF code although with ADF's capabilities I'm not sure BIML still holds its same value as it did for SSIS.

The list of issues and gaps in this tool is extensive, although as time goes on, it gets shorter. It currently includes:

1) Missing email/SMTP activity

2) Mapping data flows requires significant lag time to spin up spark clusters

3) Performance compared to SSIS. Expect copy activity to take ten times that of what SSIS takes for simple data flow between tables in the same database

4) It is missing the debug of a single activity. The workaround is setting a breakpoint on the task and doing a "rerun from activity" or setting debug on activity and running up to that point

5) OAuth 2.0 adapters lack automated support for refresh tokens

6) Copy activity errors provide no guidance as to which column is causing a failure

7) There's no built-in pipeline exit activity when encountering an error

8) Auto Resolve Integration runtime should never pick a region that you're not using (should be your default for your tenant)

9) IR (integration runtime) queue time lag. For example, a small table copy activity I just ran took 95 seconds of queuing and 12 seconds to actually copy the data. Often the queuing time greatly exceeds the actual runtime

10) Activity dependencies are always AND (OR not supported). This is a significant missing capability that forces unnecessary complex workarounds just to handle OR situations when they could just enhance the dependency to support OR like SSIS does. Did I just ask when ADF will be as good as SSIS?  

They need to fix bugs. For example:

1) The debug sometimes stops picking up saved changes for a period of time, rendering this essential tool useless during that time

2) Enable interactive authoring (a critical tool for development) often doesn't turn on when enabled without going into another part of the tool to enable it. Then, you have to wait several minutes before it's enabled which is time you're blocked from development until it's ready.  And then it only activates for up to 120 minutes before you have to go through this all over again. I think Microsoft is trying to torture developers

3) Exiting the inside of an activity that contains other activities always causes the screen to jump to the beginning of a pipeline requiring re-navigating where you were at (greatly slowing development productivity)

4) Auto Resolve Integration runtime (using default settings) often picks remote regions (not necessarily even paired regions!) to operate, which causes either an unnecessary slowdown or an error message saying it's unable to transfer the volume of data across regions

5) Copy activity often gets the error "mapping source is empty" for no apparent reason. If you play with the activity such as importing new metadata then it's happy again. This sort of thing makes you want to just change careers. Or tools. 

I have been using this product for six months.

Production operation seems to run reliably so far, however, the development environment seems very buggy where something works one day and not the next. 

So far, the performance of this solution is abysmal compared to SSIS. Especially with small tasks such as copying activity from one table to another within the same database. 

Customer support is non-existent. I logged multiple issues only to hear back from 1st level support weeks later asking questions and providing no help other than wasting my time. In one situation it was a bug where the debug function stopped working for a couple of days. By the time they got back to me, the problem went away. 

Negative

We have been and still rely on SSIS for our ETL. ADF seems to do ELT well but I would not consider it for use in ETL at this time.  Its mapping data flows are too slow (which is a large understatement) to be of practical use to us. Also, the ARM template situation is impractical for hundreds of pipelines like we would have if we converted all our SSIS packages into pipelines as a single ADF couldn't take on all our pipelines. 

Initial setup is the largest caveat for this tool. Once you've organized your Azure environment and set up DevOps pipelines, the rest is a breeze. But this is NOT a trivial step if you're the first one to establish the use of ADF at your organization or within your subscription(s). Instead of learning just an ETL tool, you have to get familiar with and establish best practices for the entire Azure and DevOps technologies. That's a lot to take on just to get some data movements operational. 

I did this in-house with the assistance of another team who uses DevOps with Azure for other purposes (non-ADF use). 

The setup cost is only the time it takes to organize Azure resources so you can operate effectively and figure out how to manage different environments (dev/test/sit/UAT/prod, etc.). Also, how to enable multiple developers to work on a single data factory without losing changes or conflicting with other changes.

We operate only with SSIS today, and it works very well for us. However, looking toward the future, we will need to eventually find a PaaS solution that will have longer sustainability.

I'm a customer. I'm using Azure Data Factory.

I find that the solution integrates well with cloud technologies, which we are using for different clouds like Snowflake and AWS. It is much more flexible in terms of transferring data from on-premise or on cloud. There is no need to create different mappings for different tables. The platform has the capability to handle metadata efficiently. So all our needs are being fulfilled with the platform we have right now.

I do not have any notes for improvement.

I have used it for almost six years.

The stability is quite good.

We wanted to move on to the cloud.

The initial setup is almost not difficult for technical people.

I used to work on Informative Support Center. Now we are using Azure Data Factory. 

I rate it eight out of ten. 

Our company uses the solution as a data pipeline. We get information outside the cloud from our factory such as data relating to production. We categorize it, clean it up, and transfer it to a database and data model. From there, we analyze the data using BI and other things. We gather information in data lake products like Microsoft Synapse and Microsoft Data Lake. 

We have two to three administrators who use the solution in a quite standard, mainstream way with nothing extreme. They handle administration, security, and development. 

It is difficult to define the total number of users because that depends on the number of data factory agents. We built the solution to have a different data factory agent for every customer. For example, if we have ten customers then we have ten users. We hope to increase usage but growth depends on our marketing efforts and how well we sell our products. 

The data factory agent is quite good and programming or defining the value of jobs, processes, and activities is easy. We have the agent installed on-premises in order to gather information.  

The cloud includes all kinds of API connections so we can easily gather information from other services. 

The solution seamlessly integrates with the Azure infrastructure. 

The pricing model should be more transparent and available online. When you start programming, you define the fields, variables, activities, and components but don't know the implication on price. You get a general idea but the more activities you add, the more you pay. It would be better to know price implications up front. 

There is a calculator you can run to simulate price but it doesn't help a lot. Practically speaking, you have to build your job and run it to see the exact price implications. This is an issue because you might realize you are paying too much so you have to reprogram or change things. 

I have been using the solution for three years. 

The solution is stable with no issues. Stability is rated a nine out of ten. 

We did have some breaches, but that was because we misconfigured something. Since we corrected it, we haven't had any issues. 

The solution is scalable with no performance issues. We haven't yet reached our limit that would require scaling. Scalability is rated an eight out of ten. 

We have discussions with our Microsoft partner all the time. 

In the last three years, we have contacted Microsoft directly three or four times. Once was for a general architectural issue and the rest were for the data factory or other items. Each time, we talked together with Microsoft and our partner. 

Support gave us answers and solved our issues. Sometimes, we didn't like the answer but we accepted that it was the correct answer. 

Support is rated a nine out of ten. 

Positive

We have not used another solution to this magnitude for real development and production. We work a little bit on Google Cloud. 

The initial setup was quite quick. Deployment was fairly simple and took less than a week. The setup got us up and running. 

After that, we had to write the implications of the data model and the kinds of activities. We are still doing this today because we make changes all the time.

The initial setup was seamless because we worked with a gold star Microsoft partner. Our side of setup was pretty quiet. We talked with our partner and told them what we needed from a security and monitoring point of view. We had a few high-level discussions from the block diagram perspective. Basically we said we need this or that, and our partner made it happen. 

The team included one person from our partner and three in-house team members with varying expertise across data modeling, security, and devops. We always worked with the same person but maybe behind the scenes he talked with coworkers. He did talk several times with Microsoft but we don't really know how many people were involved.

The solution does not require infrastructure maintenance. If we ever have issues, we can use Azure Defender to resolve them. We only make slight changes at the application level. 

We haven't calculated ROI on a formal level, but the fact is we need the solution. Because of the integration, we save a lot but haven't run exact numbers. 

The pricing model is based on usage and is not cheap. Based on our activity, we pay about $2,000 per month. 

Pricing is rated a four out of ten. 

If we didn't have the solution, we would have to find another tool because data pipelines are an essential part of our business. 

The biggest advantage to the solution is its integration with the Azure infrastructure that includes the active directory, security, Synapse, Data Lake, Power BI, and the data factory agent. 

All of the integration was a big consideration for us. We had general guidelines that said working with one vendor would provide the best integrations. The guideline was to use Microsoft unless there was an issue. 

We did not look at a third party or open source even though there are similar tools available. 

My best advice is to keep an eye on the pricing because we found out the hard way. Pricing is actually related to the way you use what the solution calls activity. This activity stuff drastically changes the coding to the rate you gather information from your client environment. 

So, when marketing guys tell you to gather information every minute, you have to weigh the heavy implications in comparison to collecting data once an hour or day. Programmers and developers designed the solution based on usage activity and building tasks or jobs. 

Pay a lot of attention to the pricing implications from the starting point of view. Technically, you can solve all issues but you need to keep an eye on the pricing. 

From a technical point of view, the solution is rated an eight out of ten. Because of pricing, the solution's overall rating is downgraded to a seven out of ten.