Azure Data Factory Reviews

I am a solution architect and work with the presales team. I generally provide solutions, study Azure Data Factory, and conduct POCs. I use Azure Data Factory as an ETL tool. I have been using it for about three years.

Azure Data Factory is a low code, no code platform, which is helpful. It provides many prebuilt functionalities that assist in building data pipelines. 

Also, it facilitates easy transformation with all required functionalities for analytics. Furthermore, it connects to different sources out-of-the-box, making integration much easier. The monitoring is very thorough, though a more readable version would be appreciable.

We use Azure Data Factory to perform transformation operations after moving away from a legacy ETL platform. The platform enables us to handle a large number of ETL jobs and integrate with other Azure services for improved automation and scalability. It supports advanced parsing of structured data formats like XML and JSON and integrates well with our reporting and analytics systems.

Data flow operations are intuitive, particularly for applying transformations and using expressions.

The platform makes it easy to preview and manipulate data during development.

Direct support for XML and JSON datasets improves processing efficiency.

Integration with Azure services supports our cloud-first approach.

Our company uses the solution as a data pipeline. We get information outside the cloud from our factory such as data relating to production. We categorize it, clean it up, and transfer it to a database and data model. From there, we analyze the data using BI and other things. We gather information in data lake products like Microsoft Synapse and Microsoft Data Lake. 

We have two to three administrators who use the solution in a quite standard, mainstream way with nothing extreme. They handle administration, security, and development. 

It is difficult to define the total number of users because that depends on the number of data factory agents. We built the solution to have a different data factory agent for every customer. For example, if we have ten customers then we have ten users. We hope to increase usage but growth depends on our marketing efforts and how well we sell our products. 

The data factory agent is quite good and programming or defining the value of jobs, processes, and activities is easy. We have the agent installed on-premises in order to gather information.  

The cloud includes all kinds of API connections so we can easily gather information from other services. 

The solution seamlessly integrates with the Azure infrastructure. 

The current use is for extracting data from Google Analytics into Azure SQL Database as a source for our EDW.  Extracting from GA was problematic with SSIS. 

The larger use case is to assess the viability of the tool for larger use in our organization as a replacement for SSIS for our EDW and also as an orchestration agent to replace SQL Agent for firing SSIS packages using Azure SSIS-IR.

The initial rollout was to solve the immediate problem while assessing its ability to be used for other purposes within the organization. And also establish the development and administration pipeline process.  

ADF allowed us to extract Google Analytics data (via BigQuery) without purchasing an adapter.  

It has also helped with establishing how our team can operate within Azure using both PaaS and IaaS resources and how those can interact. Rolling out a small data factory has forced us to understand more about all of Azure and how ADF needs to rely upon and interact with other Azure resources.

It provides a learning ground for use of DevOps Git along with managing ARM templates as well as driving the need to establish best practices for CI.  

We are a partner and operate as a reseller. Our main area is as a service provider, working with businesses in building their data pipelines, data engineering, and data analytics solutions. We often work with clients on their infrastructure, with Data Factory being a key component more commonly in these situations.

The interface of Azure Data Factory is very usable with a more interactive visual experience, making it easier for people who are not as experienced in coding to work with. The scheduling management of pipelines and how the environment is set up allows for effective troubleshooting and identifying issues quickly. Alerts and warnings make it easy to diagnose pipeline errors.

I find the most valuable feature in Azure Data Factory to be its ability to handle large datasets. The data is more scalable.

There is a problem with the integration with third-party solutions, particularly with SAP.

We mainly use it to migrate data from on-premises sources to the cloud, such as Oracle and Cisco servers. It's a good solution for integrations within the Azure environment, and it connects well with other Azure data products. However, for external configurations, we use Informatica Cloud or Informatica Data Accelerator (IDA).

For automation, we primarily rely on Snowflake and Informatica. Our strategy is not to depend on a single tool. When it's strictly on-premises to cloud, we use ADF. 

Otherwise, Informatica is more mature and integrates well with various third-party products. We also use Snowflake copy commands to load data into Snowflake. Azure Data Factory doesn't fully meet your automation requirements.

We use Informatica for pipelines that were originally in SSIS, but for new pipelines and ETL processes, we choose either Informatica or Snowflake scripts.

We have multiple banking applications running on SSIS pipelines. We're in the process of upgrading them to a hybrid cloud architecture. For that, we use Azure Data Factory to move data from on-premises to the cloud – mainly for back-end database operations and ETL transformations.

We primarily use it to load data from an on-premises SQL Server to either Blob storage or an Azure SQL data warehouse. For other integrations, especially those outside of Azure, we tend to use Informatica Cloud Services (ICS).

For structured data loading, we use it However, we use Informatica for unstructured or semi-structured data. We also use Snowflake for ETL processes and sometimes for streaming. 

In my opinion, ADF isn't as suitable for streaming – for streaming, Snowflake streamlets or Informatica structured streaming are more reliable. ADF works well for batch processing, though.

The primary use case is integrating data from different ERP systems and loading it into Azure Synapse for reporting. We use Power BI for the reporting side of it.

We also have customers who are migrating to Azure Data Factory and we are assisting them with making the transition.

It is easy to integrate.

I do not foresee any issues with security.