Azure Data Factory Reviews

I am a solution architect and work with the presales team. I generally provide solutions, study Azure Data Factory, and conduct POCs. I use Azure Data Factory as an ETL tool. I have been using it for about three years.

Azure Data Factory is a low code, no code platform, which is helpful. It provides many prebuilt functionalities that assist in building data pipelines. 

Also, it facilitates easy transformation with all required functionalities for analytics. Furthermore, it connects to different sources out-of-the-box, making integration much easier. The monitoring is very thorough, though a more readable version would be appreciable.

We use Azure Data Factory to perform transformation operations after moving away from a legacy ETL platform. The platform enables us to handle a large number of ETL jobs and integrate with other Azure services for improved automation and scalability. It supports advanced parsing of structured data formats like XML and JSON and integrates well with our reporting and analytics systems.

Data flow operations are intuitive, particularly for applying transformations and using expressions.

The platform makes it easy to preview and manipulate data during development.

Direct support for XML and JSON datasets improves processing efficiency.

Integration with Azure services supports our cloud-first approach.

We are a partner and operate as a reseller. Our main area is as a service provider, working with businesses in building their data pipelines, data engineering, and data analytics solutions. We often work with clients on their infrastructure, with Data Factory being a key component more commonly in these situations.

The interface of Azure Data Factory is very usable with a more interactive visual experience, making it easier for people who are not as experienced in coding to work with. The scheduling management of pipelines and how the environment is set up allows for effective troubleshooting and identifying issues quickly. Alerts and warnings make it easy to diagnose pipeline errors.

I find the most valuable feature in Azure Data Factory to be its ability to handle large datasets. The data is more scalable.

There is a problem with the integration with third-party solutions, particularly with SAP.

We use Azure Data Factory to build data analytics products.

Azure Data Factory helps in data integration and data orchestration in a self-service way, and it is a native component to the Azure platform.

My main use case for Azure Data Factory is to pull data from on-premises systems. Most data transformation is done through Databricks, but Data Factory mainly pulls data into different services.

The solution handles large volumes of data very well. One of its best features is its ability to integrate data end-to-end, from pulling data from the source to accessing Databricks. This makes it quite useful for our needs.

Azure Data Factory is primarily used to orchestrate workflows and move data between various sources. It supports both ETL and ELT  processes. For instance, if you have an ERP system and want to make the data available for reporting in a data lake or data warehouse, you can use Data Factory to extract data from the ERP system as well as from other sources, like CRM systems.

Data Factory allows you to pull data from multiple systems, transform it according to your business needs, and load it into a data warehouse or data lake. It also supports complex data transformations and aggregations, enabling you to generate summary and aggregate reports from the combined data. Data Factory helps you ingest data from diverse sources, perform necessary transformations, and prepare it for reporting and analysis.

I have extensive experience building things independently, with over twenty years of experience in SQL, ETL, and data-related projects. Recently, I have been using Azure Data Factory for the past two years. It has proven to be quite effective in handling large volumes of data and performing complex calculations. It allows for the creation of intricate data workflows and processes faster. Azure Data Factory is particularly useful for enterprise-level data integration activities, where you might deal with millions of records, such as in SAP environments. For example, SAP tables can contain tens or hundreds of millions of records. Managing and maintaining the quality of this data can be challenging, but Azure Data Factory simplifies these tasks significantly.

We mainly use it to migrate data from on-premises sources to the cloud, such as Oracle and Cisco servers. It's a good solution for integrations within the Azure environment, and it connects well with other Azure data products. However, for external configurations, we use Informatica Cloud or Informatica Data Accelerator (IDA).

For automation, we primarily rely on Snowflake and Informatica. Our strategy is not to depend on a single tool. When it's strictly on-premises to cloud, we use ADF. 

Otherwise, Informatica is more mature and integrates well with various third-party products. We also use Snowflake copy commands to load data into Snowflake. Azure Data Factory doesn't fully meet your automation requirements.

We use Informatica for pipelines that were originally in SSIS, but for new pipelines and ETL processes, we choose either Informatica or Snowflake scripts.

We have multiple banking applications running on SSIS pipelines. We're in the process of upgrading them to a hybrid cloud architecture. For that, we use Azure Data Factory to move data from on-premises to the cloud – mainly for back-end database operations and ETL transformations.

We primarily use it to load data from an on-premises SQL Server to either Blob storage or an Azure SQL data warehouse. For other integrations, especially those outside of Azure, we tend to use Informatica Cloud Services (ICS).

For structured data loading, we use it However, we use Informatica for unstructured or semi-structured data. We also use Snowflake for ETL processes and sometimes for streaming. 

In my opinion, ADF isn't as suitable for streaming – for streaming, Snowflake streamlets or Informatica structured streaming are more reliable. ADF works well for batch processing, though.