We changed our name from IT Central Station: Here's why
2019-03-28T02:50:00Z

What needs improvement with Azure Data Factory?


Please share with the community what you think needs improvement with Azure Data Factory.

What are its weaknesses? What would you like to see changed in a future version?

ITCS user
Guest
2626 Answers

author avatar
Top 20Real User

Data Factory is embedded in the new Synapse Analytics. The problem is if you're using the core Data Factory, you can't call a notebook within Synapse. It's possible to call Databricks from Data Factory, but not the Spark notebook and I don't understand the reason for that restriction. To my mind, the solution needs to be more connectable to its own services. There is a list of features I'd like to see in the next release, most of them related to oversight and security. AWS has a lake builder, which basically enforces the whole oversight concept from the start of your pipeline but unfortunately Microsoft hasn't yet implemented a similar feature.

2021-11-05T19:42:49Z
author avatar
Real User

The only thing I wish it had was real-time replication when replicating data over, rather than just allowing you to drop all the data and replace it. It would be beneficial if you could replicate it. Real-time replication is required, and this is not a simple task.

2021-11-04T19:59:04Z
author avatar
Top 10LeaderboardReal User

We didn't have a very good experience. The first steps were very easy but it turned out that we used Europe for a Microsoft data center, also partly abroad for our alpha notes. As soon as we started using Azure Data Factory, the bills got higher and higher. At first we couldn't understand why, but it is very expensive to put data into a data center abroad. So instead, we decided to use only Northern Europe, which worked out for a while in the beginning. And then we had nothing to show for it. They gave me a really hard time for this. Azure Data Factory should be cheaper to move data to a data center abroad for calamities in case of disasters. What I really miss is the integration of Microsoft TED quality services and Microsoft Data services. If they were to combine those features in Data Factory, I think they would have a very strong proposition. They promise something like that on Microsoft Congress. That was years ago and it's still not here.

2021-08-31T13:03:00Z
author avatar
Top 5LeaderboardReal User

There is always room to improve. There should be good examples of use that, of course, customers aren't always willing to share. It is Catch-22. It would help the user base if everybody had really good examples of deployments that worked, but when you ask people to put out their good deployments, which also includes me, you usually got, "No, I'm not going to do that." They don't have enough good examples. Microsoft probably just needs to pay one of their partners to build 20 or 30 examples of functional Data Factories and then share them as a user base.

2021-08-11T18:32:33Z
author avatar
Top 20Real User

Snowflake connectivity was recently added and if the vendor provided some videos on how to create data then that would be helpful. I think that everything is there, but we need more tutorials.

2021-05-17T14:02:46Z
author avatar
Top 20Real User

It would be better if it had machine learning capabilities. For example, at the moment, we're working with Databricks and Azure Data Factory. But Databricks is very complex to do the different data flows. It could be great to have more functionalities to do that in Azure Data Factory.

2021-04-17T15:16:14Z
author avatar
Top 20Real User

I'm more of a general manager. I don't have any insights in terms of missing features or items of that nature. Integration of data lineage would be a nice feature in terms of DevOps integration. It would make implementation for a company much easier. I'm not sure if that's already available or not. However, that would be a great feature to add if it isn't already there.

2021-03-10T08:56:59Z
author avatar
Top 20Real User

My only problem is the seamless connectivity with various other databases, for example, SAP. Our transaction data there, all the maintenance data, is maintained in SAP. That seamless connectivity is not there. Basically, it could have some specific APIs that allow it to connect to the traditional ERP systems. That'll make it more powerful. With Oracle, it's pretty good at this already. However, when it comes to SAP, SAP has its native applications, which are the way it is written. It's very much AWS with SAP Cloud, so when it comes to Azure, it's difficult to fetch data from SAP. The initial setup is a bit complex. It's likely a company may need to enlist assistance. Technical support is lacking in terms of responsiveness.

2021-02-14T15:56:02Z
author avatar
Top 5LeaderboardReal User

The need to work more on developing out-of-the-box connectors for other products like Oracle, AWS, and others.

2020-12-18T22:33:34Z
author avatar
Top 20LeaderboardReal User

We are too early into the entire cycle for us to really comment on what problems we face. We're mostly using it for transformations, like ETL tasks. I think we are comfortable with the facts or the facts setting. But for other parts, it is too early to comment on. We are still in the development phase, testing it on a very small set of data, maybe then the neatest four or bigger set of data. Then, you might get some pain points once we put it in place and run it. That's when it will be more effective for me to answer that.

2020-12-09T10:31:00Z
author avatar
Top 20LeaderboardReal User

Understanding the pricing model for Data Factory is quite complex. It needs to be simplified, and easier to understand. We have experienced some issues with the integration. This is an area that needs improvement.

2020-10-27T00:17:44Z
author avatar
Top 20Real User

The number of standard adaptors could be extended further. What we find is that if we develop data integration solutions with Data Factory, there's still quite a bit of coding involved, whereas we'd like to move in a direction with less coding and more select-and-click.

2020-10-21T11:46:00Z
author avatar
Real User

I find that Azure Data Factory is still maturing, so there are issues. For example, there are many features missing that you can find in other products. You cannot use a custom data delimiter, which means that you have problems receiving data in certain formats. For example, there are problems dealing with data that is comma-delimited.

2020-09-21T06:33:00Z
author avatar
Top 20LeaderboardReal User

The pricing scheme is very complex and difficult to understand. Analyzing it upfront is impossible, so we just decided to start using it and figure out the costs on a weekly or monthly basis.

2020-08-19T07:57:30Z
author avatar
Top 20Real User

Azure Data Factory is a bit complicated compared to Informatica. There are a lot of connectors that are missing and there are a lot of instances where I need to create a server and install Integration Runtime. The support and the documentation can be improved. There are a lot of tasks that you need to write code for.

2020-07-13T06:55:56Z
author avatar
Top 20Real User

I'm not sure if I have any complaints about the solution at the moment. There are a few bits and pieces that we would like to see improved. These include improvements related to the solution's ease of use and some quality flash upgrades. However, these are minor complaints. If the user interface was more user friendly and there was better error feedback, it would be helpful.

2020-06-15T07:33:55Z
author avatar
Real User

The setup and configuration process could be simplified.

2020-01-12T12:03:00Z
author avatar
Top 20LeaderboardReal User

The user interface could use improvement. It's not a major issue but it's something that can be improved. It has the ability to create separate folders to organize objects, Data Factory objects. But any time that we created a folder we were not able to create objects. We had to drag and drop into the folder. There were no default options. It was manual work. We offered their team our feedback and they accepted my request.

2019-12-31T09:39:00Z
author avatar
Real User

The only thing that we're struggling with is increasing the competency of my team. So we think that the Microsoft documentation is too complicated. I would like to see it more connected. I know they're working on the Snowflake data warehouse connector, but more connectors would be helpful.

2019-12-23T07:05:00Z
author avatar
Real User

Because I have not really done a really deep benchmark against competitors, I may not be familiar enough with the potential of competing products and capabilities to be able able to say what is missing or should be improved definitively. From my perspective, the pricing seems like it could be more user-friendly. Of course, nothing is ever as inexpensive as you want. Perhaps one good additional feature would be incorporating more ways to import and export data. It would be nice to have the product fit our service orchestration platform better to make the transfer more fluid.

2019-12-16T08:14:00Z
author avatar
Real User

The speed and performance need to be improved. This solution should be able to connect with custom APIs.

2019-12-12T07:48:00Z
author avatar
LeaderboardReal User

At this point in time, they should work on somehow integrating the big data capabilities within it. I have not explored it, but it would be good if somehow we could call a Spark job or something to do with the Spark SQL within ADS so that we wouldn't need a Spark tested outside. On the UI side, they could make it a little more intuitive in terms of how to add the radius components. Somebody who has been working with tools like Informatica or DataStage gets very used to how the UI looks and feels. In ADS, adding a new table or joining a new table and overriding that with an override SQL that I could customize would be helpful. Being able to debug from the design mode itself would be helpful.

2019-12-09T10:58:00Z
author avatar
Consultant

It would be helpful if they could adjust the data capture feature so that when there are source-side changes ADF could automatically figure it out. The solution needs to integrate more with other providers and should have a closer integration with Oracle BI.

2019-12-05T11:14:00Z
author avatar
MSP

The solution could use some merge statements. The solution should offer better integration with Azure machine learning. We should be able to embed the cognitive services from Microsoft, for example as a web API. It should allow us to embed Azure machine learning in a more user-friendly way.

2019-11-18T07:22:00Z
author avatar
Consultant

I think more integration with existing Azure platform services would be extremely beneficial. In the next release, it's important that some sort of scheduler for running tasks is added. A built-in scheduling mechanism for running the treasury will be a very helpful improvement.

2019-07-29T10:11:00Z
author avatar
Consultant

Data Flow is in the early stages — currently public preview — and it is growing into a tool that will offer everything other ETL tools offer. There are a few features still to come. The thing we missed most was data update, but this is now available as of two weeks ago. A feature that is confirmed as coming soon is the ability to pass in a parameter and filter, etc.

2019-03-28T02:50:00Z
Learn what your peers think about Azure Data Factory. Get advice and tips from experienced pros sharing their opinions. Updated: January 2022.
566,121 professionals have used our research since 2012.