Azure Data Lake Storage Reviews

For Amazon's S3, security is enabled from the AWS side itself. We do not use antivirus separately, as it is enabled by AWS automatically.

Integration with Azure Synapse Analytics or Azure Databricks helps our data workflows because we generally mount the volumes to the Databricks environment. The raw data is stored in Azure Data Lake, and the final layer data is mostly stored in Azure Databricks Delta files. Azure Data Lake is only for the raw data for us.

I would rate Azure Data Lake Storage around a nine out of ten.

To give Azure Data Lake Storage a ten out of ten, they need to optimize the cost. The cost factor is the only thing I would identify for all my use cases, as they are very well equipped in other areas. Even though the cost is reasonable, if they reduce it, we would benefit significantly. Additionally, there is no data lineage and no ontology tools available on Azure Data Lake Storage. I understand it is blob-based, but if they maintain some ontology tools or data cataloging and data lineage, that would be great. However, those features are currently covered under Databricks, so we are not much concerned.

We have been working with Azure Data Lake Storage for five years.

I am satisfied with the performance, scalability, and interface.

For performance and scalability, we have never faced any issues even though we handle at least a TB of data every day without any performance problems.

We have dedicated corporate support from Microsoft. Generally, it is within the agreed SLA. We have observed one or two exceptional scenarios where they cannot address the issue, but other than that, support is always within SLA.

We moved away from Palantir products and are now entirely focused on Azure and AWS.

For the initial setup of Azure Data Lake Storage, it is straightforward when using the public cloud. However, when using the private cloud, which we are using, it would take some lead time because they need to set up VPC and everything. Since we are mostly using the private cloud, it would be somewhat more difficult compared to using the public cloud.

Regarding pricing, I find it reasonable. We did not observe much price difference between AWS and Azure. Both are the cheapest compared to Palantir and everything else.

We maintain a two-platform strategy, so depending on where the applications are hosted, we will maintain our infrastructure accordingly. We would prefer to be on two data platforms rather than just one to avoid any cost risk or monopoly risk. Price-wise, when compared with the on-site costs we were having, the current pricing is reasonable. However, with increasing prices over the years, we need to assess whether on-premises becomes cheaper, in which case we would need to migrate from cloud to on-premises. We are even investigating this option. For some applications, on-premises would be sufficient and we would not want to be on the cloud. We make this decision periodically, but as of now, I do not see much cost difference between Azure and AWS. We incur the same cost whether we run the applications on Azure or AWS.

We are mostly working with S3, RDS, and AWS Glue jobs.

We are using Amazon S3.

For Microsoft products, we are using Databricks, Data Factory, Azure Data Lake services, Power BI, and Power Automate.

For Azure Data Lake Analytics, we use data lake storage but do not use analytics, as it is external for us.

We use Azure Data Lake Storage and Databricks.

We do not use the hierarchical namespace feature because we have separate namespaces for each division and separate namespaces for each project.

For assessing the effectiveness of Azure Data Lake Storage encryption and RBAC for maintaining data security, we do not do anything because we are on a private network and Azure is also privately hosted for us, so it is taken care of by the networking team.

We use HDFS compatibility in Azure Data Lake Storage.

We are still working on managing data silos through Azure Data Lake Storage and are working on making it a unified data lake storage. Currently, different projects use different data lake storage, so we still operate some projects in silos. We are building data products to overcome this by having each product and project own a certain section of data that will be shared with all other projects. This way we do not maintain duplicate data. We are still working on the data mesh concept and everything, so it is not yet complete. We are midway through.

All our requirements are currently met by Azure Data Lake Storage. We have been using it only for the data lake and are not much concerned about advanced features. Most of the features we use from the data lake to track the lineage and everything are mostly covered from the Databricks side.

Integration with third-party solutions is always compatible with whatever is on the approved list of third-party solutions on the Azure platform. If we are doing something custom, we have other options to build it. We can build a Linux machine and do it. I do not face any difficulty with integration. It will be time-consuming if it is not approved by Azure or AWS; otherwise, it is fine.

I give this review an overall rating of nine out of ten.

My main use cases for Azure Data Lake Storage involve building these solutions, designing these solutions, and evaluating this solution. Once this solution is paid, HRD, LDR in place, I go on the entire program. I lead a team with data engineering skills who are doing the ETL part of it, as well as people who are doing data transformation and those working from a data warehouse perspective. This involves the creation of tables, key miles, making the models ready, and giving them for consumption by tools such as Power BI or some other reporting tools to use these data models and create the reporting or the insight on top of that. This is my role.

Azure Data Lake Storage is purely used to store data, and from this data lake, we can segregate in simple language. We can create multiple orders based on different domains or different business units. In that one folder, we can create multiple subfolders, which can provide access to specific people, not all due to some security reasons. That type of structure can be built on Azure Data Lake Storage.

Azure Data Lake Storage is very much scalable because you can increase the capacity of this data lake on the fly.

It definitely supports my needs performance-wise. The only challenge depends on what type of storage we are using for the data lake. Is it a hot tier or archive tier? Performance will depend on that. If our data is getting into a harder mode, and if we need to hit you, the performance will be low. But the same data, if it is in a hot tier, the performance will be on the higher side.

Azure Data Lake Storage does have a difference in price depending on the data type. There is laboratory storage or the disk will be at a lower cost, but the hot tier will be at some higher cost, and those differences will be there.

I have not seen anything that could be improved in the stability of Azure Data Lake Storage. There have been some small issues, but those are purely from the configuration part of it, not really from the Azure Data Lake Storage perspective.

I have contacted Microsoft technical support at points. When we raise a ticket, definitely the first question is to send the logs. We need to upload the entire logs for them, and then they generally do the analysis. If they need more information, they will give multiple commands to deep dive or get deep logs. Again, we need to upload. They provide what we need to try out either in development before we make some changes to production. Based on prior experiences, I would say their support depends on the customer support level. If you are a premium customer, then definitely the support is really very good, and they can come onsite to tackle critical P1 issues and work with us.

My reason for deciding to work with Azure Data Lake Storage relates to my career path. If you see my experience, it is around eighteen years. In the earlier days, when I started my career, it was totally a physical environment, and I had worked on all IBM technologies. Slowly, I moved to VMware where all virtualization started, and then cloud started booming. That is how that shift has been happening.

Azure Data Lake Storage is very less priced because the difference when I compare on-premise to cloud is significant. On-premise, you need to have from day one the entire capacity, and you own that cost. But in Azure Data Lake Storage or any cloud, either it is AWS or Azure, it is a pay as you go, or you can set a fixed capacity for the next three years, which enables around thirty-five to forty percent discount from Microsoft, depending on how customers prefer their subscription. Just to give an example, if I am using one terabyte of space, I would estimate approximately not more than a hundred USD monthly. This is just a guest image I am sharing.

From a private cloud perspective, I have not worked with some similar tools to Azure Data Lake Storage. I had worked on IBM Storage, which includes systems such as DS800. Those were physical storages, not from the virtual side.

From a data strategy perspective, I am working with Azure Data Lake Storage.

It is Synapse SQL, not Snowflake. But maybe I worked with something related to Azure.

Majorly, I am working on Azure technologies. The parts and services for Azure Data Lake Storage include Data Factory, Data Lake, Power BI, Synapse, analytics, and so on.

Azure Data Lake Storage, if you ask about integration with Azure technology, it is very smooth because the in-house adapters or the connectors are available. If we need to connect outside this, we need to use some integration layer, such as Data Factory. If I need to bring the data from any other outside source to this particular data lake, then I need to create the pipeline to onboard data. But if I need to onboard it from something such as Dynamics 365, where it is running on data that needs to be taken into a lake, that is easy. I would say it will be easy because they have their own solutions, such as Microsoft Logic App and Azure Gateway Function App and so on. They have those integrations in place for Azure Data Lake Storage.

Now if you see Azure Data Lake Storage Gen two, which is the latest feature, the new things coming on the one lake are better. Your silo data is getting or customers who were studying from silo data are now with this feature coming out of that. You just need to integrate with this one lake, and you will get the entire data. So, ADL is Gen two, and now it is one lake with this new fabric feature coming from Microsoft.

I rate this review an overall eight out of ten.

I use Azure Data Lake Storage extensively. I've used it in the past, but I haven't used it in a couple of years.

We use it for analytics; we store our Databricks, Data Lake, and all that stuff there.

I've used the hierarchical namespace feature; that's the biggest difference.

We don't use Synapse, but with Databricks, it is really paired as where all of our Databricks data comes from and where it all goes to, because Databricks essentially gives us a big query interface for applications for our analysts, and that reads data from that Data Lake and writes it back in various other forms.

We don't use the HDFS compatibility in Azure Data Lake Storage.

The encryption's all built-in, so that's to say it's really easy; you just flip it on, and I believe it's on by default, so the encryption isn't really much to worry about.

We do that sometimes with different classes of data that different people need access to; you can have more granular access controls on the containers to give people access to things at an infrastructural level. There's more finer-grained control once you get into Databricks, such as row-level security and that sort of thing.

I would say the best features include that it's easy to integrate; it's certainly easy to integrate with the other things in the Microsoft portfolio.

The benefit of it integrating within the Microsoft portfolio is that it's one of the lowest level services, so it works pretty seamlessly, similar to the blob storage; it's essentially the blob storage but with more features that are analytics focused because usually that's what people are going to do with the Data Lake, which is ingest data for analytics.

It's essential; you can't really do analytics without that feature; if you did, it would be infernally slow.

It depends on the tool when it's outside of the Microsoft portfolio as to whether it integrates; it really depends on what you're doing with the other tool.

The cost is something that everyone is cost-sensitive about. There was a curious feature that I was interested in; I think it was something in preview, a blob scanning ability, the ability to have a hardware-level querying of the data there. I don't think it worked with Parquet data; I think it mainly worked with plain text data. That's something that I would want to have added, is the ability to scan and query data with Parquet, because currently, that's what you have to use Databricks to do.

They make it easy to set up, and their licensing for Azure Data Lake Storage is just pay-as-you-go, so you can get into it quite easily. The devil's in the details, because the cost as you use it can be difficult to forecast because it really depends upon all the other services and how heavily it's being used. There are other features that can catch you, such as a storage security scanner that automatically scans your storage for malware. Compliance departments and IT departments love that type of feature, so people will blanket enable that, but then that racks up tons of bills because it's reading massive files constantly that should be excluded from that type of scan because it's not a type of file capable of having malware; it's the backend database.

There are some gotchas with the pricing; you have to be careful. There's not really a way to throttle it either. The premium tier, which they recommend for analytics, can't be switched off once enabled. Similarly, if you start on standard tier, you can never upgrade it; you have to delete it and make a new one. These limitations are quite annoying.

I've had experience with Azure Data Lake Storage for several years.

Azure Data Lake Storage is stable for the most part.

As far as performance-related scalability, for our needs, it's certainly sufficient. If you have a lot of cross-region needs, then you might have to do more work because it doesn't natively store data in various regions; you have to pick a region. You can't automatically scale across the world, so for something that requires global scaling, you may have to do more work to get the scalability, but we don't have that need personally.

We have a dedicated cloud solution architect, and she's great; ever since we got her, it's been night and day, really. The main support's okay; we've been using Azure for over 10 years, so we don't usually have a lot of 'how does this work' type questions; we are pretty familiar with it.

I would rate the service probably an eight because there are certainly things that she doesn't know. You can't expect her to know everything about Azure, but she knows who does know, so things can get handled by who knows about the topic the best, and that's usually the best way to handle anything anyway.

Positive

I deal with more of the Azure umbrella things, such as CRMs, networking solutions, or backup solutions with Microsoft as well.

We're cloud only, so we don't use hyper-converged infrastructure.

It's easy; it's a straightforward setup and takes about five minutes.

More the Microsoft one; I couldn't get it to even give me the report.

I don't deal with DDoS protection or Network Watcher.

Is that the on-prem version? With Azure, I just deal with Azure Stack.

It's not an AWS product, so it wasn't purchased through the AWS marketplace.

The overall rating for Azure Data Lake Storage is 9 out of 10.

Data Lake Storage is primarily used to ingest different types of data such as file share data, raw data, or unstructured data. 

Additionally, the Data Lake Storage account is Gen Two and is integrated with various Azure resources such as Azure Cloud, Databricks, Data Factory, and SQL DB. Furthermore, it is also compatible with other solutions such as Cosmos DB, Stream Analytics, and Event Hubs.

Data Lake Storage provides valuable features such as access keys, container strings, lifecycle management, audit logs, and the soft delete option. These features help in cost-saving and provide an excellent mechanism for backing up data. 

Furthermore, Data Lake Storage can interact with any other Azure resources, providing seamless integration and connectivity.

Improvement is needed in the migration process from Lakehouse to Enterprise Data Lake (EDL). Currently, migration is only one-way possible, and it would be beneficial if this aspect could be improved. Additionally, increasing the retention period beyond the current seven days would be helpful.

I have been working with Data Lake Storage for around five years.

Stability is rated around eight to nine out of ten. It has very good reliability and dependability.

For scalability, I rate Data Lake Storage around eight to nine on a scale of one to ten.

The technical support from Microsoft is pretty good. The support team is very supportive, and most issues are resolved quickly once we get in touch with them.

Positive

The initial setup process is simple. There are different deployment methods, including deploying manually via the portal or using Terraform modules.

The pricing for Data Lake Storage depends on several factors, like the configuration for multiple or single locations and if it uses geo-redundancy storage, which is beneficial but consumes higher costs. Enabling lifecycle management and using cost-saving features can keep the expenses in check.

There are several competitors to Data Lake Storage, such as AWS, Google Cloud Storage, IBM Object Storage, and Snowflake.

It is essential to enable lifecycle management in Data Lake Storage for better cost management. Proper setup according to your use case and employing strategies like using cold or archive tiers can save cost and optimize usage.

I'd rate the solution nine out of ten.

For our current use case, we develop a solution to do PDF extraction. We store PDF files in Azure Data Lake Storage. This helps our application in AI solutions, where users can query something and get results, and we can cross-check by showing the PDF files stored in the Data Lake Storage.

One of the first selling points of the storage is the cost; it is reasonably cheap, with different options for storing data. We usually store data in a hot tier, but for archival purposes, cold storage is an affordable option. However, cold tier data take time to make available, requiring consideration between hot and cold options. Their storage is integrated with Azure services like Databricks, making it convenient for various solutions. Furthermore, by migrating from on-premises to the cloud, it reduces costs significantly, which is good for file storage solutions. Additionally, it improves efficiency and shares permissions and authorizations with its container capabilities.

In AWS, there is a feature called vector database, which could improve its versatility. Currently, Azure Data Lake Storage has blob, table, and file share, but no vector feature. With the emergence of AI technology, it would be convenient for storing vector indexes, essential for AI solutions.

I have had experience with the product for three years.

The solution is stable, with a stability rating of nine out of ten.

Scalability is good, and the system is versatile in accommodating whatever is needed to be stored.

I don't usually ask for support because the solution is stable, so I'm not aware of the support quality.

Positive

The initial setup is pretty straightforward and easy to understand. It is rated eight point five out of ten for ease.

In an enterprise environment, the infrastructure team typically handles the setup, but it can be done independently in a sandbox.

The pricing is reasonable. Azure Data Lake Storage is cheaper and provides three options for data storage tiers. However, I'm not familiar with the pricing of other solutions like AWS S3, as I haven't used them.

I have not used any other data storage solutions outside of Azure in my company.

I would recommend Azure Data Lake Storage because it is straightforward to set up and versatile for various projects, especially AI solutions requiring document storage. I rate the product around nine out of ten.

From my perspective, it is secure transfer storage out there.

In terms of the use of Azure Data Lake Storage by customers for data analytics and processing workflows, I would say that my role is to convince customers that it is the safest tool for the storage of data. You can securely connect to remote regions with the tool Azure Storage Explorer, which gives the options and possibilities to safely transfer your data from your existing storage premises and send it to Azure Cloud.

The solution's most valuable features are the tools and functions, which are primarily hosted in Azure Storage Explorer. However, you can also facilitate them from within the backup. The tool is very safe to use. It is impossible to hack the product.

Some customers residing in former Eastern European countries operate in an independent and very weak IT environment. If tools like Azure Data Lake Storage are enabled within the tool named Azure Storage Explorer, then it would be of tremendous help, but it can be really tricky. It would be great if some of the aforementioned features could be enabled, but I fully understand the complexities involved.

I used to like Azure Data Lake Storage previously. Presently, I like the fact Azure Data Lake Storage is improving rapidly by investing and honestly in assets, resources, top personnel, along with a lot of money for making Azure's storage part a bigger concept. Azure Data Lake Storage can be a danger for the large storage products.

I have been using Azure Data Lake Storage for a couple of years. I am an Azure solution architect. I work with Azure Data Lake Storage Gen2.

I am very confident that it is a stable solution. Stability-wise, I rate the solution a ten out of ten.

There was a major issue during mid-October, which affected many global businesses just for a few hours. It was the biggest issue with the tool I had been involved in for many years.

Scalability-wise, I rate the solution a ten out of ten.

Azure Lake Dade Data Lake Storage scalability has very much impacted our customer's data storage strategy since it offers options to choose the disk, scale-out, and DRC options, making everything fantastic.

All customers I have worked with over the last year are using the tool and assigning me to look after it, so it could be seven or eight businesses over the last three years, some of which are global leaders in the market, having over 12,000 employees globally.

On a scale of one to ten, where one is difficult and ten is easy, I rate the product's initial setup phase as ten. You have to understand what to do since you can be lucky and just go and click a few buttons to do the setup process. Knowing the tool's setup phase can make the product cost-effective, but if you don't know about it, then it can be costly. Combining the tool with the features of Azure Cost Management can make things much easier for you. The upcoming edition of Microsoft Copilot should make everything in the tool way easier and also other things not so expensive.

I was not directly involved in the product's deployment process, but I am subscribed to all channels associated with the deployment part, and I have many friends in Microsoft in Northern Europe, and in Sweden, where I live. Storage is one of my top skills, and my friends want to help me become a champion.

The solution is deployed on the cloud and in the on-premises version.

From one to ten, where one is cheap and ten is expensive, I rate the product price as five. It costs money, but it is cheaper by at least thirty percent if you consider the other equivalent solutions from AWS. Considering the aforementioned perspective, the tool is cheap, but you have to pay a certain amount.

There are options to choose from depending on the subscription you have and the amount of features you consume from Microsoft, so it can vary quite a bit.

For my organization, the most valuable part of the tool stems from a variety of features within SQL tables and also data, which is a combination of Azure Data Lake Storage and Azure Blob Storage. Azure Data Lake Storage Gen2 is the best, and I think it is fantastic. From my point of view, the tool is considered to be very competitive here.

I have been working with storage tools for over twenty years, so I am developing my skill sets related to cloud solution providers, mainly on Azure since I began with that in 2008.

Take a deep dive into all the possibilities and options you get because you won't be disappointed. You need to do comparisons with other CSPs and other storage vendors, like NetApp and Dell EMC.

I rate the tool a ten out of ten.

Positive

We are using the solution for call monitoring and connecting Data Lakes. We have different data at various locations, both structured and unstructured, which we use for analytics.

The most valuable feature is the scalability due to the blob, which allows us to scale our data effectively. Data is stored on the blob and Lake, maintaining connectivity through the Data factory. We can write scripts on the Data factory to handle any type of data and store it on the cloud. This has saved us time.

We have not explored the AI features. It would be beneficial if some AI features were added.

We have been using the solution for a while now. 

We have not encountered any stability issues so far.

The solution is scalable, which allows us to manage our data effectively. Due to the blob, we can scale our data.

We handle problems internally and do not need to contact customer service.

We created our solution independently and did not use others.

The setup was straightforward, and we were able to deploy our solution on Azure without any problems.

We implemented the solution ourselves.

Exploring more AI features might enhance the solution.

I'd rate the solution seven out of ten.

We have parameters to create three types of data storage. The first is staging, the second is the intermediate, and the third is the target. The target storage contains the cleanest data used for reporting tools. For specific cases in Data Lake projects, we often use files stored in formats such as CSV, which are the most useful for this type of data processing.

Most of our clients use Excel. We prioritize making data accessible in formats compatible with Excel. We aim to meet these client requirements, for example, with Excel files. However, for big data solutions, it's often more efficient to use formats, which we store in the Data Lake.

In some projects, you can usually access files, enabling accessibility, mixing, and transformation. This is useful for both our team and for data engineers. For clients, it reduces costs by optimizing performance and calls and allows for implementing a security model. Additionally, using tools like Global Storage, you can create a hybrid cloud directory or restructure data, making it more organized and easier for clients to integrate with ETL tools.

When you store your files manually, you can't ensure complete data integrity, which can impact data security.

When you make these types of releases or improvements in this solution, you can enhance the data's stability. You can also include features like security integration with Active Directory for data access and ensure compatibility for various integrations. This approach complements both structured and unstructured data, making it more suitable for big data solutions.

I have been using Azure Data Lake Storage for two years

There are some limitations regarding data scalability. In such cases, you can complement it with Databricks. Databricks has a powerful engine that can enhance your security solutions. This combination provides great data performance and a better overall solution.

Once you pass the initial two weeks, it becomes simpler to manage technical support. However, before this period, it is important to have clear instructions, documentation, or videos prepared to assist with technical support and ensure a smooth deployment process.

It's great, but it can be challenging if you don't have all the necessary documents or lack one-on-one discussions about the solutions.

Positive

The initial setup depends on the project, but when storing this type of project, you need to have access activities to provide the source of the data. It's important to have a core level of security to access and transform the data. After storing the data, you need a solution for quality assurance to ensure data integration and quality. This is crucial because poor data quality can affect the future of your solution. Implementing data quality measures is essential for the success of your solutions.

Deployment depends, but it takes about two weeks. For example, one week is typically for deployment, and the other week is for checking the data flow to ensure no errors. If there is an error, you will receive an alert to check the status of the operation.

We need an Azure DevOps professional who makes many configurations to pass for a developer in a production environment.

The solution is worth the money because it allows you to gain insights into your business and implement forecasting solutions to predict future trends. Investing in such solutions is valuable for understanding and planning for future developments.

For ETL solutions, it’s essential to stay updated with new and recurring events that might arise. Reviewing forums, articles, and media can help you identify trends and troubleshooting tips relevant to your project. This proactive approach can help you anticipate and address potential issues effectively.

Overall, I rate the solution a nine out of ten.

We use the solution for the default data platform situation.

The solution's most valuable feature is its simplicity of configuration and setup. It also has good scalability, allowing you to add more data.

Simple migrations from one data lake to another take too much time and could be improved.

I have been using Azure Data Lake Storage for a couple of months.

The solution’s stability is very good.

I rate the solution’s stability a nine out of ten.

Our clients for Azure Data Lake Storage are usually enterprise businesses.

I rate the solution’s scalability a nine out of ten.

The solution’s deployment takes a few weeks.

On a scale from one to ten, where one is difficult and ten is easy, I rate the solution's initial setup an eight out of ten.

On a scale from one to ten, where one is cheap and ten is expensive, I rate the solution's pricing a seven out of ten.

Azure Data Lake Storage has slightly impacted the speed of data access. The solution's integration capability is very good, and I rate it an eight out of ten. All AI runs on data, and it has to be stored. It is usually stored in such an environment. I would recommend the solution to other users because it has good price quality.

Overall, I rate the solution an eight out of ten.