What is our primary use case?
We have parameters to create three types of data storage. The first is staging, the second is the intermediate, and the third is the target. The target storage contains the cleanest data used for reporting tools. For specific cases in Data Lake projects, we often use files stored in formats such as CSV, which are the most useful for this type of data processing.
How has it helped my organization?
Most of our clients use Excel. We prioritize making data accessible in formats compatible with Excel. We aim to meet these client requirements, for example, with Excel files. However, for big data solutions, it's often more efficient to use formats, which we store in the Data Lake.
What is most valuable?
In some projects, you can usually access files, enabling accessibility, mixing, and transformation. This is useful for both our team and for data engineers. For clients, it reduces costs by optimizing performance and calls and allows for implementing a security model. Additionally, using tools like Global Storage, you can create a hybrid cloud directory or restructure data, making it more organized and easier for clients to integrate with ETL tools.
What needs improvement?
When you store your files manually, you can't ensure complete data integrity, which can impact data security.
When you make these types of releases or improvements in this solution, you can enhance the data's stability. You can also include features like security integration with Active Directory for data access and ensure compatibility for various integrations. This approach complements both structured and unstructured data, making it more suitable for big data solutions.
For how long have I used the solution?
I have been using Azure Data Lake Storage for two years
What do I think about the stability of the solution?
What do I think about the scalability of the solution?
There are some limitations regarding data scalability. In such cases, you can complement it with Databricks. Databricks has a powerful engine that can enhance your security solutions. This combination provides great data performance and a better overall solution.
How are customer service and support?
Once you pass the initial two weeks, it becomes simpler to manage technical support. However, before this period, it is important to have clear instructions, documentation, or videos prepared to assist with technical support and ensure a smooth deployment process.
It's great, but it can be challenging if you don't have all the necessary documents or lack one-on-one discussions about the solutions.
How would you rate customer service and support?
How was the initial setup?
The initial setup depends on the project, but when storing this type of project, you need to have access activities to provide the source of the data. It's important to have a core level of security to access and transform the data. After storing the data, you need a solution for quality assurance to ensure data integration and quality. This is crucial because poor data quality can affect the future of your solution. Implementing data quality measures is essential for the success of your solutions.
Deployment depends, but it takes about two weeks. For example, one week is typically for deployment, and the other week is for checking the data flow to ensure no errors. If there is an error, you will receive an alert to check the status of the operation.
We need an Azure DevOps professional who makes many configurations to pass for a developer in a production environment.
What was our ROI?
The solution is worth the money because it allows you to gain insights into your business and implement forecasting solutions to predict future trends. Investing in such solutions is valuable for understanding and planning for future developments.
What other advice do I have?
For ETL solutions, it’s essential to stay updated with new and recurring events that might arise. Reviewing forums, articles, and media can help you identify trends and troubleshooting tips relevant to your project. This proactive approach can help you anticipate and address potential issues effectively.
Overall, I rate the solution a nine out of ten.
Which deployment model are you using for this solution?
Public Cloud
*Disclosure: My company has a business relationship with this vendor other than being a customer. Partner