Dremio is a platform that enables you to perform high-performance queries from a data lake. It helps you manage that data in a sophisticated way. The use cases are broad, but it allows you to make extremely good use of data in a data lake. It gives you data warehouse capabilities with data lake data.
Dremio enables you to manage changes more effectively than any other data warehouse platform. There are two things that come into play. One is data lineage. If you are looking at data in Dremio, you may want to know the source and what happened to it along the way or how it may have been transformed in the data pipeline to get to the point where you're consuming it.
There's another thing called data providence. They're tied together. Data providence allows you to go back and recreate the data at any particular point in time. It's extremely important for compliance and governance issues because data changes all time. How did it change? What was it three days or months ago? You may have made some decisions based on data that was three months old, so you might need to revisit those.
It's essential for things like machine learning and deep learning, where you are generating AI models off data. When the model stops working or doesn't work as expected, you need to figure out why. You have to go back and adjust the datasets used to train the model. We do that through an open-source project called Nessie, which is their basis for providing data lineage and data province capabilities. It's super powerful.
Arrow is another open-source project for storing data in memory and performing data query operations. Data sits on a disk in one format. If you want to do anything with data, you have to load it into your computer and put it into memory so you can work with it. Arrow provides a format in memory that enables the whole library to perform various operations on that data. Every vendor has its own way of representing data in memory.
They've latched onto an industry standard and developed it so it's open. Now people can use the exact same format in memory to do operations and use the library set to perform functions on data. New developers can decide if they want to develop their own memory format or use one that's already there.
Data transfer is a massive problem when you're working with large datasets, doing advanced analytics, and trying to train machine learning or deep learning models. What happens often is companies downsample their data sets to do training on models because transferring and managing data on a deep learning or machine learning platform is too much.
They have an automated tool for building SQL queries, so you don't need to know SQL. That interface works, but it could be more efficient in terms of the SQL generated from those things. It's going through some growing pains. There is so much value in tools like these for people with no SQL experience. Over time, Dermio will make these capabilities more accessible to users who aren't database people.
The core functionality is rock solid in my experience.
Dremio is designed for massive scale-building.
I rate Dremio eight out of 10. Dremio focuses on enterprise customers, so they have a solid support organization. They have professional services and full support. They have a great customer-centered team, but there are some challenges. For example, they don't have enough staff. As more customers buy Dremio, they need more integrators to help them get started. They can't do it all themselves.
The initial setup is pretty straightforward. I've set it up in two different ways. The first was a three-node configuration. I used virtual machines to set up three different nodes and cluster data. That worked fairly well. I also created containerized versions using Kubernetes. It was much easier to do that. Dremio also has cloud versions where you don't need to do anything. You can go to the cloud and consume it.
I rate Dermio 10 out of 10. Data lake 2.0 or 3.0 is gradually replacing data warehouses, but it's not quite there yet. We can't get rid of data warehouses, but it's heading in that direction. Dremio is a phenomenal place to start with that mindset shift and architecture shift. Despite the shortcomings, Dremio has made better progress in moving the industry forward than anybody else out there.
