Dagster Labs Reviews

I perform orchestration around data by connecting raw app data using Fivetran inside BigQuery warehouse to DBT. I do this entire process using Dagster Labs.

My workflow is that Fivetran reads data from the raw data source and sends the data to BigQuery. From BigQuery, the data is read inside the BI dashboards. The entire pipeline from raw to Fivetran to BigQuery to BI dashboard is monitored inside Dagster Labs.

What I find beneficial with Dagster Labs is that it excels at pipeline monitoring. It provides good notification of schema changes and indicates whether the pipeline is stable or not.

I appreciate the visual lineage graph that Dagster Labs offers. It shows exactly which upstream table or API pull failed, timed out, or produced an anomaly.

Dagster Labs has given me better monitoring ability. I know whether data can be trusted based on what I see on the dashboards. This gives me more confidence in my data sources.

Dagster Labs has provided a good sense of the data pipeline I have been using. I have started to trust it even more because I have full confidence in whether it has been updated correctly and if it has failed. When there are any anomalies, I end up trusting my data even more. I have full confidence and control over the data pipeline. Overall, it has created a very positive impact.

One of the features I appreciate in Dagster Labs is resources, which is a cool capability to have.

There are many ways Dagster Labs can be improved. I believe the UI is very slow and it prevents loading state.

Regarding needed improvements, I think there should be very good support available for migration if someone is migrating from a legacy system to Dagster Labs. Maybe that support needs to be better in a way that there should be customer service available for it, with more documentation or GitHub documentation that details what you should do and should not do, including do's and don'ts and how it will be beneficial or not beneficial. Something approach may be helpful during migration.

I have been using Dagster Labs for around a year.

It has definitely improved my downtime by twenty percent in relation to Dagster Labs.

Based on how Dagster Labs works, I have more confidence in them. They would surely have a very good model to support all of the features they are providing.

My advice to others looking into using Dagster Labs is to take it slow and give time for things to develop. Things will get better.

I would rate this product an eight out of ten.

My main use case for Dagster Labs is data pipeline orchestration.

A specific example of how I am using Dagster Labs for data pipeline orchestration is that Dagster can be used with dbt, which is another technology that allows scheduling data modeling pipelines and integration with other tools. This integration with dbt is particularly useful, and on the monitoring side, we can add data quality monitors to run on a schedule. Additionally, Dagster Labs integrates with Slack for alerts. The specific example is running data models on a schedule.

Regarding my use case, we also use Dagster Labs to integrate with some AWS components, such as AWS Batch jobs, and we use Dagster's Dagster Pipe technology to connect with our orchestration runs within Dagster, although the process is orchestrated into AWS services. While this is not unique, it is one of our use cases. Another use case is triggering remote Spark jobs, which is connected to data pipeline orchestration, making it one of Dagster Labs' official integrations.

Dagster Labs offers excellent features, including a lot of integrations out of the box, allowing for integration with all major players in the data landscape. Another great feature is the polished UI and developer experience. Unlike some other products with clunky UIs, Dagster Labs provides a pleasant experience with a single glass panel approach. This makes it easy to quickly glance at pipeline health, asset lineage, and everything important for a data engineer is just one or two clicks away. Additionally, it is an extensible product, enabling power users to easily adapt available libraries and code to fit their use cases.

A standout feature is the dbt integration, which is probably well-known and regarded as the industry's top integration, prompting my organization and others where I worked to use Dagster Labs.

Dagster Labs has positively impacted my organization mainly with cost management, allowing us to move away from a more locked-down tool, dbt Cloud. The impact was migrating all our dbt Cloud pipelines to Dagster Labs, which allowed us to save money.

Regarding cost savings, I find that while I do not see a reduction in employee hours or improved efficiency—since users were accustomed to working with dbt Cloud and it allowed more business users to interact with the tool—with Dagster Labs, the cost savings are primarily raw savings due to the platform price, limiting access to previous pipelines.

Some points of improvement for Dagster Labs include leveraging integrations with other great tools such as SQLMesh, which many people are interested in, as shown by its traction and frequent requests on GitHub. Additionally, the billing for Dagster Labs Cloud Plus is somewhat restrictive regarding the number of seats, which often pushes users towards the enterprise plan. This raises concerns for mid-sized companies.

I have been using Dagster Labs since 2021.

Dagster Labs is stable, with a really good release cycle.

Dagster Labs is scalable based on your compute environment, allowing for serverless scaling when using Dagster Labs Cloud. While there are limitations on workload size, transitioning to your own cloud platform resolves scalability issues, particularly suitable for Kubernetes.

I only used customer support while setting things up, and although I have not needed to reach out since, my experience was good and responsive.

In previous use cases, we utilized dbt Cloud, which is primarily a tool designed for dbt and has even worse pricing. We switched to Dagster Labs, and before that, we also used open source alternatives such as Prefect and Airflow, which are acceptable tools, with Airflow being the industry standard. However, I believe Dagster Labs is moving in the right direction.

My experiences with pricing, setup cost, and licensing vary. For my current organization, we opted for Dagster Labs open source, and we are considering moving to the cloud offering. In a previous consultancy role, we selected the standard plan, and while the pricing was acceptable, it became limiting when it came to the number of seats, pushing us to choose a simpler plan. The setup cost was negligible as I had prior experience setting up Dagster Labs, and licensing was not a major concern for us.

It is difficult to quantify a return on investment, especially since I am in an individual contributor position and do not have access to metrics as I am not on the leadership side of the organization.

Before choosing Dagster Labs, we evaluated Airflow, specifically its AWS offering, and also looked at Prefect in other roles.

I would rate Dagster Labs an eight out of ten. I chose an eight out of ten mainly due to the pricing and the way it handles seats, which can be unwelcoming for smaller organizations. As a product, it is one of the best I have had the pleasure to work with.

My advice for others looking into using Dagster Labs is to be wary of the pricing model. Understanding how to best leverage Dagster Labs hinges on having a solid grasp of the tool and its extensibility, enabling users to expand use cases creatively, while also avoiding overcomplicating pipelines. Dagster Labs' documentation offers a solid foundation to begin working.

I think it is crucial for Dagster Labs to keep maintaining the open source solution, as I believe it is a good gateway for new customers and essential for the data landscape, despite the for-profit nature of the company.

The main use case for using Dagster Labs is to utilize ETL processes such as extract, transform, and load because we already have raw data, and we perform all the necessary transformations, including feature engineering and mathematical calculations, and then we load it to the front end so that customers can see how their data has evolved into something useful, allowing them to make actionable decisions.

Let me consider a scenario where a customer has raw and unreadable data. We need to identify features first. For example, a customer may have IP addresses with last seen timestamps along with raw security logs. During the initial feature engineering phase, we check for use cases such as detecting if a user on the customer side is using unauthorized resources. To accomplish this, we need to perform feature engineering on several attributes, such as the user's authorization level and the objects they are accessing. For a second example, we can detect if a user is authorizing company access after work hours by using the last seen attribute. We can perform feature engineering on such scenarios.

That covers my main use case, but we are also currently trying to use the DLT tool by using Dagster Labs so that we can extract the data as well.

The best features Dagster Labs offers are that it is very beginner-friendly, the UI is very nice, and we can backfill most of the processes. The backfill feature is the best feature I would consider.

When any new user comes in, they can really see the UI and do all sorts of things rather than going into code-level work. When it comes to backfill, if an error occurred in the past and we want to retrieve the data from that point, we can do that using backfill, which makes it good for our workflow.

Dagster Labs has positively impacted our organization by making it much easier to create data pipelines, helping us create new use cases, and allowing us to transform all of the data we have into features, which Dagster Labs has greatly assisted with.

I can say that since using Dagster Labs, we have saved time, increased productivity, and also cut down on some third-party vendor applications, which saves us money in infrastructure costs and other expenses.

Dagster Labs is currently providing a Dagster University course, but those offerings are somewhat high-level. When a user comes in and tries to learn Dagster Labs, there need to be different kinds of projects that they can understand. I do not think that Dagster University course has that.

Also, when we are doing something in Dagster Labs, we need to know that when I make a change in the UI, it needs to be reflected in the code, so as a developer, I can check on the code and understand the workflow. The UI needs to be connected to the code in real-time.

I have been using Dagster Labs for the past four to five months.

Dagster Labs is stable.

Dagster Labs's scalability is good because you can easily create different pipelines for different purposes, which is beneficial.

I would advise others looking into using Dagster Labs to utilize all of the features provided by Dagster Labs, as it has many great offerings. I would rate this product a seven out of ten.

I primarily use Dagster Labs for asset-based orchestration, scheduling and automating pipelines, and visualizing dependencies between datasets, and testing it. I test unit test cases in the pipeline.

A practical example of how I use Dagster Labs for asset-based orchestration is when I take the raw orders, clean orders, sales fact tables, daily revenue summary, and VBA dashboard. I use Dagster Labs to track the dependency between these assets. When a raw table arrives, the dependent assets are recomputed. If the sales table fails due to a schema change, Dagster Labs prevents downstream assets from running, highlighting the failed assets in the lineage graph. This allows me to check the logic behind the data, and it also allows me to rematerialize only the affected assets after the fix. This asset-centric approach makes it more reliable and easier to debug when orchestrating individual assets.

I also use Dagster Labs to create scheduled pipeline jobs for running the pipeline.

One of the best features that Dagster Labs offers is the asset-aware data orchestration.

Asset-aware data orchestration is valuable for me because it becomes easy to find out when an asset fails and how the asset failed when the entire pipeline runs. It becomes easy for me to fix the problem, rerun the asset, materialize it, and see it in the lineage graph.

Dagster Labs has positively impacted my organization because whenever there is a change in the data, it becomes easy for me to schedule the job, run the changes, and make changes to be imported to the slowly changing dimension tables. It makes my work easier.

I think it has been great for me to use Dagster Labs, and I would not suggest any areas where it could be better because I have not fully utilized Dagster Labs to its fullest potential. It has been great for me.

I do not have any improvements needed for Dagster Labs that I have not mentioned, and there is nothing else I would like to see changed or added.

I have been using Dagster Labs for quite some time, approximately one and a half years.

The specific outcomes I have noticed since using Dagster Labs include saving time and reducing errors. It becomes easy when I run the pipeline in everyday cases instead of someone manually running it. Dagster Labs makes it easy to run scheduled pipelines more often when a large chunk of data comes into the data table or the database. It reduces dependencies on manual work and personnel, and is also efficient in consuming the data that has changed in the SCD data tables.

I have not used Dagster Labs's AI capabilities, so I have no idea about its governance and security.

Regarding Dagster Labs's AI capabilities, I have not used it, so I would not give any metrics on its accuracy or reliability of output because I have not relied more on the AI capabilities of Dagster Labs.

Dagster Labs is deployed in my organization in a private cloud.

This interview was great, and I would not recommend any changes for the future. I gave this review a rating of eight.