What is your primary use case for Astro by Astronomer?

My main use case for Astro by Astronomer is to manage our data pipelines, to run DBT processes, and to run all data from the company. I use Astro by Astronomer in my day-to-day work with schedules from different pipelines, each from different areas of the company. They process data from the sources with the DBT components, and we transform the data inside Snowflake. Regarding my main use case with Astro by Astronomer, we changed it; first, we had an installation of Astro by Astronomer that was running on our AWS account, but we moved it to Astro Cloud, which is managed by Astronomer because it is easier to manage and we do not have the toil of managing the Kubernetes clusters and the problem of having to upgrade those clusters and the application. The only consideration is that each cluster on Astro Cloud has a fee from Astronomer.

My main use case for Astro by Astronomer is to orchestrate data pipelines in the cloud and orchestrate an end-to-end data flow for a Lakehouse or Data Lake for batch processes.A specific example of how I've used Astro by Astronomer in one of my recent projects is in the flagship project that we have, a framework that is in charge of orchestrating all the different ingestions that are done for the lakes the company has. I orchestrate the ingestion from different sources and then orchestrate the processing to take it to the different layers: bronze, silver, and gold. I would say it is the core of orchestration between layers within the Lakehouse. It is the core of my day-to-day work. We have more than 50 pipelines running for different lakes across different sources in different lakes, according to domains within the company. Using Airflow is the core to be able to perform all the batch processes and migrations from sources until reaching tables or the semantic layer. It not only orchestrates ingestion; it also orchestrates processing with DBT, modeling with DBT, and processing with EMR and Glue Job.

My main use case for Astro by Astronomer depends on the project use cases. For something like big data projects on ETL pipeline and ELT pipelines, it truly depends on projects to projects. I would say first you can go with something modern data warehouse and lake house orchestrations. Why do we choose Astro by Astronomer? Because it has built-in data lineage which tracks the data flow down to the table and column level, making it very easy for a data owner to know how the data is flowing. Secondly, it provides multi-tenant and multi-cloud pipeline integrations, allowing integration with any cloud vendors whether it is AWS, Azure, or GCP. Third, I would say that it has wonderful production machine learning and AI pipelines, where MLOps teams can use Astro by Astronomer to orchestrate end-to-end machine learning workflows with distributed frameworks such as Ray, Spark, or SageMaker to evaluate metrics. It creates complex DAGs for you. Lastly, on CI/CD driven analytics engineering, engineering teams moving towards data operations use Astro by Astronomer to automate testing and deployments in my current organization. Developers spin up the local Airflow environment using the Astro CLI, write some test code, and push it to GitHub or GitLab which automatically deploys to production via CI/CD. Astro by Astronomer, a commercialized version of Airflow, works on local dev and production, making it easy for people to test in the dev environment. A quick specific example of one of these use cases is that in our recent team, we run Snowflake, Databricks, and BigQuery as well as DBT. Astro by Astronomer serves as the central orchestration pipeline or orchestration plane, triggering the DBT transformations and syncing the ingest parts, such as from Kafka or Fivetran, while running quality checks on the data and notifying downstream tools, such as BI tools including Looker or Tableau. We chose Astro by Astronomer for its built-in lineage, which helps track the data flow from the table and column level. This is a use case of Astro by Astronomer that we use in multiple teams within Zalando, where we rely on different cloud software or platform as a service, with Astro by Astronomer forming the central orchestration layer. Regarding my main use cases, a small example of our e-commerce daily revenue and customer reporting occurs at 2:00 AM. The daily team pulls raw data, aggregates financial metrics, computes churn risk, and refreshes executive Looker dashboards in BI dashboards. The workflow begins when S3 ingestions come in, which go to Snowflake. When the data arrives in Snowflake, we have triggers written on Astro by Astronomer. From there, the DBT pipeline takes over, writing it into staging for clean and deduplicated data, with quality tests validating success or failure, including any malformed data issues. This way, we utilize Astro by Astronomer precisely, rather than writing messy bash operators to execute DBT runs as a single opaque block. Astro by Astronomer, developed by Astronomer Cosmos, alleviates this by allowing us to manage individual DBT models and test them in discrete observable Airflow tasks. It makes life easier by adhering to Astro CLI standards, which are module-based and consistent. It greatly aids us with granular model observability that is not found in a single DBT test and offers zero downtime secret and environment management, ensuring the environment remains consistently up. Moreover, it provides frictionless local development by allowing data engineers to run Astro dev start, spinning up a Docker context that mimics the production Airflow one-to-one, allowing for testing DBT models against a local development schema, pushing branches into GitHub, and letting CI/CD automatically deploy updated DAGs to Astro by Astronomer cloud.

I have been using Astro by Astronomer CLI as a local development environment for about a year. During this period, it has been the main tool I use to develop, test, and validate my Apache Airflow projects before deploying them to production. In production, I use Apache Airflow together with Astronomer Cosmos to orchestrate dbt pipelines. My main use case for Astro by Astronomer is using the CLI as a local development environment for Apache Airflow projects. I use it to develop, test, and validate DAGs before deploying them to production, ensuring everything works correctly. In addition, in my day-to-day work, I use Astronomer Cosmos to integrate and orchestrate dbt pipelines with Apache Airflow, running data transformations on Amazon Athena with Apache Iceberg tables. In my day-to-day work, I use Astro by Astronomer CLI to develop and test new Apache Airflow DAGs locally before publishing them to the production environment. For example, when I need to create a new data pipeline with dbt using Astronomer Cosmos, I first validate all the orchestration locally with Astro by Astronomer CLI, check that the dependencies, tasks, and integrations are working correctly, and only then do I deploy it to the production environment. This reduces errors and makes the development process much faster and more reliable.

My main use case for Astro by Astronomer is orchestrating the Airflow pipeline without hassling the backend of Airflow. I use Astro by Astronomer as a managed platform for Apache Airflow to orchestrate our data engineering workflows. We primarily use it for scheduling, dependency management, monitoring, and retrying ETL/ELT pipelines rather than executing heavy data processing itself. I use Astro by Astronomer day-to-day for ingesting data from APIs or cloud storage, running Azure Data Factory pipelines where required, triggering data based on PySpark transformations, performing data quality checks and validations, storing created data into a data warehouse or data tables, and sending notifications on success or failure through emails or notifications.

I conducted this exploration in my personal space and wanted to become aware of what features Astro by Astronomer has, so there is nothing to do with my company.

I tried Astro by Astronomer on my personal project. I attempted to rebuild some workflow, data, and data ingestion that I used to do at my previous job with Astro by Astronomer. I also tested Astro by Astronomer against other AI tools such as Gemini and Copilot to evaluate how Astro by Astronomer could create a folder structure for my project, create files, arrange configuration files, and create three folders. My main goal was to create some custom operators or to make suggestions on existing operators that are already deployed on GitHub.

We manage the deployment, governance, and everything for the different teams that want to orchestrate their workflows. All of their ingestion jobs, their analytics jobs, and their whole ETL or ELT jobs are built by the teams themselves, and we manage all of the deployment and governance for those jobs. One of the jobs we handle involves consuming data from different sources like Google Analytics and Apple Analytics. We use dltHub to have a standardization layer across the different ingestion jobs. We source the data from there, pass it through a schema check, and then sync it to our Databricks table. We also run quality checks on them and then publish the final reports. In the data platform, we use two different workspaces. One is for the admin deployments, which we, the central data team, use to orchestrate jobs such as rotating the credentials on schedule or running freshness checks and data quality checks. We then have a general workspace which different teams deploy their DAGs into. We use two branches for that: staging and main. We do not want anyone to deploy broken DAGs, so we have CI gates. Astro by Astronomer workspace cloud is connected to our main Astro by Astronomer repo. We have dispatch workflow jobs in GitHub which get triggered whenever there is a deployment in the child repos where the stakeholders deploy their DAGs. The dispatch workflow syncs the changes from the child repo to the main Astro by Astronomer repo. Based upon the change, if it is a simple DAG change, then we do not do a full deploy but just a DAG deploy, which is much faster. If there is a change in the infrastructure, then we do a full image deploy of Astro by Astronomer to that workspace or to that deployment.

My main use case for Astro by Astronomer is to proof out concepts of functionality and learn about the behind the scenes for the application. I'm not an admin on our company's instance, so my vantage point is limited. This helps me get a peek at what's behind the scenes and make better decisions based on what I've learned from that. I use Astro by Astronomer to test out determining concurrency limits and how to set them and do load balancing. I also can use it to test out different ideas for use cases and after I have given them a good test, I can possibly fold them into my work experience or personal, or I can keep them as personal projects on the side to keep exploring.

Our main use case for Astro by Astronomer involves data pipelines and general pipelines, integrating with AWS for different use cases across teams like analytics, data science, and many other different use cases.

My main use case for Astro by Astronomer is to build pipelines, specifically data pipelines, similar to an Apache Airflow tool which I use for day-to-day data engineering tasks. A specific example of a pipeline I've built with Astro by Astronomer is related to a project involving YouTube, where I obtained the data from the source and had to build a pipeline to send it to a destination called an S3 bucket so that downstream analytics can be performed on top of that, such as using it for AWS QuickSight and making dashboards.

My main use case for Astro by Astronomer is managing Airflow as an Astronomer. I am building ETL workflows that get scheduled by Airflow DAGs. I use Astronomer as an instance or as a service of Airflow. A specific example of how I am using Astro by Astronomer in my ETL workflow is that I have almost 500 to 600 different data pipelines where data gets ingested from different sources to the warehouse. The sources are hybrid by nature, and the target is also hybrid, but most of the time I am using Snowflake and Databricks to ingest data. Between the sources and targets, there are many steps to get data ingested, with different dependencies between the tasks and between the jobs. I have completed several API integrations using Airflow DAGs, which get executed based on events. The ETL jobs and data pipelines are my main use cases. I also have some online, on-demand jobs that get executed by event-driven workflows. These are the two main workflows that I am working on currently.