What is your primary use case for Astro by Astronomer?

My main use case for Astro by Astronomer is orchestrating the Airflow pipeline without hassling the backend of Airflow. I use Astro by Astronomer as a managed platform for Apache Airflow to orchestrate our data engineering workflows. We primarily use it for scheduling, dependency management, monitoring, and retrying ETL/ELT pipelines rather than executing heavy data processing itself. I use Astro by Astronomer day-to-day for ingesting data from APIs or cloud storage, running Azure Data Factory pipelines where required, triggering data based on PySpark transformations, performing data quality checks and validations, storing created data into a data warehouse or data tables, and sending notifications on success or failure through emails or notifications.

I conducted this exploration in my personal space and wanted to become aware of what features Astro by Astronomer has, so there is nothing to do with my company.

I tried Astro by Astronomer on my personal project. I attempted to rebuild some workflow, data, and data ingestion that I used to do at my previous job with Astro by Astronomer. I also tested Astro by Astronomer against other AI tools such as Gemini and Copilot to evaluate how Astro by Astronomer could create a folder structure for my project, create files, arrange configuration files, and create three folders. My main goal was to create some custom operators or to make suggestions on existing operators that are already deployed on GitHub.

We manage the deployment, governance, and everything for the different teams that want to orchestrate their workflows. All of their ingestion jobs, their analytics jobs, and their whole ETL or ELT jobs are built by the teams themselves, and we manage all of the deployment and governance for those jobs. One of the jobs we handle involves consuming data from different sources like Google Analytics and Apple Analytics. We use dltHub to have a standardization layer across the different ingestion jobs. We source the data from there, pass it through a schema check, and then sync it to our Databricks table. We also run quality checks on them and then publish the final reports. In the data platform, we use two different workspaces. One is for the admin deployments, which we, the central data team, use to orchestrate jobs such as rotating the credentials on schedule or running freshness checks and data quality checks. We then have a general workspace which different teams deploy their DAGs into. We use two branches for that: staging and main. We do not want anyone to deploy broken DAGs, so we have CI gates. Astro by Astronomer workspace cloud is connected to our main Astro by Astronomer repo. We have dispatch workflow jobs in GitHub which get triggered whenever there is a deployment in the child repos where the stakeholders deploy their DAGs. The dispatch workflow syncs the changes from the child repo to the main Astro by Astronomer repo. Based upon the change, if it is a simple DAG change, then we do not do a full deploy but just a DAG deploy, which is much faster. If there is a change in the infrastructure, then we do a full image deploy of Astro by Astronomer to that workspace or to that deployment.

My main use case for Astro by Astronomer is to proof out concepts of functionality and learn about the behind the scenes for the application. I'm not an admin on our company's instance, so my vantage point is limited. This helps me get a peek at what's behind the scenes and make better decisions based on what I've learned from that. I use Astro by Astronomer to test out determining concurrency limits and how to set them and do load balancing. I also can use it to test out different ideas for use cases and after I have given them a good test, I can possibly fold them into my work experience or personal, or I can keep them as personal projects on the side to keep exploring.

Our main use case for Astro by Astronomer involves data pipelines and general pipelines, integrating with AWS for different use cases across teams like analytics, data science, and many other different use cases.

My main use case for Astro by Astronomer is to build pipelines, specifically data pipelines, similar to an Apache Airflow tool which I use for day-to-day data engineering tasks. A specific example of a pipeline I've built with Astro by Astronomer is related to a project involving YouTube, where I obtained the data from the source and had to build a pipeline to send it to a destination called an S3 bucket so that downstream analytics can be performed on top of that, such as using it for AWS QuickSight and making dashboards.

My main use case for Astro by Astronomer is managing Airflow as an Astronomer. I am building ETL workflows that get scheduled by Airflow DAGs. I use Astronomer as an instance or as a service of Airflow. A specific example of how I am using Astro by Astronomer in my ETL workflow is that I have almost 500 to 600 different data pipelines where data gets ingested from different sources to the warehouse. The sources are hybrid by nature, and the target is also hybrid, but most of the time I am using Snowflake and Databricks to ingest data. Between the sources and targets, there are many steps to get data ingested, with different dependencies between the tasks and between the jobs. I have completed several API integrations using Airflow DAGs, which get executed based on events. The ETL jobs and data pipelines are my main use cases. I also have some online, on-demand jobs that get executed by event-driven workflows. These are the two main workflows that I am working on currently.