What is our primary use case?
We manage the deployment, governance, and everything for the different teams that want to orchestrate their workflows. All of their ingestion jobs, their analytics jobs, and their whole ETL or ELT jobs are built by the teams themselves, and we manage all of the deployment and governance for those jobs.
One of the jobs we handle involves consuming data from different sources like Google Analytics and Apple Analytics. We use dltHub to have a standardization layer across the different ingestion jobs. We source the data from there, pass it through a schema check, and then sync it to our Databricks table. We also run quality checks on them and then publish the final reports.
In the data platform, we use two different workspaces. One is for the admin deployments, which we, the central data team, use to orchestrate jobs such as rotating the credentials on schedule or running freshness checks and data quality checks. We then have a general workspace which different teams deploy their DAGs into. We use two branches for that: staging and main. We do not want anyone to deploy broken DAGs, so we have CI gates. Astro by Astronomer workspace cloud is connected to our main Astro by Astronomer repo. We have dispatch workflow jobs in GitHub which get triggered whenever there is a deployment in the child repos where the stakeholders deploy their DAGs. The dispatch workflow syncs the changes from the child repo to the main Astro by Astronomer repo. Based upon the change, if it is a simple DAG change, then we do not do a full deploy but just a DAG deploy, which is much faster. If there is a change in the infrastructure, then we do a full image deploy of Astro by Astronomer to that workspace or to that deployment.
What is most valuable?
There are a lot of new features that we have started using in Astro by Astronomer. Some of them I can list is their new AI agent, Auto. It is quite useful for investigating why a DAG has failed and getting more context around it. The other feature is Cosmos, where since most of our jobs are running dbt models, Cosmos provides us a way to run dbt natively. We can have granular-level logs and retries and lineage and everything. That is a useful feature.
Then there are Datasets where we can link different DAGs and to have a comprehensive framework and have a freshness check and quality checks on the entire stack. The Blueprint feature is quite useful since many of our ingestion jobs that are being created by stakeholders may not be quite familiar with Airflow, and with Blueprint, we provide them a way to use a no-code solution to build their DAGs. Astro by Astronomer Observe has a lot of features that we use: the observability and alerting that we set up.
Apart from that, the whole having the entire deployments and the DAGs and everything in one place is quite useful. In MWAA, everything is decentralized, and that creates a lot of friction and problems. In Astro by Astronomer, I find it really useful that everything can be in one place. Also, one of the good features in Auto, we can do the migration from, let's say, Airflow 2.x to 3.x using Auto seamlessly.
Instead of giving generic messages in, for example, a Slack channel when an Airflow DAG fails, you can use Auto to have a comprehensive root cause analysis of what has failed and based upon the context of the DAG and its historical runs, it gives a much more comprehensive analysis of what has failed, and that speeds up the fixing the bug and fixing the DAG cycle a lot more. Instead of going through the entire debugging process, having the root cause analysis provided helps tremendously. Sometimes we have to dig in a bit deeper, but it does help a lot.
Previously when we were using MWAA, since our DAGs were living in S3 buckets, the workers running in EC2 and having logging in CloudWatch, everything was scattered around the place and doing a deployment or debugging was quite a pain. Astro by Astronomer solves that seamlessly, where in one place, we have everything. Granting access control is quite easy. We have created different teams in Astro by Astronomer and integrating those teams with Okta, we can have seamless integration of which user can access which workspace. Also, whenever a user leaves the organization, their Okta is disabled and automatically they lose access. In that regard, Astro by Astronomer has helped tremendously.
When we have to do an upgrade, for example, upgrading the Airflow version, that was quite problematic in MWAA, and there used to be a downtime of approximately 40 to 45 minutes while the Airflow web server had to spin up and do its thing. That seems quite seamless in Astro by Astronomer.
Also having alerts, whenever a DAG fails, we have a direct PagerDuty integration for the important DAGs. If that DAG fails, we immediately get notified on our PagerDuty schedule. That is also one of the features that helped us tremendously. Having monitoring on top of that, Astro by Astronomer, integrating it directly with DataDog has been quite useful. That was quite challenging when using MWAA. Additionally, Astro by Astronomer seems to be quite cheaper than MWAA overall.
What needs improvement?
I would say it would be much more helpful if Astro by Astronomer can provide an MCP to use and integrate with Astro by Astronomer and their AI agent Auto, so that we can use our tools like Claude, Code, or Cursor to have a central location where we integrate that MCP in Databricks via the AI gateway, and then we can have a place where we can connect to Astro by Astronomer, Databricks, other tools such as Jira and DataDog, and then have a much more comprehensive analysis and overview and monitoring of the entire data ecosystem.
Having an MCP integration would tremendously help us to have a more comprehensive data platform that is more AI-ready. That is one of the features that I can think of. Another thought would be having an integration with Kubernetes instead of just scheduling everything on Astro by Astronomer workers. That would be helpful.
One thing I can think of is that the Astro by Astronomer local developer environment can be improved further, but perhaps that is a very niche use case that I have in mind, and it may not be useful for others.
There is room for improvement around having an improved UI and also integration with other tools.
On governance, I would say we can have a more granular level of governance even inside workspaces. Having much more granular control over people belonging to the same team in Astro by Astronomer could have different access to different DAGs, depending upon our tagging strategy. That is one of the areas I can suggest in terms of improvement.
The AI capabilities can be improved further. I would compare the current AI capabilities to newer models provided by Claude such as Opus or Codex.
For how long have I used the solution?
I have been working as a Data Engineer for more than eight years. I started at Babbel last year, so I am almost going to complete one year at Babbel.
What do I think about the stability of the solution?
It was not 100% seamless, since there were a few complexities that we had to overcome. Some of our DAGs were tightly coupled with how MWAA works, particularly the whole IAM roles and everything. Additionally, Secrets Manager was an AWS native product. In MWAA, there was a seamless integration with that, but when we migrated to Astro by Astronomer, those were some of the challenges that we had to overcome. It was quite manageable. I would give that experience an 8 out of 10.
What do I think about the scalability of the solution?
Scalability-wise, it is very good. We do not have to worry about selecting the right worker size, and it is quite scalable. We did hit the limits sometimes, but that was due to us not estimating the workload properly. However, with the monitors, the DataDog monitors and the Astro by Astronomer alerts that we had in place, we were notified quite in advance, and we could increase the worker capacity in time. That was good. We also use the Astro by Astronomer executor, which combines the features of both Celery and Kubernetes executor, so the overall experience was quite seamless.
How are customer service and support?
Customer support is great. We have support engineers helping us if we face any issues. Additionally, we have a bi-weekly call with the Astro by Astronomer product team where they guide us to the new features and help us if we can use some of their new features or if we are following the best practices in our current setup. It is quite good.
Which solution did I use previously and why did I switch?
We were using MWAA for Airflow before that. We switched because of the reasons I have already mentioned.
How was the initial setup?
Astro by Astronomer acts as our orchestration layer. All of our jobs, including our ingestion jobs, our transformation jobs, analytics, machine learning, and AI workloads, are everything orchestrated using Astro by Astronomer now.
What about the implementation team?
We had a call with the Astro by Astronomer product team, and then they pitched us the idea, and then we got onboarded with them.
What was our ROI?
Our MWAA workers and everything, including deployments and everything, used to cost us around three thousand dollars previously to run the entire stack. Now, currently, after Astro by Astronomer, we could manage it for probably one thousand five hundred to one thousand eight hundred dollars per month. That is quite a significant improvement.
What's my experience with pricing, setup cost, and licensing?
Whenever there is a deployment, Astro by Astronomer seems to have zero downtime, which used to be around 40 minutes when doing an MWAA deployment.
What other advice do I have?
If you want a one-stop solution for your orchestration, I think Astro by Astronomer is the best place to look. The overall product experience is quite good. The customer support is good. You should definitely check out Astro by Astronomer. I would rate this solution an 8 out of 10.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?