We have experienced several improvements across our organization. Our data ingestion processes previously involved copying data from S3 to Snowflake, which was fairly costly and required constant vigilance to purge old data so that our source tables would not bloat. Now we are able to move ingestion staging data to Iceberg tables, resulting in a much better experience in terms of both compute and storage costs as well as maintenance. Data transformation has also become more efficient. Starburst on Trino, combined with our SQL-native data transformation tool SQLMesh, has delivered anywhere from a two to five times improvement in compute performance across our transformation DAG. This improvement is largely due to how efficiently Trino scans relevant data without requiring any additional setup, such as defining partitions in Snowflake. In terms of cost effectiveness, we are already forecasting a 25% reduction in cloud data provider spending, even while continuing to use both Snowflake and Starburst. This is because we are able to shift a significant amount of compute to Galaxy, and the cost difference compared to our previous approach of running jobs exclusively on Snowflake is substantial.
