AWS Glue Reviews

I am using this solution for ETL programming, which involves extraction, transformation, and loading. We have a data warehouse, Redshift. We receive some data files from outside, save those files into S3, and then call our Glue jobs to process those files. We perform transformations, including various aggregations and grouping of data. Sometimes, we also perform data cleansing, refining the data in a proper way so that customers can generate reports against it. For different models, we transform the data and perform a bulk load or an ETL incremental load into the rest of the data warehouse. We have also done some tie-outs, like comparing two different datasets, ensuring they tie out as expected, and creating exception reports. We generate mismatches in the data and perform an unload process to create exception reports out of Glue. This process involves creating a report against the table if there is a data mismatch. We extract the data and put it into an S3 dump for TPUs to use for their exceptions. If a particular data file has an issue, we identify the problem and address it to improve the quality of the data.

For ETL, I feel the performance is excellent. If I create jobs in a standard way, the performance is great, and maintenance is also seamless. It is serverless, so I do not need to worry about other integrations. It runs on a serverless VM. Glue services from AWS are regularly upgraded with new properties and functions. We use PySpark and other frameworks, which help customize our ETL process in a standard way. During our process, we utilized the typical SQL ITIL process standard in Glue-oriented ETL services, which helped us improve performance and time. AWS running time, mostly during the morning, saved us time compared to the old ETL method. There are many email notifications once a job is successful or fails, and we get notifications using SES or AWS's Simple Notification Service. Glue is integrated with both email alerts, and we are responsible for managing the full data journey from end to end.

The first time I used AWS Glue was in my first project where I automated large-scale ETL pipelines using AWS Glue and Python. I also used AWS Glue for data cataloging and transformation workflows.

Since then, I consistently use AWS Glue in other projects for ETL orchestration, especially in AWS-based environments, and it is one of my go-to tools for data integration.

AWS Glue has reduced efforts by 60%, which is the main benefit. Using AWS Glue allows for automatic scaling and reduces operational time.

Regarding data catalog integration, the Glue Data Catalog helps manage metadata across various services. Because I work heavily with PySpark, and AWS Glue supports it, it is very easy for me to build and debug pipelines.

For workflow orchestration, it is easy to define triggers, dependencies, and workflows within AWS Glue. Overall, AWS Glue is very efficient and integrates well with the AWS ecosystem.

My use case is the usual: data ingestion, curation, and transformation.

I think if I'm working with big data, common languages like Python work quite nicely, which is advantageous. Despite this, it's had a limited impact just because of our particular circumstances.

I use AWS Glue for transforming data quickly, especially when dealing with small amounts of data that need to be processed swiftly. We use Glue to perform calculations and transformations required from source systems and then store the processed data in databases such as RDS or Redshift. We also use it because of its seamless integration with Apache Airflow.

AWS Glue offers a cost-effective and efficient solution for running longer sessions beyond the 15-minute restriction of AWS Lambda. This helps in processing data quickly and seamlessly.

I use AWS Glue mostly for data ingestion or data extraction from multiple sources.

AWS Glue plays a central role in AI-based solutions and machine learning workflows by efficiently handling and moving data around, contributing to streamlined operations.

We use AWS Glue for handling data-intensive tasks such as data lake creation, log analysis, machine learning pipelines, data warehouse population for analytics, and real-time data integration with AWS Lambda.

AWS Glue has increased efficiency and time saving. It simplifies and automates data pipeline processes, enabling faster data processing and analysis.

In my role as the global lead for AWS solutions and offerings, we work with various clients, including large-scale clients, to adopt and implement AWS cloud offerings. 

Our primary focus revolves around cloud lift-and-shift migration, modernization, re-platforming, rehosting, data architecture, design strategy, and implementing generative AI-specific solutions across different industries such as banking, capital insurance, energy utilities, manufacturing, automotive, semiconductor, and aerospace and defense. 

For example, we have implemented AWS Glue at several client locations, utilizing its serverless data integration capabilities during the data discovery process, enterprise transformation, cleansing, transforming, and centralizing data.

AWS Glue has significantly improved our data quality, enhancing the data by removing duplicates and providing timely and efficient insights. 

It also aids in real-time data processing, reducing effort and cost due to its serverless architecture. These features ensure we maintain the highest level of scalability, reliability, and security compliance.