AWS Glue Reviews

I am using this solution for ETL programming, which involves extraction, transformation, and loading. We have a data warehouse, Redshift. We receive some data files from outside, save those files into S3, and then call our Glue jobs to process those files. We perform transformations, including various aggregations and grouping of data. Sometimes, we also perform data cleansing, refining the data in a proper way so that customers can generate reports against it. For different models, we transform the data and perform a bulk load or an ETL incremental load into the rest of the data warehouse. We have also done some tie-outs, like comparing two different datasets, ensuring they tie out as expected, and creating exception reports. We generate mismatches in the data and perform an unload process to create exception reports out of Glue. This process involves creating a report against the table if there is a data mismatch. We extract the data and put it into an S3 dump for TPUs to use for their exceptions. If a particular data file has an issue, we identify the problem and address it to improve the quality of the data.

For ETL, I feel the performance is excellent. If I create jobs in a standard way, the performance is great, and maintenance is also seamless. It is serverless, so I do not need to worry about other integrations. It runs on a serverless VM. Glue services from AWS are regularly upgraded with new properties and functions. We use PySpark and other frameworks, which help customize our ETL process in a standard way. During our process, we utilized the typical SQL ITIL process standard in Glue-oriented ETL services, which helped us improve performance and time. AWS running time, mostly during the morning, saved us time compared to the old ETL method. There are many email notifications once a job is successful or fails, and we get notifications using SES or AWS's Simple Notification Service. Glue is integrated with both email alerts, and we are responsible for managing the full data journey from end to end.

My use case is the usual: data ingestion, curation, and transformation.

I think if I'm working with big data, common languages like Python work quite nicely, which is advantageous. Despite this, it's had a limited impact just because of our particular circumstances.

I use the solution in my company for building datalake and for a variety of data sources like Oracle, MongoDB, and other multiple data sources, like SQL server, and AWS S3 buckets as a datalake storage tool, and then further we use AWS Glue to process it and move to AWS' search engine which will be like a lakehouse solution.

AWS Glue handles a huge volume of data, and it is a serverless tool. We don't need to put extra servers as long as the job runs or goes on with AWS Glue. Maintaining other options like AWS EMR can be very costly when handling the company's functions. AWS Glue creates a data catalog.

I use AWS Glue for transforming data quickly, especially when dealing with small amounts of data that need to be processed swiftly. We use Glue to perform calculations and transformations required from source systems and then store the processed data in databases such as RDS or Redshift. We also use it because of its seamless integration with Apache Airflow.

AWS Glue offers a cost-effective and efficient solution for running longer sessions beyond the 15-minute restriction of AWS Lambda. This helps in processing data quickly and seamlessly.

We have a lot of microservices written in Glue, which are responsible for triggering based on certain events. The solution will be responsible for another container to containerize them and run over the cloud. We use the solution for different purposes, including data computing.

You do not need many frameworks to run Glue. It's an interpreted language that does not require to be compiled at all.

I had the source data, which was unstructured and non-fixable, and my responsibility was to convert it into structured data. For this task, I used PySpark as the programming language. With Python, I implemented the creation of a data frame using Glue jobs. Since Glue jobs are a serverless mechanism, I deployed my code into the Glue job, and that's how I got the job done.

We no longer had to worry much about infrastructure management because AWS Glue is serverless, and Amazon takes care of the underlying infrastructure. This allowed us to focus on the code and application logic without concerns about scaling, CPU management, or handling fluctuations in flow. The serverless nature of Glue jobs relieved us from these infrastructure-related worries.

I use AWS Glue mostly for data ingestion or data extraction from multiple sources.

AWS Glue plays a central role in AI-based solutions and machine learning workflows by efficiently handling and moving data around, contributing to streamlined operations.