AWS Glue Reviews

The first time I used AWS Glue was in my first project where I automated large-scale ETL pipelines using AWS Glue and Python. I also used AWS Glue for data cataloging and transformation workflows.

Since then, I consistently use AWS Glue in other projects for ETL orchestration, especially in AWS-based environments, and it is one of my go-to tools for data integration.

AWS Glue has reduced efforts by 60%, which is the main benefit. Using AWS Glue allows for automatic scaling and reduces operational time.

Regarding data catalog integration, the Glue Data Catalog helps manage metadata across various services. Because I work heavily with PySpark, and AWS Glue supports it, it is very easy for me to build and debug pipelines.

For workflow orchestration, it is easy to define triggers, dependencies, and workflows within AWS Glue. Overall, AWS Glue is very efficient and integrates well with the AWS ecosystem.

AWS Glue is efficient, but a common issue I face is that when the Glue jobs run, especially compared to other orchestration tools, the job logs can be difficult to trace when troubleshooting complex PySpark transformations.

The Glue jobs mainly lack robust version controls for job and script integration directly into the console.

When comparing to tools such as Airflow, Glue workflows are still relatively basic in terms of flexibility and complex branching. AWS has improved over time, but it still needs improvement compared to other tools.

I have more than five plus years of experience.

For the initial setup with AWS Glue, I find it easy to set up the data catalog and create Glue jobs using the visual editor or the visual code. Setting permission sets via IAM rules can be a bit tricky at the start, but we ensure Glue has access to AWS S3, Redshift, and other services.

Once the role is configured, it runs smoothly. For advanced configurations, connecting to VPCs and setting up connections with JDBC sources takes more time compared to my cloud experience, but overall, for someone with cloud and ETL experience, the setup is manageable and well done.

My experience with the support team from Amazon has been positive, as technical issues are usually addressed very quickly, and the support engineers provide detailed guidance and follow-ups.

For complex Glue-related problems such as job failures or permission issues, their documentation is good, but having direct access to support helps cut down troubleshooting time significantly.

In my experience, they have been proactive in suggesting best practices and optimizations when contacted. While the basic support types and response times can be slower, overall, I have had a positive experience with AWS support.

I would rate AWS support eight out of ten because they are technically strong and helpful in debugging complex Glue and cloud issues, and they are very responsive.

Positive

Regarding AWS Glue's pricing, it is not more expensive; rather, it is very reasonable, but it is not cheap.

I would recommend AWS Glue to other people.

AWS Glue is very user-friendly and allows for full management of ETL pipelines, which includes extract, transform, load, and helps prepare and combine large amounts of data for analytics, machine learning, and application development.

It automates heavy lifting ETL jobs, including schema discovery, code generation, and job scheduling. I recommend AWS Glue because it is serverless, requires no infrastructure management, is very scalable and flexible, and can handle small to large datasets.

It integrates easily with AWS services, making it simple to build data pipelines, and it supports both Python and Scala for ETL scripts.

Overall rating: 8 out of 10.

I am using this solution for ETL programming, which involves extraction, transformation, and loading. We have a data warehouse, Redshift. We receive some data files from outside, save those files into S3, and then call our Glue jobs to process those files. We perform transformations, including various aggregations and grouping of data. Sometimes, we also perform data cleansing, refining the data in a proper way so that customers can generate reports against it. For different models, we transform the data and perform a bulk load or an ETL incremental load into the rest of the data warehouse. We have also done some tie-outs, like comparing two different datasets, ensuring they tie out as expected, and creating exception reports. We generate mismatches in the data and perform an unload process to create exception reports out of Glue. This process involves creating a report against the table if there is a data mismatch. We extract the data and put it into an S3 dump for TPUs to use for their exceptions. If a particular data file has an issue, we identify the problem and address it to improve the quality of the data.

For ETL, I feel the performance is excellent. If I create jobs in a standard way, the performance is great, and maintenance is also seamless. It is serverless, so I do not need to worry about other integrations. It runs on a serverless VM. Glue services from AWS are regularly upgraded with new properties and functions. We use PySpark and other frameworks, which help customize our ETL process in a standard way. During our process, we utilized the typical SQL ITIL process standard in Glue-oriented ETL services, which helped us improve performance and time. AWS running time, mostly during the morning, saved us time compared to the old ETL method. There are many email notifications once a job is successful or fails, and we get notifications using SES or AWS's Simple Notification Service. Glue is integrated with both email alerts, and we are responsible for managing the full data journey from end to end.

Improvements in the UI are needed, as it is challenging to understand some functionalities. Glue is quite customizable, but technical issues, particularly with internal errors, can pose challenges. New version upgrades can also be problematic. For example, migrating jobs from version 3.0 to 4.0 can present compatibility issues. Changes might be needed to ensure the job fits all versions. Despite these challenges, upgrading our systems enhances performance. Learning the latest functionalities is crucial, and while challenging, it is a vital part of staying current and ensuring an efficient ETL process.

I have used it for the past four years.

Scalability is rated eight out of ten. It is beneficial to upgrade jobs, and we conduct extensive testing in development before migrating to production. For jobs requiring multiple RAM usage, we increase the number of workers accordingly. Security groups are used to support scalability, which is rated eight out of ten.

We engage with AWS support for insights on the latest upgrades. Upgrades occur every four months, and new developments coincide with version updates. For instance, while working on version 4.0, we encountered issues extracting data from Oracle RDS into DynamoDB. Initially, the denormalization process worked well, but issues arose after upgrading to 5.0. AWS support provided assistance in mapping fields correctly. Despite challenges, AWS's documentation is reliable, and careful reference often resolves missed upgrade details.

Neutral

We use AWS Data Pipeline and CodeCommit to integrate Glue jobs. Development occurs in the dev environment, and changes are migrated to production using pipelines. This process involves coordinating with a DevOps team for pipeline integration. Glue jobs read Python files from S3 buckets, and CodeCommit is used for code repositories. Lambda functions and code formation templates are used for resource creation. The SES service is used for notifications. The setup requires development work, but it enhances overall process reliability.

Pricing is medium. Costing depends on resource usage, and cost optimization may involve redesigning jobs for flexibility. AWS provides cost optimization tools to estimate Glue job costs. The pricing model is usage-based, allowing for flexible costs.

Azure has Data Factory, and Informatica is another competitor for ETL solutions. Cloud-based ETL solutions, like Glue, are preferred over Azure Data Factory due to continuous improvements. While lacking Azure experience, Glue is the preferred choice based on my personal experience.

I recommend using Glue Studio for convenient and comfortable utilization. It is a very good ETL solution, especially for medium-level applications. Cloud solutions like Glue make management easy. I rate the solution an eight out of ten.

My use case is the usual: data ingestion, curation, and transformation.

I think if I'm working with big data, common languages like Python work quite nicely, which is advantageous. Despite this, it's had a limited impact just because of our particular circumstances.

I actually don't like it. It is good, however, I find it quite clunky and code-heavy, which is my biggest problem. I am quite technical, yet when it comes to data pipelines, I prefer using GUIs over code-based calls. It might be easier with a different skill set. 

I am using an ETL tool that allows writing without needing dedicated data engineers, however, it's a bit more technical and requires some skills. It works well for big data, yet for typical financial services, other data pipelines perform better. Glue can be said to be more for techies. 

They have introduced a GUI for drag-and-drop, and it's still primitive, in my view. With AWS, I gather data from multiple sources, clean it up, normalize it, de-duplicate it, and make it presentable. Glue is mainly used for this purpose. I have worked with it a few times, but it's not my tool of choice.

I have used it for probably two to three years. It is also quite expensive.

It is pretty stable and very functional. I would give it an eight out of ten for stability.

AWS is highly scalable.

AWS support is excellent.

When promoting things to production, it is a serverless offering, so there's nothing to deploy as such.

AWS is worthwhile if I have a limited number of pipelines and do not justify getting an off-the-shelf tool. There is overhead involved in getting an ETL tool installed and running. If I have a limited number of pipelines, I may not want to bother setting up something else. The configuration takes time and costs money. I advocate using Glue in such cases.

There are no licensing costs, however, it is still quite expensive. AWS charges based on runtime, which can be quite pricey. I would probably give it a four out of ten in terms of affordability.

Primarily, I considered Matillion.

The solution is expensive and can easily incur charges if endpoints are left running accidentally. I do not find it very user-friendly, and still it has its place. 

For limited pipelines, Glue on AWS, being serverless, is justifiable. However, with running hundreds or thousands of pipelines, I avoid coding it. For very big data loads, I would use EMR instead. Glue was initially used because it could be launched directly from the AWS console without startup costs or configuration. It has its place, but not in my space. Overall, I would rate the product five out of ten.

I use the solution in my company for building datalake and for a variety of data sources like Oracle, MongoDB, and other multiple data sources, like SQL server, and AWS S3 buckets as a datalake storage tool, and then further we use AWS Glue to process it and move to AWS' search engine which will be like a lakehouse solution.

AWS Glue handles a huge volume of data, and it is a serverless tool. We don't need to put extra servers as long as the job runs or goes on with AWS Glue. Maintaining other options like AWS EMR can be very costly when handling the company's functions. AWS Glue creates a data catalog.

The drawbacks associated with the product stem from the fact that, based on the data volume, it can become very costly. There is a huge cost if the source system is not properly designed. If the changes are frequent and not valid, then, initially, you will use huge amounts of data in the ETL. The biggest challenges are associated with AWS Glue's costs, and it takes one-third of my entire pipeline cost.

I have been using AWS Glue for two years. I am a customer of the tool.

It is a very scalable solution. The problem with the tool is that it has a huge set of options, so there are a lot of hidden costs involved. Suppose you have to get into data quality, the billing needs to be done for the data quality part. As per what I learned, there is a need to scrape off AWS Glue and look for other solutions.

We use Amazon's services to provide technical support for the product. If you want to have support, Oracle and others offer a single support, and other tools have a direct support window. For Amazon, we need to pay 10 percent of my billing amount for the tool to get support services. Whether to raise a support ticket or not is an issue since ten percent is a huge amount. My company ends up using all the options without help from support. It is very difficult for any common man to understand why there is a need to pay ten percent for support. If I find an issue in the product, and I need to get support from AWS to fix it, then I need to pay ten percent of the tool's bill amount to Amazon. AWS is a very tricky tool because everything is evolving nowadays. AWS engineers are getting hired from other places, and even after that, if I am not getting any technical support, then things will be very nasty. There are some good engineers who help users outside the normal support cycle, but it doesn't meet their needs.

I rate the technical support a four out of ten.

Neutral

The product's installation phase is easy, especially since it is serverless.

The solution can be deployed in a day or two since it is serverless. AWS Glue alone doesn't work perfectly. If we have the right data model, then we can use AWS Glue. Inside AWS Glue, we use PySpark.

My company is in the mode of scraping off AWS Glue. My company is not approving the budget for us to use AWS Glue. I am trying to see some solutions that are not costly, like the ClickHouse database, which is an open-source tool.

The costs of the tool are huge, especially when moving from the source to the datalake.

I rate the tool an eight on a scale of one to ten, where one is expensive, and ten is expensive. I cannot have any predictability factor regarding the costs associated with the tool.

The main piece of AWS Glue is the ETL part. AWS Glue is for ETL to deal with S3 data sources to Redshift. We use AWS Glue for the CDC.

As the product is serverless, the tool runs fine. Most of the maintenance and monitoring are among the biggest challenges of the tool.

In terms of the product's ability to handle data volumes during scaling needs, I would say that though it offers the area of data volumes, the challenges are associated with costs.

The latency would be there if the source had a huge amount of data coming in, and so based on it, it would read the source system sequentially because of the way the CDC works. If I need to capture the change in the source change in the order, it can happen, and if you have a better network, you can also scale up by bringing the source to S3 or AWS Glue. When you can scale up, it is not really relevant for the group. The latency is not because of AWS Glue but because when it comes to ETL or CDC, I need to process it the same way I do it with AWS Glue. I cannot do parallel processing, and I need to do it sequentially.

I don't see any AI capabilities in the product, and it is more of an ETL solution.

As the product has many problems, people are moving to Bare Metal and other cloud services. Our company has spent a lot of time investigating what AWS Glue does, including the time required to use it to maintain the servers.

I need to spend on the product's maintenance along with the other activities for which I need to make payments to use the solution. Once you are able to predict the data volumes and other factors that are there over the cloud, it is possible to predict what my server will cost for the next five years and then get the servers at a very low price instead of depending on AWS.

Though it is a good solution, it is not cost-sensitive.

I rate the tool a six out of ten.

I use AWS Glue for data processing. Some of my colleagues have data for software, and I use AWS Glue to transform and inspect this data.

It's very good to manage. It is easy to integrate other products with AWS. 

Glue integrates with other AWS processes and networks. So, it's quite easy to integrate.

I've worked with AI integration but I haven't gone into much depth on that topic.

AWS Glue's error handling is difficult. 

The errors in AWS are very hard to handle. The screen is very hard to understand. 

I have to use CloudWatch, but whatever our error was, the new ones, and so on. I would test this with someone. It's not so easy for me, and there are more things related to this.

I have been using it for a year and a half. 

I would rate the stability a nine out of ten. 

I would rate the scalability a seven out of ten. 

My data is small, so we need to consider more days. We need to deal with what we have, but I understand the documentation. 

Some people find it hard, but I rated it a seven. In my company, TechOps uses AWS with about 1,200 users.

I worked with Databricks. In my opinion, Databricks is improving and is easier to use. It's more user-friendly, and I think it's better overall.

I work with a big company, and most of it is already quickly done, like using something that is a blueprint. This configuration stuff is already working in another place. The only thing I have to do with the cloud is the remote configuration.

AWS can be expensive.

Overall, I would rate it a seven out of ten. I would recommend it.

I use AWS Glue mostly for data ingestion or data extraction from multiple sources.

AWS Glue plays a central role in AI-based solutions and machine learning workflows by efficiently handling and moving data around, contributing to streamlined operations.

One aspect that I would like to highlight is the Glue Crawler, which we utilize when working with large datasets to ensure the schema updates seamlessly without requiring end-team knowledge. 

Additionally, the pipeline orchestration and scheduling of ETL pipelines in AWS Glue are also highly effective. AWS Glue supports AI-driven projects and DAG transformations by facilitating efficient data handling required for machine learning workflows.

The point for improvement in AWS Glue would be the dynamic allocation of resources while utilizing Lambda functions. Currently, Lambda functions encounter a time-out error after fifteen minutes, necessitating an improvement in this area. Moreover, more practical examples and resources for advanced features would be beneficial for users.

With AWS Glue, I have one year of experience.

In terms of stability, there are areas that could be improved, particularly with Lambda functions and step functions. There should be better scaling for parallel computation. Despite this, AWS's overall stability is quite good.

AWS Glue receives a nine out of ten for scalability, although we feel certain functions should scale better for enhanced parallel computation.

I would rate the technical support for AWS Glue as ten out of ten. The support is high-quality.

Positive

The initial setup of AWS Glue could be straightforward for new users of AWS services, but there should be more practical examples and guides for advanced features to assist newcomers in learning complex concepts.

Currently, I work on the implementation within our development team, focusing on cloud services.

I find the pricing for AWS Glue quite affordable. For students or new users, AWS offers free credits, and as usage increases, the pay-as-you-go model provides flexibility without being expensive.

I would recommend AWS Glue to others and rate the overall solution nine out of ten.

I use AWS Glue for transforming data quickly, especially when dealing with small amounts of data that need to be processed swiftly. We use Glue to perform calculations and transformations required from source systems and then store the processed data in databases such as RDS or Redshift. We also use it because of its seamless integration with Apache Airflow.

AWS Glue offers a cost-effective and efficient solution for running longer sessions beyond the 15-minute restriction of AWS Lambda. This helps in processing data quickly and seamlessly.

AWS Glue is very quick to start without cold starts, unlike AWS Lambda. It performs calculations and transformations efficiently, making it a preferred choice for data processing. Additionally, its integration with Apache Airflow enhances orchestration capabilities.

The process of entering environment variables in AWS Glue requires navigating to a different page, which could be streamlined. Providing an easy-to-use interface for variable management within Glue itself would enhance the user experience.

I've been using AWS Glue for a year.

In my past two years of experience, AWS Glue has been stable without any downtime.

AWS Glue scales effectively to handle varying data loads, from megabytes to gigabytes, and can handle records in both thousands and millions. I rate its scalability nine out of ten.

The technical support is responsive and effective, providing solutions quickly. They are supportive and involve senior teams if needed, ensuring issues are resolved efficiently.

Positive

We initially used AWS Lambda, but its 15-minute runtime restriction led us to switch to AWS Glue for longer sessions.

The initial setup of AWS Glue is straightforward and easy via the graphical user interface within the AWS Management Console.

AWS Glue is slightly more expensive than AWS Lambda. I rate its cost seven out of ten. However, the price is considered justified for the ease and service AWS Glue provides.

We considered AWS Lambda and Amazon EMR as alternate solutions for data processing.

New users should understand their requirements and evaluate the cost-effectiveness of using AWS Glue versus alternatives like EMR or Lambda. Starting small and contemplating cost impacts and available support are also recommended steps.

I'd rate the solution eight out of ten.

We use AWS Glue for handling data-intensive tasks such as data lake creation, log analysis, machine learning pipelines, data warehouse population for analytics, and real-time data integration with AWS Lambda.

AWS Glue has increased efficiency and time saving. It simplifies and automates data pipeline processes, enabling faster data processing and analysis.

The AWS Glue Data Catalog provides metadata management and schema discovery. AWS Glue simplifies data transformation with automatic schema detection, incremental data updates, and integration with other AWS services.

It enables us to analyze data stored in Amazon S3 using SQL, which is manageable and cost-effective.

AWS Glue should be more reliable and faster in processing. Enhancing the speed of data processing would be beneficial.

I have been using AWS Glue for more than one year.

AWS Glue is generally considered stable and reliable for data integration, especially for larger scale production environments. Its serverless architecture and integration with other AWS services contribute to its stability.

AWS Glue is scalable because of its serverless nature, which allows for easy scaling without needing to manage any infrastructure.

The technical support from AWS is very reliable. I would rate it nine out of ten.

Positive

I did not use any other cloud data integration solutions before AWS Glue.

The initial setup process for AWS Glue involved setting up Glue resources, creating roles and permissions, and developing ETL scripts and jobs. It took about half an hour.

The deployment process required three developers.

While specific data on time and cost savings was not provided, AWS Glue's benefits include increased efficiency and time-saving.

The approximate cost for ETL jobs is about 0.44 USD, which is mostly covered by the company. Employees do not purchase AWS Glue solutions individually.

AWS Glue is highly recommended for data engineers due to its ability to build and maintain data pipelines, ensure data quality and integrity, and its integration with UI tools. It offers data preparation, machine learning integration, and governance.

I would rate it a ten out of ten.

In my role as the global lead for AWS solutions and offerings, we work with various clients, including large-scale clients, to adopt and implement AWS cloud offerings. 

Our primary focus revolves around cloud lift-and-shift migration, modernization, re-platforming, rehosting, data architecture, design strategy, and implementing generative AI-specific solutions across different industries such as banking, capital insurance, energy utilities, manufacturing, automotive, semiconductor, and aerospace and defense. 

For example, we have implemented AWS Glue at several client locations, utilizing its serverless data integration capabilities during the data discovery process, enterprise transformation, cleansing, transforming, and centralizing data.

AWS Glue has significantly improved our data quality, enhancing the data by removing duplicates and providing timely and efficient insights. 

It also aids in real-time data processing, reducing effort and cost due to its serverless architecture. These features ensure we maintain the highest level of scalability, reliability, and security compliance.

AWS Glue is fully managed, providing an easy-to-use integration environment to create, run, and monitor ETL jobs. It's broadly compatible and seamlessly integrates with other AWS services like Amazon S3, Redshift, and Athena. It's flexible with data integration, manages various data formats (JSON, ORC, CSV, etc.), and is serverless, eliminating the need for infrastructure management.

Its ease of use, cost-effectiveness, and highly secure architecture are some of the most valuable features.

There could be an enhanced way of managing pure metadata management or data cataloging. 

Additionally, while it covers a wide range of integrations with AWS services, integrating with certain additional or legacy products is not seamless and can be complex. 

Increasing support for more programming languages and improving advanced analytics capabilities could also be beneficial.

We have been working with AWS Glue for almost three-plus years now.

We haven't faced any stability issues with AWS Glue. It is a scalable solution, provided that the right design principles and workload management are implemented.

AWS Glue is a scalable solution due to its serverless architecture and efficient design.

My team handles interactions with AWS for technical support, ensuring our design architectures are scalable, flexible, and well-integrated. We often reach out to the AWS team to double-check our implementation mechanisms and guidelines.

Positive

The initial setup of AWS Glue is straightforward due to its serverless architecture and fully managed nature. Specific prerequisites need to be followed, such as setting up data sources, configuring IAM permissions, creating crawlers, and running ETL jobs.

My team escalates technical questions to AWS support, ensuring our design architectures are optimal. We have a partnership with AWS, and the technical team frequently reaches out to AWS for guidance on scalability, flexibility, and integration mechanisms.

We have seen an efficient process with AWS Glue, providing the right return on investment at the right time. It ensures efficiency for our clients, giving them the desired ROI within their expected timelines.

Follow the right design principles and involve AWS at the right time to leverage the most current features and offerings from AWS Glue. Ensuring the right architecture will mitigate any issues. I'd rate the solution eight out of ten.