Amazon EMR simplifies big data processing by offering integration with popular tools. It's scalable and cost-efficient, enabling fast processing while managing infrastructure effortlessly. It's designed for users aiming to streamline data workflows and leverage its batch processing capabilities effectively.
| Product | Mindshare (%) |
|---|---|
| Amazon EMR | 9.7% |
| Cloudera Distribution for Hadoop | 14.4% |
| Apache Spark | 14.1% |
| Other | 61.8% |
| Company Size | Count |
|---|---|
| Small Business | 6 |
| Midsize Enterprise | 4 |
| Large Enterprise | 10 |
| Company Size | Count |
|---|---|
| Small Business | 90 |
| Midsize Enterprise | 25 |
| Large Enterprise | 95 |
Amazon EMR is a managed service that provides robust features for big data processing. It integrates seamlessly with S3, EC2, Hive, and Spark to facilitate sophisticated data transformation tasks and infrastructure management. It allows organizations to run data lakes, Spark, and Hadoop clusters effortlessly, offering flexibility with on-demand execution and extensive scalability. The platform is valued for its strong processing speed and comprehensive security features, making it ideal for complex data engineering projects. It supports both batch processing and real-time workflows, designed to eliminate hardware management while maintaining cost efficiency and stability.
What are the key features of Amazon EMR?Amazon EMR is implemented by industries such as healthcare and tech processing for complex data tasks like building data lakes or financial data processing. It supports AI-driven analytics and data engineering projects, integrating with SageMaker for predictions and maintaining workflows in public health applications, allowing professionals in different fields to manage data pipelines, resource utilization, and job execution efficiently.
Amazon EMR was previously known as Amazon Elastic MapReduce.
| Author info | Rating | Review Summary |
|---|---|---|
| Senior Chief Engineer (Enterprise System Presales/Postsales) at a tech vendor with 10,001+ employees | 3.0 | I've used Amazon EMR mainly for Spark-based ETL, but we've shifted 95% to EKS due to better cost control. EMR is simple and integrates well but lacks detailed UI and is costlier compared to pod-level control on EKS. |
| Senior Technical Engineer at a transportation company with 5,001-10,000 employees | 4.0 | I use Amazon EMR for data processing and ETL, finding its customizable features valuable, though it could improve with AI/ML capabilities. It's stable, integrates well with AWS services, and support is excellent, but setup could be simpler. |
| Lead AWS Data Engineer at Fission Labs | 5.0 | I have used Amazon EMR for data engineering projects, focusing on job submission and resource management. Its cluster properties and integration with big data tools are valuable, though improvements are needed in retries, data handling, scalability, and integration. |
| Vice President -Product Management at Paytm | 4.5 | I use Amazon EMR to help American Express process and analyze data. Its multiple connectors and cost-effective processing are valuable, though Spark jobs are slower. The service integrates well with other storage solutions, potentially saving 20% in costs. |
| Director of Data Science at HealthWorks Analytics | 3.5 | I find Amazon EMR valuable for maintaining client workflows in the healthcare sector due to its security features as a managed service. It offers a high ROI but can be expensive if not carefully managed on AWS. |
| Senior Software Development Engineer at Yahoo! | 3.0 | I use EMR with Spark for large cloud jobs. While stable and scalable, its very slow start-up time is a significant problem for frequent, short tasks, impacting costs. Support is also slower than desired. I'd rate it a 6/10. |
| Data Governance Manager at VPBFC | 4.5 | We use Amazon EMR for data processing, reading data from S3 and writing it back there post-processing. Its affordable pricing and seamless integration with open-source platforms are valuable, though its stability could improve. We haven't considered alternate solutions. |
| Lead Data Engineer at Seven Lakes Enterprises, Inc. | 3.0 | I use Amazon EMR to efficiently analyze data and process millions of records within hours. It offers diverse tools and support, simplifying development both on-prem and in the cloud. Improvement is needed in third-party integration and advance notification of module changes. |
| AWS / Big Data Engineer at Waste Management, Inc. | 4.0 | I found Amazon EMR Serverless ideal for deploying online applications, as it simplifies resource management. While pricing could be improved, the solution offers better ROI and substantial savings. We migrated from a server-based setup to AWS for this service. |
| Development Engineer at Signify | 4.0 | We use Amazon EMR primarily for AWS services in B2B project management, appreciating its streamlined features. However, we've faced compatibility issues with Boolean operations in older versions. We haven't calculated our ROI, and no alternate solutions were considered. |
We are using Amazon EMR for Spark as our platform involves a lot of ETL. We are using Amazon EMR for that, but in the last year, we have moved to Kubernetes, AWS EKS, where we are running Spark. Our Amazon EMR usage has been reduced as we have moved to the EKS side.
We still have some of our pipelines running on Amazon EMR, but 95% of the pipelines are moved to the EKS.
Amazon EMR provides out-of-the-box functionality because we can deploy and get Spark functionality over Hadoop. Another feature is on-demand pipeline execution. Whenever Amazon EMR is required, we can provision it with no need to keep it open for long.
EC2 is itself given by Amazon EMR. Amazon EMR provisions worker nodes and master nodes. With other services, we have found it satisfactory for our use case. With S3, we were able to integrate efficiently.
Amazon EMR provides out-of-the-box solutions with Spark and Hive. Regarding customization, I am not from the data team, so I do not have much information about this, but we have not faced many problems with customization. We moved to EKS considering the cost factor because Amazon EMR provisions full EC2. When we use EKS, where apart from the pipelines our whole infrastructure exists, though it also gives us EC2, we can utilize at the pod level. The compute demand has decreased and it has reduced the cost significantly. Moving to EKS has helped us considerably.
Amazon EMR is simple.
I have used AWS Glue with S3 for making tables and databases, but regarding Amazon EMR, I do not remember much as we are currently using it very minimally.
This is my observation: In EKS, we have had to deploy by ourselves because EKS does not provide the Hadoop framework, Spark, Hive, and everything, but we have completed all the deployment ourselves. Whereas Amazon EMR provides all these things. The cost factor differs significantly. When you run Spark application on EKS, you run at the pod level, so you can control the compute cost. But in Amazon EMR, when you have to run one application, you have to launch the entire EC2.
In Qubole, the interface was very good. I could see many details because in Amazon EMR console, very few details are available. In Qubole, at one link, you can get all the details of what is happening, how the processes are running, and the cost decreased by using Qubole. I found Qubole more user-friendly and cost-effective. From the security point of view, we had to open some access rights to Qubole, which might be a drawback in comparison to Amazon EMR which is native to AWS.
I have been working with Amazon EMR for the last four years, though not very extensively.
We have enterprise tech support with very good technical assistance. We get all call support, screen sharing support, and immediate support, so there are no problems.
Neutral
I am working on Amazon EMR but not extensively.
Basically, our work is data transformation. Our pipelines work on that exclusively. We have Spark applications, and earlier, we used Amazon EMR extensively which did all this ETL.
There is no problem in security because it is part of the AWS ecosystem. All the IAM and everything works properly. Security-wise, we have never faced any issues. Nobody has reported any inability to perform certain tasks in Amazon EMR.
I would rate Amazon EMR a 6 out of 10. The lower rating is due to potential difficulties for less technical users in accessing Spark commands and notebooks, and the cost factor.
The features at Amazon EMR that I have found most valuable are fully customizable functions.
I am using Amazon EMR for data transformation, where we load the data. The entire ETL process is encompassed within it. The first thing is the egress part, where we pull the data from various sources, which include our main core databases, the systems that include Oracle, MySQL, SQL Server, and Postgres. We bring data from all these sources and use Amazon EMR specifically to combine all this data, clean the data, and combine it at a particular level. We bring everything to a particular level to make it more production-ready, and we simplify the data because every system has its own problems. We are using it to clean the data and transform the data in such a way that the end-user can get the insights faster.
I feel some lack of functionality in Amazon EMR. I have thoughts on what would be great to see in the product, such as AI/ML features or additional options.
I have been working with Amazon EMR for five years.
I would rate the stability of Amazon EMR as eight out of ten.
Customizable cluster configurations have been utilized in my organization.
I would rate the technical support from Amazon as ten out of ten. They are always there to help.
Positive
The setup process for Amazon EMR may have some issues and could be more simplified for some basic users.
I would rate the price for Amazon EMR, where one is high and ten is low, as a good one.
I could compare Amazon EMR with a product from another vendor, but I would need more context.
I find it easy to integrate Amazon EMR with other AWS services like S3 or EC2 for data processing needs. I would rate this review as eight out of ten.
Neutral
I am a solution provider using Amazon EMR. Our main customer, American Express, pulls data from multiple sources, processes it, and provides analytics reporting. Amazon EMR supports AI-driven projects by allowing on-the-fly execution of algorithms to segregate data based on character properties.
Amazon EMR has multiple connectors that can connect to various data sources. The service charges are based on processing only, depending on the resources used, which can help save money. It is easy to integrate with other services for storage, allowing data to be shifted to cheaper storage based on usage.
Spark jobs take longer on Amazon EMR compared to previous experiences. This aspect could be improved to make them more efficient.
I have been using Amazon EMR for five years.
Amazon EMR is quite stable now.
Based on requirements, Amazon EMR scales easily and works well for enterprise clients.
Customer support is quite helpful, with acceptable response times.
Positive
The setup of Amazon EMR is straightforward, and I would rate it as eight out of ten for ease.
The tool is said to help save costs by up to 20%.
The cost of Amazon EMR is a little bit expensive, especially considering the support package, which includes a gold package.
Amazon EMR is easy to implement, offers features beneficial for scaling, and provides useful metrics.
I recommend using it, and my overall rating is nine out of ten. It offers better features that help in scaling and provides useful metrics through its console.
We use the tool to maintain workflows on behalf of our customers. We use it in the medical field, particularly in public health and government healthcare. However, our primary role is maintaining and developing the solution for our clients rather than using it internally.
The security of the managed workflow and the managed services are the best features for us. Since we inherited their security model and it's all managed services, those are the key benefits for our clients.
The solution can become expensive if you are not careful.
I have been using the product for three years.
I rate the solution's stability an eight point five out of ten.
The tool is as scalable as you need it to be. We're not in the petabyte range yet, but we've scaled it up by over a hundred with a couple of customers. In terms of data, we're looking at hundreds of gigabytes to hundreds of terabytes. So, it's been highly scalable.
Amazon EMR's deployment ease is five out of ten. I rate it so because of our lack of understanding and training. It was not difficult. The timeline depends on our client. We typically schedule about a six-week deployment. We've done it in as little as two weeks, but six weeks is probably the average for getting things up to speed.
You would need four to six resources to handle it. We have solution architects, security SMEs, and business process engineers. In the background, we also have developers. The main roles are solutions architect, security architect, and business process engineer.
We have a staff of about nine people, but that's spread across several clients. We probably have a couple of people working part-time for our smallest client. None of our staff are dedicated full-time to any single client. We maintain and develop the solutions with these nine staff members, occasionally calling in extra help as needed.
Based on customer feedback, the return on investment has been high, especially for those who have used on-premise systems previously. They're likely seeing an ROI of at least two to one on their investment, sometimes even more.
I rate the tool's pricing a five out of ten. It can be expensive since it's a managed service, and if you are not careful, you can run into unexpected charges. You can make a mistake that costs you tens of thousands of dollars. That's happened to us twice, so I'm sensitive to it. We're still trying to work on that. Our smallest client probably spends a hundred thousand dollars yearly on licensing, while our largest is well over a million.
I'm very impressed. It's certainly a contender. I don't keep up much with the current competition, but we've enjoyed the experience with Amazon, and they've been very supportive. So we get a lot of attention from them, which biases me toward a high recommendation for Amazon EMR. I rate it a seven out of ten.
We're focusing on AI integration now, especially on Amazon. It was complicated, but they're doing a great job, especially with Sagemaker. We've been building large learning models, and the most significant thing for us has been applying these models to proprietary data on Amazon. They're focusing on that aspect.
We usually use EMR with Spark and bring in Airflow or something. For example, when we grade big jobs from on-prem to the cloud, we do it in EMR with Spark.
In the main application, we use Hive, Spark, and Flink, and there are some additional small tools. We have to do this job inside EMR itself and run from Airflow. We also use a service from Amazon, and we manage jobs through Airflow. There's lots of scheduling, et cetera, that happens. Before, we used Jenkins. Now, we also integrate from Jenkins to Airflow and its processes. It's easier to have everything connected via Amazon.
The problem for us is it starts very slow. They need to improve the start time. If we use a long-running EMR, it costs a lot of money. However, when we start, for example, a job, if the job runs for one hour, it's normal as it starts in about ten minutes. If we want, for example, to run each five minutes, it's a problem if it takes ten minutes to start. It's a little bit weird that you cannot use the service within a short period.
The support could be better.
We've been using the solution for about one year.
We've had some issues here and there, however, for the most part, the solution is stable.
The solution can scale quite well. It's not a problem.
A lot of teams use the solution. We have two or three teams that use it.
Technical support could be faster. They don't respond as quickly as we would like.
We used an EC2 service. We did manually manage it via our engineers. We also used Knime and looked into Kinesis.
You don't need to deploy anything. You just start. You go to the computer and start using EMR.
Sometimes we found some strange things. We've had some exceptions. Various versions can be used, however.
I'm not sure about to cost of the licensing. However, it does depend on the integration, for example, and many other factors.
I'd recommend the solution to others to try.
I'd rate it a six out of ten.
We use Amazon EMR for data processing. We read the data from S3 and then write the processed data back to S3. It allows users to access the data through a web interface.
The product has affordable pricing. We can integrate it easily with multiple open-source platforms.
The product's stability could be even better.
The product’s stability is good. However, it could be improved a bit. I rate it nine out of ten.
Amazon EMR can scale to a large cluster. In our case, there are 40 EC2 instances. We are processing a lot of tables with half a billion rows per day.
It is designed for end users and pivot users who need to detect data patterns and generate reports. EMR is not a fixed-scale service, but it can be customized to meet the needs of any specific use case.
We need to have a data pipeline tool to ensure consistent data processing for the initial setup. We create a framework, read the code, and execute it in a data catalog. The size of the maintenance team depends on the project and the use cases. Usually, one backup team of four or five DevOps executives takes care of the backend and database.
We need to separate our environments into production and development. We use GitHub for source control, Jenkins for the deployment pipeline, and a standard CI/CD tool to deploy code changes into production.
We need to develop a deployment framework so developers only need to provide the code for their projects. The underlying engine then deploys the code, reads it, addresses the EMR filter, executes it, and completes the data processing.
You only pay for the EC2 fees for Amazon EMR. The number of clusters you launch and the time you use them determine the fee. There is no additional license fee, as Amazon EMR is a fully managed service. You can use any open-source software that is compatible with it. There is no need to pay extra for third-party software.
EMR is a good engine for specific use cases. It is mostly used for data processing, but it does have some real-time streaming capabilities. If you want to stream hundreds of millions of events daily from mobile apps and web apps, then an in-memory technology would be better than EMR.
I rate Amazon EMR a nine out of ten.
EMR is used to analyze data for projects where we wish to ingest multiple data sources into our analysts' data lakes. EMR is further used to process millions of roles within a fixed span of hours.
It gives us multiple options from Infra to the WES and different tools. It has a variety of options and support systems. Plus, it makes life easy for our developers, whether in the cloud or on-prem. Related DevOS installations are quick and easy to maintain.
Interdependencies with a third-party or open source solution should be improved. Modules and strategies should be better handled and notified early in advance. Maybe if AWS starts releasing AWS-certified or AWS-verified installations, that will generate even more confidence just like OpenJet, it'll add a specific version.
The solution is mostly stable with certain data and code related issues.
Amazon EMR is a scalable solution. We have around twenty users, but they are engineering users. Outside, we have, our analytics product, which is being used for most of our clients. We have plans of upgrading the usage of the solution.
The technical support team is good. The team is thorough with their knowledge and are quick to respond.
The initial setup of Amazon EMR is straightforward if you know the basics and have knowledge of big data and architecture. The AWS documentation also helps with the deployment. Initially, the deployment takes a few hours and then it becomes easy.
The maintenance depends on your application and specific requirements, but for deployment, you don't need a much bigger team, probably a team of one cloud guy should be enough.
The deployment can be done in-house.
There is a small fee for the EMR system, but the major cost components are the underlying infrastructure resources that we actually use.
My advice would be to do a dependency analysis to understand the limitations before planning to move in with Amazon EMR. I would rate the overall solution a six out of ten.
EMR Serverless is useful for online deployments that require a serverless architecture. We were previously using a server-based architecture, but we have since switched to serverless.
The product is very well-designed. It is easy to use, and it is very cost-effective.
The solution eliminates the need to manage hardware resources. Everything is managed for you, so it is a very easy-to-use service.
There is room for improvement in pricing.
It is stable. It just seems to be stable right now. We are monitoring it closely again, so it is stable right now.
It is highly scalable. There are around 2,000 end users using this solution in our organization.
The customer service and support are great.
Positive
The initial setup is straightforward.
The deployment process took around two months. There's nothing much to do. It's a very easy process. We just need to configure all of them for the service, and it will be done.
For maintenance of the solution, we require managers, savings, engineers, developers
The ROI is better. It's giving a good amount of savings to the company.
It is expensive.
You should use it. It will be really helpful in the business. It will really help the business. So it is highly recommended to use it.
Overall, I would rate the solution an eight out of ten.
We primarily use the solution for AWS services. For example, in a typical B2B project management, some coders are sequencing at different speeds and managing the overall ports from other coders. We deploy the solution on cloud.
We value the streamlined solution, and the project management is very streamlined using almost all the features.
We have had issues with the boolean mathematical operation in 2. X's big version is working in newer versions because the old version of the solution does not support it, which is a compatibility issue that can be improved. In addition, the legacy versions of the solution are not supported in the new versions.
We have been using this solution for six months.
I rate the scalability a seven out of ten, and approximately seven amazon users who are architects, project managers, and testers use the solution in our organization. We intend to gradually increase the usage in our organization by at least five to ten percent annually.
Positive
The initial setup is straightforward, and deployment takes between 15 and 20 days.
We implemented the solution through a vendor team.
We have not calculated our ROI.
The licensing costs are expensive. I rate them a seven out of ten, with one being the least expensive and ten being the most expensive.
I rate the solution an eight out of ten. The solution is good, but its compatibility with older versions can be improved. I recommend the solution to users considering implementing it in their organizations.
