Spark SQL alternatives and competitors

A use case of this solution, for one of our clients with a large database of letters with addresses, is to predict if a person still lives at the listed address or if they have moved to another. We leverage EMR and SageMaker in AWS. 

EMR is cloud-based and managed through the cloud. 

One of the valuable features about this solution is that it's managed services, so it's pretty stable, and scalable as much as you wish. It has all the necessary distributions. With some additional work, it's also possible to change to a Spark version with the latest version of EMR. It also has Hudi, so we are leveraging Apache Hudi on EMR for change data capture, so then it comes out-of-the-box in EMR. 

Amazon EMR is continuously improving, but maybe something like CI/CD out-of-the-box or integration with Prometheus Grafana. 

I have been working with this solution for three years. 

This solution is pretty stable. 

It's managed services, so it's scalable as much as you wish. 

There are something like 40 to 50 people using EMR in my organization. 

We are an AWS Premier Partner, so we have all the necessary support and the ability to contact product teams. 

We didn't use any other products before implementing EMR. Some of our clients have Cloudera distributions, but we prefer EMR. 

The installation is straightforward because you can do it from the AWS Console or with Terraform. You can do it yourself. 

We implement this solution ourselves. On our team, we have admins, data engineers, DevOps engineers, and MLOps engineers. We have 40 or 50 data engineers. 

You don't need to pay for licensing on a yearly or monthly basis, you only pay for what you use, in terms of underlying instances. 

We have a range of clients in addition to the client with the large database of addresses. Another client is a large blockchain company and we do analytics for them, using Bare Metal and Hadoop, but not EMR. We're also doing Spark Streaming, Spark SQL, and some queries with Impala. We also have a company that enriches data from mobile companies, in terms of GAL locations of cell phones, with a variety of data from other sources to predict profitability.

I rate Amazon EMR an eight out of ten. It's continuously improving, and now it's possible to manage EMR directly from SageMaker Notebook. It's continuously evolving. I would recommend EMR to others because it's pretty straightforward, so onboarding doesn't take much time.