IT Central Station is now PeerSpot: Here's why

Apache Spark OverviewUNIXBusinessApplication

Apache Spark is #1 ranked solution in top Hadoop tools, #2 ranked solution in top Compute Service tools, and #2 ranked solution in top Java Frameworks. PeerSpot users give Apache Spark an average rating of 8 out of 10. Apache Spark is most commonly compared to Spring Boot: Apache Spark vs Spring Boot. Apache Spark is popular among the large enterprise segment, accounting for 70% of users researching this solution on PeerSpot. The top industry researching this solution are professionals from a computer software company, accounting for 21% of all views.
Apache Spark Buyer's Guide

Download the Apache Spark Buyer's Guide including reviews and more. Updated: July 2022

What is Apache Spark?

Spark provides programmers with an application programming interface centered on a data structure called the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way. It was developed in response to limitations in the MapReduce cluster computing paradigm, which forces a particular linear dataflowstructure on distributed programs: MapReduce programs read input data from disk, map a function across the data, reduce the results of the map, and store reduction results on disk. Spark's RDDs function as a working set for distributed programs that offers a (deliberately) restricted form of distributed shared memory

Apache Spark Customers

NASA JPL, UC Berkeley AMPLab, Amazon, eBay, Yahoo!, UC Santa Cruz, TripAdvisor, Taboola, Agile Lab, Art.com, Baidu, Alibaba Taobao, EURECOM, Hitachi Solutions

Apache Spark Video

Apache Spark Pricing Advice

What users are saying about Apache Spark pricing:
  • "Apache Spark is open-source. You have to pay only when you use any bundled product, such as Cloudera."
  • "Since we are using the Apache Spark version, not the data bricks version, it is an Apache license version, the support and resolution of the bug are actually late or delayed. The Apache license is free."
  • Apache Spark Reviews

    Filter by:
    Filter Reviews
    Industry
    Loading...
    Filter Unavailable
    Company Size
    Loading...
    Filter Unavailable
    Job Level
    Loading...
    Filter Unavailable
    Rating
    Loading...
    Filter Unavailable
    Considered
    Loading...
    Filter Unavailable
    Order by:
    Loading...
    • Date
    • Highest Rating
    • Lowest Rating
    • Review Length
    Search:
    Showingreviews based on the current filters. Reset all filters
    Kürşat Kurt - PeerSpot reviewer
    Software Architect at Akbank
    Real User
    Top 10
    Provides fast aggregations, AI libraries, and a lot of connectors
    Pros and Cons
    • "AI libraries are the most valuable. They provide extensibility and usability. Spark has a lot of connectors, which is a very important and useful feature for AI. You need to connect a lot of points for AI, and you have to get data from those systems. Connectors are very wide in Spark. With a Spark cluster, you can get fast results, especially for AI."
    • "Stream processing needs to be developed more in Spark. I have used Flink previously. Flink is better than Spark at stream processing."

    What is our primary use case?

    We just finished a central front project called MFY for our in-house fraud team. In this project, we are using Spark along with Cloudera. In front of Spark, we are using Couchbase.  Spark is mainly used for aggregations and AI (for future usage). It gathers stuff from Couchbase and does the calculations. We are not actively using Spark AI libraries at this time, but we are going to use them.   This project is for classifying the transactions and finding suspicious activities, especially those suspicious activities that come from internet channels such as internet banking and mobile banking. It tries to find out suspicious activities and executes rules that are being developed or written by our business team. An example of a rule is that if the transaction count or transaction amount is greater than 10 million Turkish Liras and the user device is new, then raise an exception. The system sends an SMS to the user, and the user can choose to continue or not continue with the transaction.

    How has it helped my organization?

    Aggregations are very fast in our project since we started to use Spark. We can tell results in around 300 milliseconds. Before using Spark, the time was around 700 milliseconds.  Before using Spark, we only used Couchbase. We needed fast results for this project because transactions come from various channels, and we need to decide and resolve them at the earliest because users are performing the transaction. If our result or process takes longer, users might stop or cancel their transactions, which means losing money. Therefore, fast results time is very important for us.

    What is most valuable?

    AI libraries are the most valuable. They provide extensibility and usability. Spark has a lot of connectors, which is a very important and useful feature for AI. You need to connect a lot of points for AI, and you have to get data from those systems. Connectors are very wide in Spark. With a Spark cluster, you can get fast results, especially for AI. 

    What needs improvement?

    Stream processing needs to be developed more in Spark. I have used Flink previously. Flink is better than Spark at stream processing.
    Buyer's Guide
    Apache Spark
    July 2022
    Learn what your peers think about Apache Spark. Get advice and tips from experienced pros sharing their opinions. Updated: July 2022.
    610,812 professionals have used our research since 2012.

    For how long have I used the solution?

    I am a Java developer. I have been interested in Spark for around five years. We have been actively using it in our organization for almost a year.

    What do I think about the stability of the solution?

    It is the most stable platform. As compare to Flink, Spark is good, especially in terms of clusters and architecture. My colleagues who set up these clusters say that Spark is the easiest.

    What do I think about the scalability of the solution?

    It is scalable, but we don't have the need to scale it.  It is mainly used for reporting big data in our organization. All teams, especially the VR team, are using Spark for job execution and remote execution. I can say that 70% of users use Spark for reporting, calculations, and real-time operations. We are a very big company, and we have around a thousand people in IT. We will continue its usage and develop more. We have kind of just started using it. We finished this project just three months ago. We are now trying to find out bottlenecks in our systems, and then we are ready to go.

    How are customer service and support?

    We have not used Apache support. We have only used Cloudera support for this project, and they helped us a lot during the development cycle of this project. 

    How was the initial setup?

    I don't have any idea about it. We are a big company, and we have another group for setting up Spark.

    What other advice do I have?

    I would advise planning well before implementing this solution. In enterprise corporations like ours, there are a lot of policies. You should first find out your needs, and after that, you or your team should set it up based on your needs. If your needs change during development because of the business requirements, it will be very difficult.  If you are clear about your needs, it is easier to set it up. If you know how Spark is used in your project, you have to define firewall rules and cluster needs. When you set up Spark, it should be ready for people's usage, especially for remote job execution.  I would rate Apache Spark a nine out of ten.

    Which deployment model are you using for this solution?

    On-premises
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    Director at Nihil Solutions
    Real User
    Top 5Leaderboard
    Stable and easy to set up with a very good memory processing engine
    Pros and Cons
    • "The memory processing engine is the solution's most valuable aspect. It processes everything extremely fast, and it's in the cluster itself. It acts as a memory engine and is very effective in processing data correctly."
    • "The graphical user interface (UI) could be a bit more clear. It's very hard to figure out the execution logs and understand how long it takes to send everything. If an execution is lost, it's not so easy to understand why or where it went. I have to manually drill down on the data processes which takes a lot of time. Maybe there could be like a metrics monitor, or maybe the whole log analysis could be improved to make it easier to understand and navigate."

    What is our primary use case?

    When we receive data from the messaging queue, we process everything using Apache Spark. Data Bricks does the processing and sends back everything the Apache file in the data lake. The machine learning program does some kind of analysis using the ML prediction algorithm.

    What is most valuable?

    The memory processing engine is the solution's most valuable aspect. It processes everything extremely fast, and it's in the cluster itself. It acts as a memory engine and is very effective in processing data correctly.

    What needs improvement?

    There are lots of items coming down the pipeline in the future. I don't know what features are missing. From my point of view, everything looks good.

    The graphical user interface (UI) could be a bit more clear. It's very hard to figure out the execution logs and understand how long it takes to send everything. If an execution is lost, it's not so easy to understand why or where it went. I have to manually drill down on the data processes which takes a lot of time. Maybe there could be like a metrics monitor, or maybe the whole log analysis could be improved to make it easier to understand and navigate.

    There should be more information shared to the user. The solution already has all the information tracked in the cluster. It just needs to be accessible or searchable.

    For how long have I used the solution?

    I started using the solution about four years ago. However, it's been on and off since then. I would estimate in total I have about a year and a half of experience using the solution.

    What do I think about the stability of the solution?

    The stability of the solution is very, very good. It doesn't crash or have glitches. It's quite reliable for us.

    What do I think about the scalability of the solution?

    The scalability of the solution is very good. If a company has to expand it, they can do so.

    Right now, we have about six or seven users that are directly on the product. We're encouraging them to use more data. We do plan to increase usage in the future.

    How are customer service and technical support?

    I'm a developer, so I don't interact directly with technical support. I can't speak to the quality of their service as I've never directly dealt with them.

    Which solution did I use previously and why did I switch?

    We did previously use a lot of different mechanisms, however, we needed something that was good at processing data for analytical purposes, and this solution fit the bill. It's a very powerful tool. I haven't seen other tools that could do precisely what this one does.

    How was the initial setup?

    The initial setup isn't too complex. It's quite straightforward.

    We use CACD DevOps from deployment. We only use Spark for processing and for the Data Bricks cluster to spin off and do the job. It's continuously running int he background.

    There isn't really any maintenance required per se. We just click the button and it comes up automatically, with the whole cluster and the Spark and everything ready to go.

    What's my experience with pricing, setup cost, and licensing?

    I'm unsure as to how much the licensing is for the solution. It's not an aspect of the product I deal with directly.

    What other advice do I have?

    We're customers and also partners with Apache.

    While we are on version 2.6, we are considering upgrading to version 3.0.

    I'd rate the solution nine out of ten. It works very well for us and suits our purposes almost perfectly.

    Which deployment model are you using for this solution?

    On-premises
    Disclosure: My company has a business relationship with this vendor other than being a customer: Partner
    Buyer's Guide
    Apache Spark
    July 2022
    Learn what your peers think about Apache Spark. Get advice and tips from experienced pros sharing their opinions. Updated: July 2022.
    610,812 professionals have used our research since 2012.
    Manager - Data Science Competency at a tech services company with 201-500 employees
    Consultant
    Fast-performance, cost-effective, and runs in a cloud-agnostic environment
    Pros and Cons
    • "One of the key features is that Apache Spark is a distributed computing framework. You can help multiple slaves and distribute the workload between them."
    • "When you are working with large, complex tasks, the garbage collection process is slow and affects performance."

    What is our primary use case?

    My main task is working on predictive analytics, and Apache Spark is one of the tools that I utilize in this role. Primarily, we work with the predictive analysis of very large amounts of data.

    Apache Spark is also helpful for data pre-processing, including data cleaning.

    This solution is cloud-agnostic. You can use it with an EC2 instance and you can even install it on-premises. Some environments have it installed in VMs.

    What is most valuable?

    One of the key features is that Apache Spark is a distributed computing framework. You can have multiple slaves and distribute the workload between them.

    Another feature is memory-based computing. This is unlike Hadoop, which relies on storage. As it uses in-memory data processing, Spark is very fast.

    What needs improvement?

    When you are working with large, complex tasks, the garbage collection process is slow and affects performance. This is an area where they need to improve because your job may fail if it is stuck for a long time while memory garbage collection is happening. This is the main problem that we have.

    For how long have I used the solution?

    I have been working with Apache Spark for the past four years.

    What do I think about the stability of the solution?

    This product is pretty stable. Companies like Facebook, Uber, and Netflix are all using Apache Spark. It's stable enough to be used all over the world.

    What do I think about the scalability of the solution?

    In our team that works on this, we have approximately 10 people.

    How are customer service and support?

    There is no official support for this solution. Because it's open-source and there is no cost involved, there is nobody to contact for support. Our own internal team of experts, which work on different problems, both support and contribute to the platform.

    Which solution did I use previously and why did I switch?

    I work on several open-source frameworks including Python, Scikit-learn, TensorFlow, PyTorch, H20.ai, and R. We don't endorse proprietary tools so we aren't working with them.

    How was the initial setup?

    With respect to the initial setup, it's neither easy nor very difficult. Our team has experience so it is not difficult for them. However, for a person that is new to using it, the setup might be very difficult.

    What about the implementation team?

    We have a team of experts in my company, and they handle it very well.

    What's my experience with pricing, setup cost, and licensing?

    This is an open-source tool, so it can be used free of charge. There is no cost involved.

    What other advice do I have?

    We are not using the current version of this platform, Spark 3. However, we do know that it is used in the market and it has new features. We will eventually move to it.

    My advice for anybody who wants to use Apache Spark is that they have two options. The first is Databricks, which are the creators of Apache Spark, and use their proprietary version. If you choose this option then you will have to pay for the product.

    If instead, you use Apache Spark, then you can rely on your own expert in-house team for support, maintenance, and deployment. In this option, you don't have to pay anything to anybody outside of your company.

    I would rate this solution an eight out of ten.

    Which deployment model are you using for this solution?

    Hybrid Cloud
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    Flag as inappropriate
    Oscar Estorach - PeerSpot reviewer
    Chief Data-strategist and Director at theworkshop.es
    Real User
    Top 5Leaderboard
    Scalable, open-source, and great for transforming data
    Pros and Cons
    • "The solution has been very stable."
    • "It's not easy to install."

    What is our primary use case?

    You can do a lot of things in terms of the transformation of data. You can store and transform and stream data. It's very useful and has many use cases.

    What is most valuable?

    Overall, it's a very nice tool.

    It is great for transforming data and doing micro-streamings or micro-batching.

    The product offers an open-source version.

    The solution has been very stable.

    The scalability is good.

    Apache Spark is a huge tool. It has many use cases and is very flexible. You can use it with so many other platforms. 

    Spark, as a tool, is easy to work with as you can work with Python, Scala, and Java.

    What needs improvement?

    If you are developing projects, and you need to not put them in a production scenario, you might need more than a cluster of servers, as it requires distributed computing.

    It's not easy to install. You are typically dealing with a big data system.

    It's not a simple, straightforward architecture. 

    For how long have I used the solution?

    I've been using the solution for three years.

    What do I think about the stability of the solution?

    The stability is very good. There are no bugs or glitches and it doesn't crash or freeze. It's a reliable solution. 

    What do I think about the scalability of the solution?

    We have found the scalability to be good. If your company needs to expand it, it can do so.

    We have five people working on the solution currently.

    How are customer service and technical support?

    There isn't really technical support for open source. You need to do your own studying. There are lots of places to find information. You can find details online, or in books, et cetera. There are even courses you can take that can help you understand Spark.

    Which solution did I use previously and why did I switch?

    I also use Databricks, which I use in the cloud.

    How was the initial setup?

    When handling big data systems, the installation is a bit difficult. When you need to deploy the systems, it's better to use services like Databricks.

    I am not a professional admin. I am a developer for and design architecture.

    You can use it in your standalone system, however, it's not the best way. It would be okay for little branch codes, not for production.

    What's my experience with pricing, setup cost, and licensing?

    We use the open-source version. It is free to use. However, you do need to have servers. We have three or four. they can be on-premises or in the cloud. 

    What other advice do I have?

    I have the solution installed on my computer and on our servers. You can use it on-premises or as a SaaS.

    I'd rate the solution at a nine out of ten. I've been very pleased with its capabilities. 

    I would recommend the solution for the people who need to deploy projects with streaming. If you have many different sources or different types of data, and you need to put everything in the same place - like a data lake - Spark, at this moment, has the right tools. It's an important solution for data science, for data detectors. You can put all of the information in one place with Spark.

    Which deployment model are you using for this solution?

    On-premises
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    NitinKumar - PeerSpot reviewer
    Director of Enginnering at Sigmoid
    Real User
    Top 5Leaderboard
    Easy to code, fast, open-source, very scalable, and great for big data
    Pros and Cons
    • "Its scalability and speed are very valuable. You can scale it a lot. It is a great technology for big data. It is definitely better than a lot of earlier warehouse or pipeline solutions, such as Informatica. Spark SQL is very compliant with normal SQL that we have been using over the years. This makes it easy to code in Spark. It is just like using normal SQL. You can use the APIs of Spark or you can directly write SQL code and run it. This is something that I feel is useful in Spark."
    • "Its UI can be better. Maintaining the history server is a little cumbersome, and it should be improved. I had issues while looking at the historical tags, which sometimes created problems. You have to separately create a history server and run it. Such things can be made easier. Instead of separately installing the history server, it can be made a part of the whole setup so that whenever you set it up, it becomes available."

    What is our primary use case?

    I use it mostly for ETL transformations and data processing. I have used Spark on-premises as well as on the cloud.

    What is most valuable?

    Its scalability and speed are very valuable. You can scale it a lot. It is a great technology for big data. It is definitely better than a lot of earlier warehouse or pipeline solutions, such as Informatica.

    Spark SQL is very compliant with normal SQL that we have been using over the years. This makes it easy to code in Spark. It is just like using normal SQL. You can use the APIs of Spark or you can directly write SQL code and run it. This is something that I feel is useful in Spark.

    What needs improvement?

    Its UI can be better. Maintaining the history server is a little cumbersome, and it should be improved. I had issues while looking at the historical tags, which sometimes created problems. You have to separately create a history server and run it. Such things can be made easier. Instead of separately installing the history server, it can be made a part of the whole setup so that whenever you set it up, it becomes available.

    For how long have I used the solution?

    I have been using this solution for around five years.

    What do I think about the stability of the solution?

    There were bugs three to four years ago, which have been resolved. There were a couple of issues related to slowness when we did a lot of transformations using the Width columns. I was writing a POC on ETL for moving from Informatica to Spark SQL for the ETL pipeline. It required the use of hundreds of Width columns to change the column name or add some transformation, which made it slow. It happened in versions prior to version 1.6, and it seems that this issue has been fixed later on.

    What do I think about the scalability of the solution?

    It is very scalable. You can scale it a lot.

    How are customer service and technical support?

    I haven't contacted them.

    How was the initial setup?

    The initial setup was a little complex when I was using open-source Spark. I was doing a POC in the on-premise environment, and the initial setup was a little cumbersome. It required a lot of set up on Unix systems. We also had to do a lot of configurations and install a lot of things. 

    After I moved to the Cloudera CDH version, it was a little easy. It is a bundled product, so you just install whatever you want and use it.

    What's my experience with pricing, setup cost, and licensing?

    Apache Spark is open-source. You have to pay only when you use any bundled product, such as Cloudera.

    What other advice do I have?

    I would definitely recommend Spark. It is a great product. I like Spark a lot, and most of the features have been quite good. Its initial learning curve is a bit high, but as you learn it, it becomes very easy.

    I would rate Apache Spark an eight out of ten.

    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    Co-Founder at a tech vendor with 11-50 employees
    Real User
    Top 5Leaderboard
    Handles large volume data, cloud and on-premise deployments, but difficult to use
    Pros and Cons
    • "Apache Spark can do large volume interactive data analysis."
    • "Apache Spark is very difficult to use. It would require a data engineer. It is not available for every engineer today because they need to understand the different concepts of Spark, which is very, very difficult and it is not easy to learn."

    What is our primary use case?

    The solution can be deployed on the cloud or on-premise.

    How has it helped my organization?

    We are using Apache Spark, for large volume interactive data analysis.

    MechBot is an enterprise, one-click installation, trusted data excellence platform. Underneath, I am using Apache Spark, Kafka, Hadoop HDFS, and Elasticsearch.

    What is most valuable?

    Apache Spark can do large volume interactive data analysis.

    What needs improvement?

    Apache Spark is very difficult to use. It would require a data engineer. It is not available for every engineer today because they need to understand the different concepts of Spark, which is very, very difficult and it is not easy to learn.

    For how long have I used the solution?

    I have been using Apache Spark for approximately 11 years.

    What do I think about the stability of the solution?

    The solution is stable.

    What do I think about the scalability of the solution?

    Apache Spark is scalable. However, it needs enormous technical skills to make it scalable. It is not a simple task.

    We have approximately 20 people using this solution.

    How was the initial setup?

    If you want to distribute Apache Spark in a certain way, it is simple. Not every engineer can do it. You need DevOps specialized skills on Spark is what is required.

    If we are going to deploy the solution in a one-layer laptop installation, it is very straightforward, but this is not what someone is going to deploy in the production site.

    What's my experience with pricing, setup cost, and licensing?

    Since we are using the Apache Spark version, not the data bricks version, it is an Apache license version, the support and resolution of the bug are actually late or delayed. The Apache license is free.

    What other advice do I have?

    We are well versed in Spark, the version, the internal structure of Spark, and we know what exactly Spark is doing. 

    The solution cannot be easier. Everything cannot be made simpler because it involves core data, computer science, pro-engineering, and not many people are actually aware of it.

    I rate Apache Spark a six out of ten.

    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    Flag as inappropriate
    AmitMataghare - PeerSpot reviewer
    Associate Director at PwC
    Real User
    Top 20
    High performance, beneficial in-memory support, and useful online community support
    Pros and Cons
    • "One of Apache Spark's most valuable features is that it supports in-memory processing, the execution of jobs compared to traditional tools is very fast."
    • "Apache Spark could improve the connectors that it supports. There are a lot of open-source databases in the market. For example, cloud databases, such as Redshift, Snowflake, and Synapse. Apache Spark should have connectors present to connect to these databases. There are a lot of workarounds required to connect to those databases, but it should have inbuilt connectors."

    What is our primary use case?

    Apache Spark is a programming language similar to Java or Python. In my most recent deployment, we used Apache Spark to build engineering pipelines to move data from sources into the data lake.

    What is most valuable?

    One of Apache Spark's most valuable features is that it supports in-memory processing, the execution of jobs compared to traditional tools is very fast.

    What needs improvement?

    Apache Spark could improve the connectors that it supports. There are a lot of open-source databases in the market. For example, cloud databases, such as Redshift, Snowflake, and Synapse. Apache Spark should have connectors present to connect to these databases. There are a lot of workarounds required to connect to those databases, but it should have inbuilt connectors.

    For how long have I used the solution?

    I have been using Apache Spark for approximately five years.

    What do I think about the stability of the solution?

    Apache Spark is stable.

    What do I think about the scalability of the solution?

    I have found Apache Spark to be scalable.

    How are customer service and support?

    Apache Spark is open-source, there is no team that will give you dedicated support, but you can post your queries on the community forums, and usually, you will receive a good response. Since it's open-source, you depend on freelance developers to respond to you, you cannot put a time limit there, but the response, on average, is pretty good.

    How was the initial setup?

    If Apache Spark is in the cloud, setting it up will require only minutes. If it's on Amazon, GCP, or Microsoft cloud, it'll take minutes to set everything up. However, if you are using the on-premise version, then it might take some time to set up the environment.

    What other advice do I have?

    I rate Apache Spark an eight out of ten.

    Which deployment model are you using for this solution?

    Public Cloud

    If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?

    Amazon Web Services (AWS)
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    Flag as inappropriate
    Senior Test Automation Consultant / Architect at a tech services company with 11-50 employees
    Consultant
    Useful for big data and scientific purposes, but needs better query handling, stability, and scalability
    Pros and Cons
    • "It is useful for handling large amounts of data. It is very useful for scientific purposes."
    • "We are building our own queries on Spark, and it can be improved in terms of query handling."

    What is our primary use case?

    We are using it for big data. We are using a small part of it, which is related to using data.

    What is most valuable?

    It is useful for handling large amounts of data. It is very useful for scientific purposes.

    What needs improvement?

    There are some difficulties that we are working on. It is useful for scientific purposes, but for commercial use of big data, it gives some trouble.

    They should improve the stability of the product. We use Spark Executors and Spark Drivers to link to our own environment, and they are not the most stable products. Its scalability is also an issue.

    We are building our own queries on Spark, and it can be improved in terms of query handling.

    For how long have I used the solution?

    In my company, it has been used for several years, but I have been using it for seven months.

    What do I think about the scalability of the solution?

    It is not scalable. Scalability is one of the issues.

    How are customer service and support?

    It is open source from my point of view. So, there is no support.

    What other advice do I have?

    I would advise not using it if you don't have experienced users inside your organization. If you have to figure it all out on your own, then you shouldn't start with it.

    Overall, I would rate it a six out of 10. For a commercial use case, it is a six out of 10. For scientific purposes, it is an eight out of 10.

    Which deployment model are you using for this solution?

    On-premises
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    Flag as inappropriate
    Buyer's Guide
    Download our free Apache Spark Report and get advice and tips from experienced pros sharing their opinions.
    Updated: July 2022
    Buyer's Guide
    Download our free Apache Spark Report and get advice and tips from experienced pros sharing their opinions.