Try our new research platform with insights from 80,000+ expert users

Apache Hadoop vs Apache Spark comparison

 

Comparison Buyer's Guide

Executive Summary

Review summaries and opinions

We asked business professionals to review the solutions they use. Here are some excerpts of what they said:
 

ROI

Sentiment score
6.5
Apache Hadoop provides cost-effective data storage and processing, though ROI varies based on analytics use and sophistication.
Sentiment score
7.3
Apache Spark reduces operational costs by up to 50%, offering high ROI and efficient performance despite infrastructure expenses.
 

Customer Service

Sentiment score
6.4
Customer service varies by Hadoop distributor, with Hortonworks rated highly; support depends on vendor, community resources, or external vendors.
Sentiment score
6.1
Apache Spark support ranges from vibrant community help to paid vendor plans, with experiences varying based on user needs.
It's not structured support, which is why we don't use purely open-source projects without additional structured support.
 

Scalability Issues

Sentiment score
7.6
Apache Hadoop excels in scalability, allowing easy cluster expansion and efficient data handling, ideal for varied organizational needs.
Sentiment score
7.7
Apache Spark is scalable, efficiently manages large workloads, and is praised for stability, adaptability, and expansive capabilities.
It is a distributed file system and scales reasonably well as long as it is given sufficient resources.
 

Stability Issues

Sentiment score
7.3
Apache Hadoop's stability, rated 8/10, improves with newer versions, though minor issues exist with memory and data processing.
Sentiment score
7.5
Apache Spark is stable and reliable, with improved versions addressing issues, widely used by major tech companies.
Continuous management in the way of upgrades and technical management is necessary to ensure that it remains effective.
 

Room For Improvement

Apache Hadoop needs improved usability, integration, security, support, and performance for efficient high-volume data processing and better community resources.
The problem with Apache Hadoop arose when the guys that originally set it up left the firm, and the group that later owned it didn't have enough technical resources to properly maintain it.
 

Setup Cost

Enterprise Hadoop offers cost benefits but varies with deployment type and distribution, impacting smaller organizations more heavily.
 

Valuable Features

Apache Hadoop excels with a scalable, cost-effective system handling diverse data types, integrating with tools, and supporting big data analytics.
If you don't do the upgrades, the platform ages out, and that's what happened to the Hadoop content.
 

Categories and Ranking

Apache Hadoop
Average Rating
7.8
Reviews Sentiment
6.7
Number of Reviews
40
Ranking in other categories
Data Warehouse (7th)
Apache Spark
Average Rating
8.4
Reviews Sentiment
7.4
Number of Reviews
66
Ranking in other categories
Hadoop (1st), Compute Service (4th), Java Frameworks (2nd)
 

Mindshare comparison

Apache Hadoop and Apache Spark aren’t in the same category and serve different purposes. Apache Hadoop is designed for Data Warehouse and holds a mindshare of 5.1%, down 5.4% compared to last year.
Apache Spark, on the other hand, focuses on Hadoop, holds 17.7% mindshare, down 21.1% since last year.
Data Warehouse
Hadoop
 

Q&A Highlights

it_user1272297 - PeerSpot reviewer
Apr 19, 2020
 

Featured Reviews

Sushil Arya - PeerSpot reviewer
Provides ease of integration with the IT workflow of a business
When working with Kafka, I saw that the data came in an incremental order. The incremental data processing part is still not very effective in Apache Hadoop. If the data is already there, it can be processed very effectively, especially if the data is coming in every second. If you want to know the location of some data every second, then such data is not processed effectively in Apache Hadoop. I can say that one of the features where improvements are required revolves around the licensing cost of the tool. If the tool can build some licensing structures in a pay-per-use manner, organizations can get the look and feel of Apache Hadoop. Apache Hadoop can offer a licensing structure of the product that can be seen as similar to how AWS operates. Apache Hadoop can look into the capability of processing incremental data. The tool's setup process can be a scope of improvement. Also, it is not very simple because while doing the setup, we need to do all the server settings, including port listing and firewall configurations. If we look at other products on the market, then they can be made simpler. There are certain shortcomings when it comes to the product's technical support part, making it an area where improvements are required. The time frame for the resolution is an area that needs to be improved. The overall communication part of the technical support team also needs improvement.
Dunstan Matekenya - PeerSpot reviewer
Open-source solution for data processing with portability
Apache Spark is known for its ease of use. Compared to other available data processing frameworks, it is user-friendly. While many choices now exist, Spark remains easy to use, particularly with Python. You can utilize familiar programming styles similar to Pandas in Python, including object-oriented programming. Another advantage is its portability. I can prototype and perform some initial tasks on my laptop using Spark without needing to be on Databricks or any cloud platform. I can transfer it to Databricks or other platforms, such as AWS. This flexibility allows me to improve processing even on my laptop. For instance, if I'm processing large amounts of data and find my laptop becoming slow, I can quickly switch to Spark. It handles small and large datasets efficiently, making it a versatile tool for various data processing needs.
report
Use our free recommendation engine to learn which Data Warehouse solutions are best for your needs.
856,873 professionals have used our research since 2012.
 

Answers from the Community

it_user1272297 - PeerSpot reviewer
Apr 19, 2020
Apr 19, 2020
I haven't used SQream personally. However, if you are only considering GPU based rdbms's please check the following https://hackernoon.com/which-gpu-database-is-right-for-me-6ceef6a17505
2 out of 4 answers
Russell Rothstein - PeerSpot reviewer
Jan 27, 2020
Morten, the most popular comparisons of SQream can be found here: https://www.itcentralstation.com/products/sqream-db-alternatives-and-competitors The top ones include Cassandra, MemSQL, MongoDB, and Vertica.
reviewer1219965 - PeerSpot reviewer
Jan 27, 2020
I haven't used SQream personally. However, if you are only considering GPU based rdbms's please check the following https://hackernoon.com/which-gpu-database-is-right-for-me-6ceef6a17505
 

Top Industries

By visitors reading reviews
Financial Services Firm
34%
Computer Software Company
12%
University
5%
Energy/Utilities Company
5%
Financial Services Firm
27%
Computer Software Company
13%
Manufacturing Company
7%
Comms Service Provider
6%
 

Company Size

By reviewers
Large Enterprise
Midsize Enterprise
Small Business
 

Questions from the Community

What do you like most about Apache Hadoop?
It's primarily open source. You can handle huge data volumes and create your own views, workflows, and tables. I can also use it for real-time data streaming.
What is your experience regarding pricing and costs for Apache Hadoop?
The product is open-source, but some associated licensing fees depend on the subscription level. While it might be free for students, organizations typically need to pay for their subscriptions. Th...
What needs improvement with Apache Hadoop?
The problem with Apache Hadoop arose when the guys that originally set it up left the firm, and the group that later owned it didn't have enough technical resources to properly maintain it. This wa...
What do you like most about Apache Spark?
We use Spark to process data from different data sources.
What is your experience regarding pricing and costs for Apache Spark?
Apache Spark is open-source, so it doesn't incur any charges.
What needs improvement with Apache Spark?
There is complexity when it comes to understanding the whole ecosystem, especially for beginners. I find it quite complex to understand how a Spark job is initiated, the roles of driver nodes, work...
 

Overview

 

Sample Customers

Amazon, Adobe, eBay, Facebook, Google, Hulu, IBM, LinkedIn, Microsoft, Spotify, AOL, Twitter, University of Maryland, Yahoo!, Cornell University Web Lab
NASA JPL, UC Berkeley AMPLab, Amazon, eBay, Yahoo!, UC Santa Cruz, TripAdvisor, Taboola, Agile Lab, Art.com, Baidu, Alibaba Taobao, EURECOM, Hitachi Solutions
Find out what your peers are saying about Snowflake Computing, Oracle, Teradata and others in Data Warehouse. Updated: May 2025.
856,873 professionals have used our research since 2012.