Try our new research platform with insights from 80,000+ expert users

Apache Spark vs Google Cloud Dataflow comparison

 

Comparison Buyer's Guide

Executive Summary

Review summaries and opinions

We asked business professionals to review the solutions they use. Here are some excerpts of what they said:
 

ROI

Sentiment score
6.6
Apache Spark enhances machine learning, cutting operational costs by up to 50%, with efficiency reliant on resources and expertise.
Sentiment score
5.6
Google Cloud Dataflow was appreciated for cost savings and time efficiency, though some considered its impact not fully assessable yet.
 

Customer Service

Sentiment score
5.9
Apache Spark support feedback varies, with mixed reviews on community forums, vendor support, and documentation adequacy.
Sentiment score
6.6
Google Cloud Dataflow support varies, with users praising technical resolution but highlighting inconsistent response times and accessibility.
Google's support team is good at resolving issues, especially with large data.
The fact that no interaction is needed shows their great support since I don't face issues.
Whenever we have issues, we can consult with Google.
 

Scalability Issues

Sentiment score
7.5
Apache Spark excels in scalability, efficiently handling large data workloads with ease, though it requires skilled infrastructure management.
Sentiment score
7.3
Google Cloud Dataflow excels in scalability and efficiency, making it ideal for real-time data processing and dynamic needs.
Google Cloud Dataflow has auto-scaling capabilities, allowing me to add different machine types based on pace and requirements.
Google Cloud Dataflow can handle large data processing for real-time streaming workloads as they grow, making it a good fit for our business.
As a team lead, I'm responsible for handling five to six applications, but Google Cloud Dataflow seems to handle our use case effectively.
 

Stability Issues

Sentiment score
7.5
Apache Spark is generally stable, trusted by companies; newer versions enhance reliability, though memory issues may arise without proper configuration.
Sentiment score
8.3
Google Cloud Dataflow is stable, reliably handles tasks, and benefits from automatic scaling, with minor issues on complex tasks.
MapReduce needs to perform numerous disk input and output operations, while Apache Spark can use memory to store and process data.
I have not encountered any issues with the performance of Dataflow, as it is stable and backed by Google services.
The job we built has not failed once over six to seven months.
The automatic scaling feature helps maintain stability.
 

Room For Improvement

Apache Spark requires improvements in scalability, usability, documentation, memory efficiency, real-time processing, and broader language support for better performance.
Google Cloud Dataflow needs better Kafka integration, improved error logs, reduced startup time, and enhanced Python SDK features.
Outside of Google Cloud Platform, it is problematic for others to use it and may require promotion as an actual technology.
I would like to see improvements in consistency and flexibility for schema design for NoSQL data stored in wide columns.
Dealing with a huge volume of data causes failure due to array size.
 

Setup Cost

Apache Spark is cost-effective but may incur expenses from hardware, cloud resources, or commercial support, impacting deployment costs.
Google Cloud Dataflow is praised for cost-effectiveness and scalability, offering competitive pricing influenced by pipeline complexity and company size.
It is part of a package received from Google, and they are not charging us too high.
 

Valuable Features

Apache Spark offers fast in-memory processing, scalable analytics, MLlib for machine learning, SQL support, and seamless integration with languages.
Google Cloud Dataflow offers seamless integration, multi-language support, scalability, and serverless data handling for efficient batch and streaming processes.
Not all solutions can make this data fast enough to be used, except for solutions such as Apache Spark Structured Streaming.
It supports multiple programming languages such as Java and Python, enabling flexibility without the need to learn something new.
Google Cloud Dataflow's features for event stream processing allow us to gain various insights like detecting real-time alerts.
The integration within Google Cloud Platform is very good.
 

Categories and Ranking

Apache Spark
Average Rating
8.4
Reviews Sentiment
7.3
Number of Reviews
67
Ranking in other categories
Hadoop (1st), Compute Service (4th), Java Frameworks (2nd)
Google Cloud Dataflow
Average Rating
8.0
Reviews Sentiment
7.1
Number of Reviews
14
Ranking in other categories
Streaming Analytics (7th)
 

Mindshare comparison

Apache Spark and Google Cloud Dataflow aren’t in the same category and serve different purposes. Apache Spark is designed for Hadoop and holds a mindshare of 19.2%, down 20.2% compared to last year.
Google Cloud Dataflow, on the other hand, focuses on Streaming Analytics, holds 6.0% mindshare, down 7.7% since last year.
Hadoop
Streaming Analytics
 

Featured Reviews

Omar Khaled - PeerSpot reviewer
Empowering data consolidation and fast decision-making with efficient big data processing
I can improve the organization's functions by taking less time to make decisions. To make the right decision, you need the right data, and a solution can provide this by hiring talent and employees who can consolidate data from different sources and organize it. Not all solutions can make this data fast enough to be used, except for solutions such as Apache Spark Structured Streaming. To make the right decision, you should have both accurate and fast data. Apache Spark itself is similar to the Python programming language. Python is a language with many libraries for mathematics and machine learning. Apache Spark is the solution, and within it, you have PySpark, which is the API for Apache Spark to write and run Python code. Within it, there are many APIs, including SQL APIs, allowing you to write SQL code within a Python function in Apache Spark. You can also use Apache Spark Structured Streaming and machine learning APIs.
Jana Polianskaja - PeerSpot reviewer
Build Scalable Data Pipelines with Apache Beam and Google Cloud Dataflow
As a data engineer, I find several features of Google Cloud Dataflow particularly valuable. The ability to test solutions locally using Direct Runner is crucial for development, allowing me to validate pipelines without incurring the costs of full Dataflow jobs. The unified programming model for both batch and streaming processing is exceptional - requiring only minor code adjustments to optimize for either mode. This flexibility extends to language support, with robust implementations in both Java and Python, allowing teams to leverage their existing expertise. The platform's comprehensive monitoring capabilities are another standout feature. The intuitive interface, Grafana integration, and extensive service connectivity make troubleshooting and performance tracking highly efficient. Furthermore, seamless integration with Google Cloud Composer (managed Airflow) enables sophisticated orchestration of data pipelines.
report
Use our free recommendation engine to learn which Hadoop solutions are best for your needs.
865,295 professionals have used our research since 2012.
 

Top Industries

By visitors reading reviews
Financial Services Firm
26%
Computer Software Company
10%
Manufacturing Company
7%
Comms Service Provider
7%
Financial Services Firm
17%
Manufacturing Company
12%
Retailer
11%
Computer Software Company
9%
 

Company Size

By reviewers
Large Enterprise
Midsize Enterprise
Small Business
 

Questions from the Community

What do you like most about Apache Spark?
We use Spark to process data from different data sources.
What is your experience regarding pricing and costs for Apache Spark?
Apache Spark is open-source, so it doesn't incur any charges.
What needs improvement with Apache Spark?
There is complexity when it comes to understanding the whole ecosystem, especially for beginners. I find it quite complex to understand how a Spark job is initiated, the roles of driver nodes, work...
What do you like most about Google Cloud Dataflow?
The product's installation process is easy...The tool's maintenance part is somewhat easy.
What is your experience regarding pricing and costs for Google Cloud Dataflow?
Pricing is normal. It is part of a package received from Google, and they are not charging us too high.
What needs improvement with Google Cloud Dataflow?
I am not sure, as we built only one job, and it is running on a daily basis. Everything else is managed using BigQuery schedulers and Talend. However, occasionally, dealing with a huge volume of da...
 

Also Known As

No data available
Google Dataflow
 

Overview

 

Sample Customers

NASA JPL, UC Berkeley AMPLab, Amazon, eBay, Yahoo!, UC Santa Cruz, TripAdvisor, Taboola, Agile Lab, Art.com, Baidu, Alibaba Taobao, EURECOM, Hitachi Solutions
Absolutdata, Backflip Studios, Bluecore, Claritics, Crystalloids, Energyworx, GenieConnect, Leanplum, Nomanini, Redbus, Streak, TabTale
Find out what your peers are saying about Apache, Cloudera, Amazon Web Services (AWS) and others in Hadoop. Updated: August 2025.
865,295 professionals have used our research since 2012.