Apache Spark Reviews

With spark SQL we've now the capabilities to analyse very large quantities of data located in S3 on Amazon at very low cost comparing other solution we checked. 

We also use our own Spark cluster to aggregate data on near real time and save the result on MySQL database. 

We've started new projects using the machine learning library ML.

Until Spark we didn't have the ability to analyse this quantity of data we're talking about two TB/hour. So we're now able to produce a lot of reports, and are also able to develop machine learning based analysis to optimize our business. 

We've central access to every piece of data in the company including finance, business, debug etc. and the ability to join all this data together.

Spark is actually very good for batch analysis much more good than Hadoop, it's much simple, much more quicker etc., but it actually lacks the ability to perform real-time querying like Vertica or Redshift.  

Also, it is more difficult for an end user to work with Spark than normal database. even comparing with analytic database like Vertica or Redshift.

We're now using Spark-Streaming and Spark-SQL for almost 2 years. 

We're working on AWS so we need to have a managed environment. We've choose to go with a solution based on Chef to deploy and configure the spark clusters. Tip : if you don't have any devops you can use the ec2 script (provided by spark distro) to deploy cluster on amazon. We've tested it and work perfectly.  

Spark Streaming is difficult to stabilize as you're always dependant to your stream flow. If you start to be late on the consumer you've a serious problem. We've encountered a lot of stability issue to configure it as expected

It's linked to stability in our case it's takes time to evaluate what is the correct size of the cluster you need. It's very important to always add to you jobs monitoring to be able to understand what's the problem. We use datadog as monitoring platform

Yes to make this job we've used a MySQL database. We switch because MySQL is not a scalable solution and we've reach it's limits.

Setup a spark cluster can be difficult. it's related to your clustering strategy. There is 4 solution at least. 

ec2 script : work only on Amazon AWS

Standalone : manually configuration (hard)

Yarn : to leverage your already existing Hadoop environment.

Mesos : to use with your other Mesos ready application

We use Databricks as online DB ad hoc query. It's work on AWS as managed service, it manage for you the cluster creation, configuration and monitoring.

Give a notebook oriented user interface to query any data source using Spark: DB, Parquet, CSV, Avro etc...

Yes we've started to evaluate analytics databases : vertica, exasol, and other for all the them the price was an issue regarding the quantity of data we want to manipulate.

The most important feature of Apache Spark is that it provides large scale data processing with negligible latency at the cost of commodity hardwares. Spark framework is just a blessings over Hadoop, as the later does not allow fast processing of data, which is accomplished by the in-memory data processing of Spark.

Apache Spark is a framework, which allows one organization to perform business & data analytics, at a very low cost, as compared to Ab-Initio or Informatica. Thus, by using Apache Spark in place of those tools, one organization can achieve huge reduction in cost, & without compromising with any data security & other data related issues, if controlled by an expert Scala programmer  & Apache Spark does not bear the overheads of Hadoop of having high latency. All these points, by which my organization is being benefitted as well.

Question of improvement always comes to mind of the developers. Just like the most common need of the developers, if a user-friendly GUI along with 'drag & drop' feature can be attached to this framework, then it would be easier to access it.

Another thing to mention, there always is a place for improvement in terms of the memory usage. If in future, it is achievable to use less memory for processing, it would obviously be better.

We've had no issues with deployment.

See above regarding memory usage.

We've had no issues with scalability.

My advice to others would be just to use Apache Spark for large scale data processing, as it provides good performance at low cost, unlike Ab-Initio or Informatica. But the main problem is, now in the market, there are not many people certified in Apache Spark.

• RDDs
• DataFrames
• Machine learning libraries

Faster time to parse and compute data. It makes web-based queries for plotting data easier.

It needs to be simpler to use the machine learning algorithms supported by Octave (example polynomial regressions, polynomial interpolation).

I've been using it for one year.

There have been no issues with the deployment.

There have been no issues with the stability.

There have been no issues with the scalability.

We still rely on user forums for my answers. We do not use a commercial product yet.

The initial set-up was easy. I have not explored using this on AWS clusters.

We did an in-house implementation and development for our regression tool.

The ROI will be an in-house product to do machine learning analytics on data obtained from customer.

We did not evaluate any other products.

It's easy to use and has a learning curve.

Spark Streaming, which allows you to construct event-driven information systems and respond to the events in near-real time.

Apache Spark’s ability to perform batch processing at one second or less intervals is the most transformative and less pervasive for any data processing application. The ingested data can also be validated and verified for quality early in the data pipeline.

Apache Spark as a data processing engine has come a long way since its inception. Although you are able to perform complex transformations using Spark libraries, the support for SQL to perform transformations is still limited. You can alleviate some of these limitations by running Spark within Hadoop ecosystem and by leveraging the fairly evolved HiveQL.

I've used it for 16 months.

The enterprise scale deployment of Apache Spark is slightly involved to derive its full potential of stability, scalability and security. However, some Hadoop vendors like Cloudera have integrated Spark data processing engine into their Hadoop platforms and have made it easier to deploy, scale and secure.

This is an open source technology and is dependent on community support. The Apache Spark community is vibrant and it is easy to find answers to questions. The enterprises can also get commercial support from Hadoop vendors such as Cloudera. I recommend enterprises to inspect Hadoop vendors’ commitment to open source as well as their ability to curate Apache Spark technology into the rest of the ecosystem before signing up for a commercial support or subscription.

The initial set-up is straightforward as long as you have picked a right Hadoop distribution.

I recommend engaging an experienced Hadoop vendor during the planning and initial implementation phases of the project. You will be able to avoid any potential pitfalls or reduce overall project time by having a Hadoop expert guiding you during the initial stages of the project.

I evaluated some other technologies such as Samza but community backing for Apache Spark stood out.

I also suggest having a Chief Technologist who has extensive experience in architecting several Big Data solutions. They should be able to communicate in business as well as technology language. Their expertise should range from infrastructure to application development and have command of Hadoop technologies.

Spark is relatively easy to deploy, with rich features in handling big data. Spark Core, Spark SQL, Spark MLlib are used mostly in our applications.

I use Spark to process large amount of data in the energy industry.

Good tool to analyse Spark application performance. Right now there are still many parameters to tune in order to get good performance of Spark application, I would like to see the auto tuning of parameters.

I've been using Spark for seven months.

There were no issues with the deployment.

I ran into Spark application performance issues. For instance, Spark JDBC write performance needs to be improved.

There were no issues with the scalability.

I use Apache open source. Everything is on our own.

I use Apache open source. Everything is on our own.

I evaluated Hadoop-based solution, and chose Spark due to the fast processing and ease of use.

The initial setup is not complex. The online documents are pretty good.

I implemented it in-house.

Get to know how Spark works, what are job, stage, task, DAG, etc., and it will help you to write Spark application.

Streaming data processing

In the previous version, we use Storm to handle real-time data, however its performance doesn't meet the requirement. Spark Streaming's micro-batch mode helps improving performance. Also, Spark provides lots of high-level APIs, which reduces duplication of work.

Better monitoring ability. Especially monitoring integration with customer codes.

I've used it for one year.

We met some standalone deployment issues, which showed that its stability is not that good. So we plan to switch to Yarn or Mesos mode

I have to say it is bad. I can only ask for help in the Google group. However, it is run in the developer-for-developer style. There are almost no people from databricks. I also use a Cassandra-Spark-connector, and Datastax has at least one dedicated person to help the community.

Not that straightforward in terms of standalone deployment, there are some tricks which are not mentioned in the docs.

We did it in-house.

So far we have no plan to switch to commercial license.

I love Spark over other solutions.

There are several valuable features.

• Interactive data access (low latency)
• Batch ETL-style processing
• Schema-free data models
• Algorithms

We have 1000x improvement in performance over other techniques. It's enabled interactive self-service access to data.

Better integration of BI tools wold be a much appreciated improvement.

I've used it for about 14 months.

I haven't had any issues with deployment.

It's been stable for us.

It's scaled without issue.

Customer service is excellent.

Technical support is excellent.

Yes, we previously used Oracle, from which we ported our data.

The initial setup was simple.

We implemented it with our in-house team.

Be sure to Uuse the Apache versions and avoid vendor-specific extensions.

It allows the loading and investigation of very lard data sets, has MLlib for machine learning, Spark streaming, and both the new and old dataframe API.

We're able to perform data discovery on large datasets without too much difficulty.

It needs better documentation as well as examples for all the Spark libraries. That would be very helpful in maximizing its capabilities and results.

I've used it for over nine months now.

I haven't encountered any issues with deployment.

There have been no stability issues.

I haven't had any scalability issues. It scales better than Python and R.

I haven't had to use customer service.

I haven't had to use technical support.

I previously used Python and R, but neither of these scaled particularly well.

The initial setup was complex. It was not easy getting the correct version and dependencies set up.

I implemented it in-house on my own!

It's open-source, so ROI is inapplicable.

Learn Scala as this will greatly reduce the pain in starting off with Spark.