Apache Spark Reviews

It allows the loading and investigation of very lard data sets, has MLlib for machine learning, Spark streaming, and both the new and old dataframe API.

We're able to perform data discovery on large datasets without too much difficulty.

It needs better documentation as well as examples for all the Spark libraries. That would be very helpful in maximizing its capabilities and results.

I've used it for over nine months now.

I haven't encountered any issues with deployment.

There have been no stability issues.

I haven't had any scalability issues. It scales better than Python and R.

I haven't had to use customer service.

I haven't had to use technical support.

I previously used Python and R, but neither of these scaled particularly well.

The initial setup was complex. It was not easy getting the correct version and dependencies set up.

I implemented it in-house on my own!

It's open-source, so ROI is inapplicable.

Learn Scala as this will greatly reduce the pain in starting off with Spark.

\Spark Streaming, Spark SQL and MLib in that order.

We have been using Spark to do a lot of batch and stream processing of inbound data from Apache Kafka. Scaling Spark on YARN is still an issue but we are getting acceptable performance.

Like I said scalability is still an issue, also stability. Spark on Yarn still doesn't seem to have programming submission api, so have to rely on spark-submit script to run jobs on YARN. Scala vs Java API have performance differences which will require sometimes to code in Scala.

Have Scala developers at hand. Base Java competency will not be enough during optimization rounds.

We primarily use the solution for security analytics.

The scalability has been the most valuable aspect of the solution.

The management tools could use improvement. Some of the debugging tools need some work as well. They need to be more descriptive. 

The 2.3 version is quite stable. All of our customers use it, there are around 100,000+ users, and it runs 24/7.

The scalability is very good.

You actually buy Cloudera along with it. You don't really get any support, except you need support.

In previous companies, we used MySQL platform and solutions like ArcSight and Splunk. We switched for scalability. MySQL wasn't going to scale, and we don't use Splunk at this company.

The initial setup was complex. It is a complex tool. It's a lot to do with how you will use it. There is a lot to set up. They need to put a lot of scripts to it. There's nearly 60 to set up. When you set up the cloud, it takes about a day to set up. If you set it up on-premise, you know, on hardware, it only takes about a week.

I would rate this solution eight out of 10. 

The main feature that we find valuable is that it is very fast. In terms of big data, the main feature is that the data is in so many different nodes. It goes through many data nodes so whenever we use the data, it enables us to parse the data from different data nodes. 

We use big data manager but we cannot use it as conditional data so whenever we're trying to fetch the data, it takes a bit of time. There is some latency in the system and latency in the data caching. The main issue is that we need to design it in a way that data will be available to us very quickly. It takes a long time and the latest data should be available to us much quicked. 

We don't have any problems with stability. 

I'm not the one who would contact their support if we needed it. 

The initial setup is straightforward. 

The advice that I would give to someone considering this solution is that the quality of data has key streaming capabilities like velocity. This means how quickly you are going to refer to the data. These things matter by designing the solution. We need to take these things out. 

I would rate Apache Spark an eight out of ten. 

To make it a ten they should improve the speed. The data storage capacity means we can inject somewhere in the user database in more efficient ways.