Apache Spark Reviews

Spark Streaming, which allows you to construct event-driven information systems and respond to the events in near-real time.

Apache Spark’s ability to perform batch processing at one second or less intervals is the most transformative and less pervasive for any data processing application. The ingested data can also be validated and verified for quality early in the data pipeline.

Apache Spark as a data processing engine has come a long way since its inception. Although you are able to perform complex transformations using Spark libraries, the support for SQL to perform transformations is still limited. You can alleviate some of these limitations by running Spark within Hadoop ecosystem and by leveraging the fairly evolved HiveQL.

I've used it for 16 months.

The enterprise scale deployment of Apache Spark is slightly involved to derive its full potential of stability, scalability and security. However, some Hadoop vendors like Cloudera have integrated Spark data processing engine into their Hadoop platforms and have made it easier to deploy, scale and secure.

This is an open source technology and is dependent on community support. The Apache Spark community is vibrant and it is easy to find answers to questions. The enterprises can also get commercial support from Hadoop vendors such as Cloudera. I recommend enterprises to inspect Hadoop vendors’ commitment to open source as well as their ability to curate Apache Spark technology into the rest of the ecosystem before signing up for a commercial support or subscription.

The initial set-up is straightforward as long as you have picked a right Hadoop distribution.

I recommend engaging an experienced Hadoop vendor during the planning and initial implementation phases of the project. You will be able to avoid any potential pitfalls or reduce overall project time by having a Hadoop expert guiding you during the initial stages of the project.

I evaluated some other technologies such as Samza but community backing for Apache Spark stood out.

I also suggest having a Chief Technologist who has extensive experience in architecting several Big Data solutions. They should be able to communicate in business as well as technology language. Their expertise should range from infrastructure to application development and have command of Hadoop technologies.

I am using Apache Spark for the data transition from databases. We have customers who have one database as a data lake.

The most valuable feature of Apache Spark is its ease of use.

Apache Spark can improve the use case scenarios from the website. There is not any information on how you can use the solution across the relational databases toward multiple databases.  

I have been using Apache Spark for approximately 18 months.

Apache Spark is stable.

We are using Apache Spark across multiple nodes and it is scalable.

We have approximately five people using this solution.

The technical support from Apache Spark is very good.

I rate Apache Spark an eight out of ten.

Streaming telematics data.

It's a better MR, supports streaming and micro-batch, and supports Spark ML and Spark SQL.

It supports streaming and micro-batch.

Better data lineage support.

• RDDs
• DataFrames
• Machine learning libraries

Faster time to parse and compute data. It makes web-based queries for plotting data easier.

It needs to be simpler to use the machine learning algorithms supported by Octave (example polynomial regressions, polynomial interpolation).

I've been using it for one year.

There have been no issues with the deployment.

There have been no issues with the stability.

There have been no issues with the scalability.

We still rely on user forums for my answers. We do not use a commercial product yet.

The initial set-up was easy. I have not explored using this on AWS clusters.

We did an in-house implementation and development for our regression tool.

The ROI will be an in-house product to do machine learning analytics on data obtained from customer.

We did not evaluate any other products.

It's easy to use and has a learning curve.

Our use case for Apache Spark was a retail price prediction project. We were using retail pricing data to build predictive models. To start, the prices were analyzed and we created the dataset to be visualized using Tableau. We then used a visualization tool to create dashboards and graphical reports to showcase the predictive modeling data.

Apache Spark was used to host this entire project.

The processing time is very much improved over the data warehouse solution that we were using.

The most valuable features are the storage engine, the memory engine, and the processing engine.

I would like to see integration with data science platforms to optimize the processing capability for these tasks.

I have been using Apache Spark for the past year.

We have not been in contact with technical support.

The initial setup is straightforward. It took us around one week to set it up, and then the requirements and creation of the project flow and design needed to be done. The design stage took three to four weeks, so in total, it required between four and five weeks to set up.

I would rate this solution an eight out of ten.

We primarily use the solution to integrate very large data sets from another environment, such as our SQL environment, and draw purposeful data before checking it. We also use the solution for streaming very very large servers. 

It is a very fast solution. It's very easy to use. There are many RPis with many languages like Scala, Java, R, and Python. The greatest advantage of Spark is that we can initiate many kinds of analytics including SQL analytics, graphics analytics, etc. 

The solution needs to optimize shuffling between workers.

I've been using the solution for four or five years.

The solution is very stable.

The solution is scalable. My understanding is version 3.0 has renewed scaling capabilities and will be able to do so automatically.

Apache is an open-source platform so there is no technical support.

We use both on-premises and public and private cloud deployment models. We're partners with Databricks.

I'm a consultant. Our company works for large enterprises such as banks and energy companies. 17 of our workers use Apache Spark.

With the cloud, there are many companies that integrate Spark. Most projects in big data around the world use Spark, indirectly or directly. 

I'd rate the solution eight out of ten.

Used for building big data platforms for processing huge volumes of data. Additionally, streaming data is critical.

It provides a scalable machine learning library so that we can train and predict user behavior for promotion purposes.

Machine learning, real time streaming, and data processing are fantastic, as well as the resilient or fault tolerant feature.

I would suggest for it to support more programming languages, and also provide an internal scheduler to schedule spark jobs with monitoring capability.

The most valuable feature is the Fault Tolerance and easy binding with other processes like Machine Learning, graph analytics. The community is growing and hence executing ML in a distributed fashion is quite good.

Previously we were using Hadoop MapReduce to reduce the Google Ngrams (3TB), which took us approximately five days on our cluster. After using Spark, we were able to accomplish this task within hours.

This product is already improving as the community is developing it rapidly. More ML based algorithms should be added to it, to make it algorithmic-rich for developers.

Two and a half years.

No, I did not encounter any problems with the stability. It is also quite backwards compatible.

No I did not as of now, it is quite scalable. Using simple scripts you can add as many workers as you want.

This is a very good product for the big data analytics and integrates well with other parts like Machine Learning and graph analytics.