Spark SQL Reviews

I employ Spark SQL for various tasks. Initially, I gathered data from databases, SAP systems, and external sources via SFTP, storing it in blob storage. Using Spark SQL within Jupyter notebooks, I define and implement business logic for data processing. Our CI/CD process managed with Azure DevOps, oversees the execution of Spark SQL scripts, facilitating data loading into SQL Server. This structured data is then used by analytics teams, particularly in tools like Power BI, for thorough analysis and reporting. The seamless integration of Spark SQL in this workflow ensures efficient data processing and analysis, contributing to the success of our data-driven initiatives.

I find Spark SQL's seamless integration of SQL queries with Spark programs and its use of DataFrames and Datasets particularly valuable. While we mostly stick to traditional T-SQL, Spark SQL brings flexibility to handle large-scale data processing. The ability to write SQL queries, even with minor adjustments for functions like LICA, simplifies our data transformation. Although the syntax differs from traditional SQL, Spark SQL's efficiency in managing distributed data and its simplicity in expressing complex operations make it an essential part of our data pipeline.

We are using PySpark for big data processing, like multiple competitors of stock. We process it in in-memory using data frames and Spark SQL. We are using it along with the database to process the big data, especially the special Azure data.

We are using PySpark. Databricks itself provides an environment that is pre-installed with Spark.

We are using Spark SQL in Databricks. There are three ways to code: Task Equals, PySpark, and Scala. Some team members are also using Spark SQL. They were new to Databricks and recent graduates, but they knew SQL and were able to code in Spark SQL. It's a great tool to use altogether because they were able to learn along with it. They were implementing some basic stuff.

We have an HDFS environment for archiving data when there is an enormous volume of data, and the solution helps retrieve data from our HDFS archive. Developers use the solution for business analytics.

One of Spark SQL's most beautiful features is running parallel queries to go through enormous data. We can run queries in parallel and retrieve more data and results in aggregation. Spark does this faster than Python.

We used the solution for analytics of data and statistical reports from content management platforms.

I find the Thrift connection valuable.

Spark SQL has enabled us to perform our data preparation tasks within our analytical code. 

Spark SQL gives us a handful of methods to design queries based on its own syntax and also incorporates the regular SQL syntax within tasks. It's also a good tool for querying and depicting data. It's very good for us and it's helpful that we have access to a lot of good documentation. 

We do have some use cases, like analysis and risk-based use cases, that we've provided and prepared for companies in order to evaluate, but not many. The business units have so many things that we don't know how to help formulate into another tool and utilize as a use case. They also have so many requirements and costs.

I work for a financial institution, so every solution that they need to consider has to be on-premise.

I'm actually just evaluating and up scaling my skill sets with this solution right now.

The speed of getting data, as our TBs are big and it's a lot of data. 

We use this solution for data engineering, data transformation, repairing data for machine learning and doing queries. We have between 30 and 40 users making use of this solution.

Certain data sets that are very large are very difficult to process with Pandas and  Python libraries. Spark SQL has helped us a lot with that.

We primarily use the solution as our data warehouse. We use it for data science.

Overall the solution is excellent.

The solution is continuing to evolve and mature over time.