I employ Spark SQL for various tasks. Initially, I gathered data from databases, SAP systems, and external sources via SFTP, storing it in blob storage. Using Spark SQL within Jupyter notebooks, I define and implement business logic for data processing. Our CI/CD process managed with Azure DevOps, oversees the execution of Spark SQL scripts, facilitating data loading into SQL Server. This structured data is then used by analytics teams, particularly in tools like Power BI, for thorough analysis and reporting. The seamless integration of Spark SQL in this workflow ensures efficient data processing and analysis, contributing to the success of our data-driven initiatives.

We are using PySpark for big data processing, like multiple competitors of stock. We process it in in-memory using data frames and Spark SQL. We are using it along with the database to process the big data, especially the special Azure data.

We are using PySpark. Databricks itself provides an environment that is pre-installed with Spark.

We have an HDFS environment for archiving data when there is an enormous volume of data, and the solution helps retrieve data from our HDFS archive. Developers use the solution for business analytics.

We use this solution for data engineering, data transformation, repairing data for machine learning and doing queries. We have between 30 and 40 users making use of this solution.

Our company uses the solution to create pipelines and data sets. The ETL process transforms the data and certain written aggregations convert the raw data to data sets. The data sets are then exported to tables for dashboards. 

We used the solution for analytics of data and statistical reports from content management platforms.

The primary use case of this solution is to function within a distributed ecosystem. Spark is part of EMR, a Hadoop distribution, and is one of the tools in the ecosystem. You are not working with Hadoop in a vacuum—you leverage Spark, Hive, HBase—because it is just a distributed ecosystem. It has no value within itself. 

This solution can be deployed both on the cloud and on Cloudera distributions. 

I am using this solution for data validation and writing queries.

We use it to gather all the transaction data. We have Hadoop and Spark in our system, and we use some easy process flows for transport. 

Our primary use case is for building a data pipeline and data analytics. 

We do have some use cases, like analysis and risk-based use cases, that we've provided and prepared for companies in order to evaluate, but not many. The business units have so many things that we don't know how to help formulate into another tool and utilize as a use case. They also have so many requirements and costs.

I work for a financial institution, so every solution that they need to consider has to be on-premise.

I'm actually just evaluating and up scaling my skill sets with this solution right now.

We primarily use the solution as our data warehouse. We use it for data science.

The primary use is to process big data. We were connecting into and we were applying sentiment analysis via hardware.