We mostly provide cloud data warehousing platforms for major banks in Canada. What we're trying to do is to create a standard platform environment that is compliant with the regulatory requirements imposed by the government and financial overseeing institutions for the banks. We help them to onboard the lines of business to these platforms and migrate the existing workloads to the cloud platforms.
The ability to scale out services on-demand and scale them down when they are not required is most valuable. You are in control of your expenditures, and you are also in control of the horsepower that you need. That's a major advantage.
The major challenge that we're seeing with Azure Synapse is around security concerns. The way it is working right now, it has Managed VNet by Microsoft option, similar to the implementation of Azure Databricks, which may pose a concern for financial institutions. For managed environments, the banks have very strict policies around data being onboarded to those environments. For some confidential applications, the banks have the policy to encrypt it with their own key, so it is sort of like Bring Your Own Key, but it is not possible to manage the resources with Microsoft or Databricks, which is probably the major challenge with Azure Synapse.
There should be more compatibility with SQL Server. It should be easier to migrate solutions between different environments because right now, it is not really competitive. It is not like you can go and install SQL Database in some other environment. You will have to go through some migration projects, which probably is one of the major showstoppers for any bank. When they consider Synapse, they not only consider the investment in the actual service; they also consider the cost of the migration process.
When you scale out or scale down your system, it becomes unavailable for a few minutes. Because it is a data warehouse environment, it is not such a huge deal, but it would be great if they can improve it so that the platform is available during the change of configuration.
I have been using this solution for two years.
Generally, it is stable. We all heard about the Active Directory issue earlier this week, but it was not related to Synapse. It was related to the Azure platform.
Its scalability is good. The only thing is that when you scale out or scale down your system, it becomes unavailable for a few minutes. Because it is a data warehouse environment, it is not such a huge deal, but it would be great if they can improve it so that the platform is available during the change of configuration.
We're working with banks, and they have great support because Microsoft has multiple representatives closely monitoring each account. Whenever there is an issue, they're being proactive. They're making a lot of money out of it. Most of the banks, on average, spend between 30 to 50 million a year on Azure. They're pretty large accounts, and Microsoft has dedicated people supporting everything related to Azure.
We deliver platforms to different banks. Some of the banks go with Synapse, and some of the banks go with Snowflake. Overall, these are two major alternatives available right now.
There are multiple differences in terms of the support of different workloads. When one of the banks made a decision to go with Snowflake, the major reason for it was the support for the multi-cloud environment. The major pro of Synapse is the service, and the major con is that when you decide to move out of Synapse, you would have to rewrite the entire thing, whereas, with Snowflake, it would be just simple migration to different cloud providers.
It is relatively straightforward as long as you understand what you're doing.
You don't really need to maintain it. That's the entire point of the cloud. You pay for it to be maintained.
We do deal with monitoring and other similar things, but most of the activities are automated. Overall, it doesn't require a lot of labor around it. We're delivering the platform as infrastructure as a cloud, so everything is going through the pipeline.
It goes by the usage, and there are some limits. Synapse goes by particular pricing, and it is expensive. Both Azure Synapse Analytics and Snowflake are pretty expensive. They don't have standard pricing. They deal with each customer differently.
For working with Synapse, you need to have an understanding and knowledge of the product to take full advantage of it. Synapse has a lot of features in terms of scalability, such as resource management, distribution, and partitioning. There are a lot of things that you need to consider when you go for it. It is not a simple database that you put in there, and it is running itself.
I would rate Microsoft Azure Synapse Analytics a seven out of ten.
BigQuery is a PaaS solution. There's only one version available on Google Cloud. Because it's deployed on cloud, it will update automatically.
If I'm collaborating with Google Data Cloud, I can use the cache, and I don't have to pay again and again. There are some performance features like partitioning, which you can do based on an integer, and it improves the performance a lot. There's also the Array function. You can also enable Spark on BigQuery, which is actually faster than any other Spark. If you use Dataproc, Spark on BigQuery is much faster.
Spark will actually eliminate the usage of a lot of Adobe legacy things. It will act as a Spark SQL.
It is not that cost-friendly, but it is very performance-friendly. There are also machine learning features.
For example, if I have a query, and I have done everything to improve it, the query will still take 15 minutes. With other columnar databases like Snowflake, you can actually increase your VM size or increase your machine size, and you can buy more memory and it will start working faster, but that's not available in BigQuery. You have to actually open a ticket and then follow it up with Google support.
I have been using this solution for two and a half years.
BigQuery is very stable. It is getting used a lot.
It is definitely scalable. You do not have to do any configurations. It will be able to handle petabytes of data.
Technical support is excellent. It is Google, and they always provide the best. We haven't needed to contact Google for BigQuery specifically, but I have contacted Google support for other things and they were pretty responsive.
I have experience with Snowflake.
I was working on a project where we were building systems and loading the data manually. Once we moved to BigQuery, we saw ROI in terms of cost savings. We saw 27% performance improvement in most of our queries. Our total costs were reduced by about 17%. In terms of cost and time, we were able to save effort.
There was some learning and training involved, which lasted six months, so we saw the real ROI after a year.
I would rate this solution 8 out of 10.
My advice is to first identify your use case. If you have Google Cloud then you have two databases to compare, BigQuery and Snowflake. BigQuery is typically used to analyze petabytes of data. If you're looking for transitional query, then you should have a different system. BigQuery cannot handle unstructured data, so that is one thing you have to think about.
In terms of latency, if you want single-digit millisecond latency then BigQuery is not good. It is very fast, but if you want single-digit millisecond latency, then you probably have to go to a no-SQL database solution.
My suggestion is to analyze your use case and then map it with the BigQuery features.
Redshift is a managed service for data warehouses.
I find the most valuable features to be the MPP style of processing, which mostly all of the data warehouses provide. The ability to integrate all other AWS services, such as NSS and S3, with little effort is very helpful. The service is well maintained, there are update patches frequently.
We recently moved from the DC2 cluster to the RA3 cluster, which is a different node type and we are finding some issues with the RA3 cluster regarding connection and processing. There is room for improvement in this area. We are in talks with AWS regarding the connection issues.
In an upcoming release, I would like to have a Snowflake-like feature where we can create another cluster in the same data warehouse, with the same data. You can create a different cluster and compute nodes for each of your use cases, for retail, and for your data analyst all while keeping your underlying data safe.
Additionally, the cluster resize process takes down the cluster for too long, approximately 15 minutes. There are limitations to the size, you can resize only by a multiplier of two, for example, if you have four nodes then you can either go to eight nodes or you can come down to two nodes. There should be fewer limitations.
I have been using Amazon Redshift for approximately one year.
There are three people in my organization using this soltuon.
Their technical support is good. They provide assistance when we need it but since we were experiencing a connection issue it was taking longer to get a resolution. We had to involve a vendor to get it resolved for us.
I have previously used ClickHouse.
The installation was easy.
The price of the solution is reasonable. According to the RA3 cluster particularly, it provides 128 GB of storage with only four nodes. If you can manage your computations processes with the help of materialized views and proper queries. I think the IP clusters are very useful and overall fair for the price.
We have been evaluating Snowflake and are in the POC phase. If it passes our quality tests then we will be moving to it soon.
I rate Amazon Redshift a seven out of ten.
We used to work on business intelligence projects, so we really enjoyed the service-like performance. It's been great.
The business intelligence is very good.
We have found that the analytics environment is excellent.
We've discovered that the solution is quite a complex product, which can make it difficult to sell. Snowflake, for example, is simpler and therefore an easy sell.
The customization can sometimes be difficult to achieve.
We've been working with the solution for a very long time. It's been more than 15 years personally and at least 15 years at this company, so it's been well over a decade at this point.
We don't deal with bugs or glitches and the product doesn't crash or freeze. The only real issues we come across are related to customization. It's quite stable. We haven't had problems.
We have about 100 people using the product currently.
Our clients are quite sizeable companies.
I've never used technical support and therefore can't really comment on their responsiveness or level of knowledge. I have an internal team that assists with troubleshooting as necessary.
As a consultancy, we work with a variety of products. This includes Oracle and Snowflake, for example.
We are an integrator of the solution. Our clients are sizeable companies, and there are several providers involved in different projects with Exadata. We're not the only administrators.
I personally haven't handled an installation. We're mostly focused on projects related to BI and analytics. Someone else handles the setup and we just use the product or have our clients use it. I don't have a sense of whether the implementation is difficult or complex or how long deployments take.
We are a consultancy. We are partners with Oracle, with Snowflake, and with other vendors, software vendors.
We work in different deployment models with our customers. We try to fit our customers' needs. It depends on the customer and the project in terms of which deployment model we'll recommend or use.
Overall, I would rate the solution a nine out of ten. It's very good. We've had a great experience using it over the years.
We use SAP Business Warehouse to host the data. We are hosting financial services data.
I like that it integrates to open-source.
SAP Business Warehouse could integrate better with other ETL tools.
There is some complexity to the setup. It would be helpful if the setup could be simplified.
I would like to see more MLOps capabilities included.
I have been using SAP Business Warehouse for one year.
We are using a previous version.
We did not experience any bugs or glitches with SAP Business Warehouse. It's a stable product.
SAP Business Warehouse is scalable, but you need to understand the product well. It requires some expertise.
We have five people in our organization who use this solution.
Technical support is good. They were able to assist us in some specific issues we were experiencing.
I have some experience with Snowflake.
We did not previously use another solution, this was a new implementation.
The initial setup can be a bit complex. It's not easy, but not overly complex, It's medium.
It's moderately expensive, but definitely has its branding and price value for it.
SAP Business Warehouse is a good option if you are interested in memory analytics including HANA.
I would rate SAP Business Warehouse an eight out of ten.
