Azure Stream Analytics Reviews

I have worked on many use cases involving IoT platforms over the last 10 years. For instance, with a large client in the U.S., I helped track vehicles for Paccar and other companies such as Amazon, URS, or UPS. They monitor many vehicles traveling across the U.S., capturing data regarding vehicles' problems or accidents. This information is processed using Kafka or Event Hubs and pushed to our platform for analytics, where report tools such as Power BI visualize data.

For example, if any telemetry or real analytics data or streaming data comes from IoT, Azure provides the combination of Event Hubs and Streaming Analytics, and in open source, Kafka is used. For inter-system communication, the pub and sub model allows some services to publish data to the queue or topic in any services such as RabbitMQ, while Azure offers Service Bus. If you want to use real-time telemetry data, capturing things such as share market data or vehicles on highways, Event Hubs is recommended to pull that data. We can validate basic data schema using Azure Stream Analytics, determining if data processing should succeed or fail; if it succeeds, we process it with our platform, specifically our ADLS Gen2. For any basic validation that fails, another pipeline can be created for further investigation. Azure Stream Analytics can be utilized on top of Event Hubs, and we can achieve storage along with serverless Azure Function services for internal processing, or use open-source tools such as Kafka if needed.

I widely use AKS, Azure Kubernetes Service, Azure App Service, and there are APM Gateway kinds of things. I also utilize API Management and Front Door to expose any multi-region application I have, including Web Application Firewalls, and many more—around 20 to 60 services. I use Key Vault for managing secrets and monitoring Azure App Insights for tracing and monitoring. Additionally, I employ AI search for indexer purposes, processing chatbot data or any GenAI integration. I widely use OpenAI for GenAI, integrating various models with our platform.

I extensively use hybrid cloud solutions to connect on-premise cloud or cloud to another network, employing public private endpoints or private link service endpoints. Azure DevOps is also on my list, and I leverage many security concepts for end-to-end design. I consider how end users access applications to data storage and secure the entire platform for authenticated users across various use cases, including B2C, B2B, or employee scenarios. I also widely design multi-tenant applications, utilizing Azure AD or Azure AD B2C for consumers.

Azure Stream Analytics reads from any real-time stream; it's designed for processing millions of records every millisecond. They utilize Event Hubs for this purpose, as it allows for event processing. After receiving data from various sources, we validate and store it in a data store. Azure Stream Analytics can consume data from Event Hubs, applying basic validation rules to determine the validity of each record before processing.

There is a need for improvement in reprocessing or validation without custom code. Azure Stream Analytics currently allows some degree of code writing, which could be simplified with low-code or no-code platforms to enhance performance.

The purpose is to handle fast-moving data. For example, Azure Stream Analytics processes more data every second, which is why it's recommended for real-time streaming.

The service itself allows scaling, with capabilities to process data across multiple regions effectively, which is crucial for applications demanding constant monitoring, such as healthcare or financial services.

Having worked with Azure for 23 years, I find the support excellent due to abundant training materials and resources available globally, making it easier for adaptation.

Positive

Azure charges in various ways based on incoming and outgoing data processing activities. Choosing between pay-as-you-go or enterprise models can affect pricing, and depending on data volume, charges might increase substantially.

Kinesis is equivalent to Event Hubs in the AWS ecosystem, where Lambda functions are akin to Azure Functions, and S3 buckets compare to Blob Storage. While there are many competitors, Azure remains my preferred choice based on client needs.

I am still using Microsoft Azure Blob Storage as well as other 200 services; out of 200 services, mostly 50 to 60 services I'm using.

I used to use the Blob Storage for page blob or block blob; there was no requirement for using File Storage. Blob Storage is widely used for storing any binary format such as files or any images.

Blob Storage supports only binary format and lacks hierarchical namespaces. However, Azure Data Lake Service Gen2 supports hierarchical namespaces, accommodating various file formats including PDF or JSON. We can process it using PySpark, Scala, or SQL, and it allows data storage in multiple formats.

Azure Stream Analytics is entirely cloud-based and does not support any on-premise setup. For on-premise solutions, Apache Kafka could be considered as an alternative.

All services are platform services and can follow a pay-as-you-go model or enterprise licensing. Setting up involves configuration by our cloud engineering team, including exposing APIs, which facilitates incoming requests to Event Hubs for processing.

Azure services often aim for an SLA of 99.9. By configuring multi-region integrations and ensuring proper routing and load balancing, we can maintain high availability, even during potential failures.

For big data processing, three languages are supported in any data analytics platform, including Scala, SQL, and PySpark. I predominantly use PySpark, which supports big data and SQL, allowing us to validate conditions and filter based on input.

We can apply retry concepts or compensating techniques for Azure Stream Analytics. For instance, if basic validation fails, we can revalidate and reprocess the data through another pipeline. We implement various patterns such as retry patterns or circuit breakers to ensure proper data processing and business validations based on requirements.

On a scale of 1-10, I rate Azure Stream Analytics a 9.

I personally work with Azure Stream Analytics for personal and smaller projects in a testing environment, in a lab environment, inside of this larger project.

Azure Stream Analytics helps with processing fast-moving data in my IoT applications because it's quicker to escalate. We have a very good level of optimization process regarding costs. It's more elastic in our view. This is not totally confirmed, but in the smaller tests we made, this is very possible, and we have experience in Azure. We think it could be very optimized in the future, and we could optimize the performance process because the execution is faster inside of the same resource group, subscription, and overview. It's more integrated.

In terms of optimizations, I think it could be much more reliable in terms of cost-benefit. We have a target for reduction in terms of performance of more than 40%, 50%, eventually more. I'm being very pessimistic.

In terms of costs that we currently have, to not have this stream data implemented yet and fully automatically deployable, it's also a concern. If we don't have it, we have more costs in people to do it manually and to have these demands objectively implemented in a heavier way. If we have done this in different streams platforms, in different clouds, it will be harder to optimize in the time-lapses that we have.

Azure Stream Analytics' event processing impacts my real-time financial transactions, but this is not a straight answer. There are some variables. For example, if you need to calculate some pricing, and you need some variables in that trigger event, you have to have a more delayed process on your side. The development and execution of those transactions could take a little bit longer. I was talking about re-engineering the process because we could receive this from the master data systems, already prepared. We are doing everything on our own, and I want to discharge this to the specific applications that do already that calculation with some algorithms that they have but we do not apply it. Then we can receive it. However, these transactions that occur right now, in terms of high-level understanding and evaluation on my part, are quite responsive to our needs.

Azure Stream Analytics is easier for Databricks. In this ecosystem that we are developing in this project, we use many transformations, and we are catching data from different sources in a particular area of commodities. This convergence in one resource group or even two, with Azure Stream Analytics, is easier and quicker. Everything could be converged in a high-level overview in terms of cost, performance, and scaling up the teams. I think it could be more interesting in that particular area because technology, in terms of performance, has other solutions. But based on these variables that I described, they could take more complexity in terms of organization, in terms of networking, and this could affect performance, not only in terms of the streaming of the data but also in the continuous process of development.

I utilize SQL-based language queries with Azure Stream Analytics, and we are also analyzing and doing filtering in the Databricks process, a batch process that executes very quickly. In this case, we have no filtering analytics in the demand front-end. We already know exactly based on the requests that we need to generate the transaction, and the filtering is almost defined there. For making a catch-up for all the records, if some transactions need conversion exchange ratios because the customer has an invoice in dollars and will pay by euros, in that particular transaction, we need to give filtering to have the total amount of the debt based on the list of invoices that are prioritized by the due date. That filtering is quite quick. It's one of the tests that we made, and it's very easy for the development team.

The built-in monitoring tools in Azure Stream Analytics assist me in maintaining analytics workflows. The team is using Azure Insights for that, and they are quite lean. I am not yet fully involved as I need to mature this process first and try to understand a little bit more about the monitoring. But the team is very familiar, and they always say that they really appreciate it because it's much more effective. The things that I'm concerned about are the accounts and the logs. That is also very important. The threshold to the time lapse of logs that we need is also in discussion with the team, but I'm not so familiar yet.

Regarding integration with Azure Event Hubs and Azure Blob Storage, we have Blob storage. When we receive some requests, we also have some requests that come up with a pipeline based on sharing CSVs and text files. We use Blob storage to receive that. We also have some other systems, target systems that receive some files, and we convert that data. It's a very small percentage, but it occurs.

With the deployment of Azure Stream Analytics, there are many challenges. I am working with a DevOps team that I'm part of as a counselor. I'm working with them because they are working in other parts of the ecosystem, and gradually we are trying to identify all the challenges and how we can address these issues that we have to optimize the process of deployment. There are many variables and requirements, maybe some that are easier to configure in this pipeline, but there are many variables that we need to stabilize or normalize before we could think about the full automatic deployment. Eventually, there will be in one first phase, before maturity, a kind of semi-automatism depending on a human trigger in part of the process.

Regarding technical support for Azure Stream Analytics, it's not good. It's better than Dynamics, which we also use. Azure is better. Sometimes it depends on the skills of the person that is providing the help. Sometimes we receive very simple and awkward answers. Then there is some support person that we catch, and we note their name because they're very knowledgeable. We don't want to encounter the same person again. Because we have a good contract with Microsoft, we try to push the pre-sales in my country to arrange that specific support person. The knowledge base is medium through the site and through the documentation, but sometimes it is very high level.

I have been familiar with Azure Stream Analytics for about one year, one year and a half.

I have knowledge of Microsoft Azure DevOps Server, Azure Stream Analytics, and Azure Synapse Analytics, but we are not using it yet. My rating for Azure Stream Analytics is 8 out of 10.

Currently, I am working extensively with Microsoft Azure, and I also have some experience with Google, AWS, Snowflake, and Databricks. 

I work mainly with the primary vendors in the data landscape at present. I am looking for new partners, and Palantir seems to be a good option. Microsoft Azure is used for many tasks, and I also undertake some projects around OpenAI in Azure and ChargeDpT.

It was easy for me to use from the beginning. I am accustomed to working with Microsoft. It is quite easy for my technicians to understand, and the learning curve is not steep. This is the best aspect for me, along with the native connectors and integration with other Microsoft products. 

It is just a part of React integrators. Sometimes I perform maintenance, but not as a managed service provider. I deal with several systems, like ERPs, CRMs, and so on, to process the data and present it to our management.

There are too many products in the Azure landscape, which sometimes leads to overlap between them. Microsoft continuously releases new products or solutions, which can be frustrating when determining the appropriate features from one solution over another. 

A cost comparison between products is also not straightforward. They should simplify their portfolio. The Microsoft licensing system is confusing and not easy to understand, and this is something they should address. 

In the future, I may stop using Stream Analytics and move to other solutions. I discussed Palantir earlier, which is something I want to explore in depth because it allows me to accomplish more efficiently compared to solely using Azure. 

Additionally, the vendors should make the solution more user-friendly, incorporating low-code and no-code features. This is something I wish to explore further.

I have been using it for approximately two to three years.

My job involves technicians, and I am not directly aware of any stability issues. However, I am sure I have encountered some challenges since such solutions rarely work flawlessly out of the box. They require significant effort and fine-tuning to function effectively.

Scalability is not particularly complex, as far as I know. More personnel might be necessary since the issue often lies in the integration with the rest of your landscape rather than the product itself. 

You need to maintain it, and it's more about evolution to utilize more features or access more information that I may have overlooked initially. 

Maintenance requires a couple of people, however, it's not a full-time endeavor. If full-time dedication is needed, half of a staff member's time should suffice.

I have not encountered any limitations from my point of view. It is possible I might in the future. So far, I haven't experienced any. Any time I needed assistance, they were helpful. My experience has been positive.

Positive

When I purchased the product, it was comparable to other Azure products. From my point of view, it should be cheaper now, considering the years since its release. The Azure solution is better now, and competitors, even within Microsoft, may offer solutions that could make it cheaper.

Palantir is another option. It's a solution I want to explore in-depth because it allows for more efficient operations compared to relying solely on Azure for problem-solving.

I am considering partnering with them for data analytics work. It's something I'm interested in exploring. I may stop using Stream Analytics in the future and move to other types of solutions. 

There are various solutions available now that can cover more needs. I want to explore these options. 

Overall product rating: nine out of ten.

My main use cases for Azure Stream Analytics involve processing telemetry data coming from various devices, specifically sensors from mining.

Predictive maintenance has been one of my use cases to design for with Azure Stream Analytics.

Azure Stream Analytics helps me predict and mitigate potential failures, and one specific example I could provide is monitoring incoming temperatures. I am building messaging applications that look at temperature data from outside, processing that, and providing reporting to my customers to indicate the impact of weather changes. Based on that, they make business decisions on how they will reroute their product deliveries and other related matters.

The best features of Azure Stream Analytics are that it's easy to set up and configure. It's very accurate and uses existing technologies in terms of writing queries, utilizing standard query languages such as SQL, Spark, and others to provide information. It's very easy for people who have experience with these technologies to pick it up and run with it, and it integrates effectively with multiple different technologies such as Power BI and reporting technologies.

Regarding the SQL-like query language of Azure Stream Analytics, I definitely find it beneficial for creating complex queries. The SQL language is very good. In Azure Stream Analytics, it's a mixture of SQL and Spark functionality because it has been slightly rewritten to handle what Azure Stream Analytics is designed to do. In terms of understanding and picking up that language, it's been quite easy if you know SQL or one of the programming languages.

There were challenges with Azure Stream Analytics. When I initially started, the learning curve was difficult because I didn't have knowledge of the service. There's setup time required to get it integrated with different services such as Power BI, so it's not a straight out-of-the-box configuration. Additionally, the security and integration keys need to be managed fairly securely.

I don't have any recommendations currently in terms of improvements for Azure Stream Analytics. I would suggest that they could improve the UI elements and perhaps introduce artificial intelligence into it.

Regarding AI elements, I expect AI to be introduced into Azure Stream Analytics for a couple of reasons. I want developers to develop faster and release to market sooner. Any coding challenges they face can be helped by the implementation of AI within the tool itself, such as using chatbots for assistance, automated testing, and code formatting.

My experience with Microsoft customer support has been very good. They've managed to answer all my questions and provide help in a timely manner.

I am not aware of any support improvements needed at this time.

I did not evaluate any other solutions before working with Azure Stream Analytics for the same purpose. SQL Server, which is also a Microsoft offering, has its own version of analytics that helps with streaming, but I haven't used it extensively. I have done some research on it, but it's an on-premise service as opposed to being a cloud service. The on-premise solution has to be scaled better and costs more because you need to buy additional hardware, whereas in the cloud, you can scale quite effectively.

The initial setup regarding Azure Stream Analytics depends on your requirements. The basic installation and configuration are fairly straightforward. If you want to make changes, it takes longer and becomes more complex. Additionally, as security gets involved, it becomes a bigger consideration.

We did the deployment of Azure Stream Analytics ourselves as we are the consultants.

Regarding the cost of Azure Stream Analytics, I believe the price is reasonable for the tool.

For those who want to use Azure Stream Analytics, I recommend they plan out their services and their use cases very carefully because it's a cloud service and the costs increase the longer they leave it on. They need to do proper modeling and proper architecture before they implement the solution.

I was monitoring and analyzing data from Azure Stream Analytics infrastructure. I do not remember examples of the main use cases for analytics or my most common use cases for this tool.

I used to work with Application Performance Monitoring tools such as Azure Monitor and Elastic Observability, specifically Azure Monitoring tools. Most of the tools that I used or worked with were the built-in ones in Azure Stream Analytics itself. I worked mostly with Azure Stream Analytics infrastructure. The services I worked with in Azure involved networking, security, and development.

It does not always give you the right reason or the correct reason. For example, if a service is stopped, it just tells you that it stopped and started. It does not give you any good insight as to why it has been stopped or what the reason was. Additionally, you cannot build certain analytics; on other monitoring tools, sometimes you can build scripts to identify what went wrong and correct it immediately.

I have been working with Azure Stream Analytics for almost two years.

The solution does not need any type of configuration. It comes as a built-in feature with the services you're using. It's spread all over the services on Azure Stream Analytics, and it's free of charge. Once you build it, you will be charged for the amount of logs that are being submitted or created.

In a case or two, I had some sort of issue with the solution or some sort of questions.

In terms of integration, it just works with Azure Stream Analytics, with Azure services itself. It works with Azure Stream Analytics itself and does not integrate with other tools.

The support on critical issues depends on the level of subscription that you have with Microsoft itself. Their support is very excellent. They understand the case immediately. They start to propose solutions and give you help. If needed, they can work with you and you can connect with them just to explain more. Most of the time, they ended up helping you very effectively.

Positive

I have been looking for a job for the last year, so I am not using any of those tools anymore.

The solution does not need any license; it comes with your subscription.

Most of the tools that I used were the built-in monitoring tools that come with the service itself, or Health monitoring tools. These are basic monitoring tools that do not give you deep insight; they provide basic monitoring to give you an alert if something went wrong.

I worked with Azure Stream Analytics, which gives you some insight about what's wrong in your app services or databases. However, it's not very comprehensive.

It wasn't particularly helpful, but sometimes it can serve the purpose of monitoring to identify the reason behind problems that occurred. It was helpful when failures happened; it was easier to find where they occurred to fix issues.

The data I analyzed for this solution was in NoSQL. It's not a straightforward solution; you need to dig deeper to know what the reason actually is. Some sort of automation and automated response capabilities would be beneficial. Having worked with some good monitoring tools, this is not that helpful as you cannot build analytics and predictions on these types of monitoring.

I rate Azure Stream Analytics 5 out of 10.

Neutral

The critical use case was managing telemetry data, as well as multiple messaging topics coming from various sensors or triggering points, such as logistics events. These were handled to capture real-time use cases from IoT devices.

The most valuable aspect is the SQL option that Azure Stream Analytics provides. These inbuilt solutions offer more flexibility within an enterprise level. If we don't use Stream Analytics, which has features that facilitate direct data transportation to end destinations, we might have to manage complex setups or activities coordination, potentially increasing costs.

More flexibility in terms of writing queries and accommodating additional facilities would be beneficial. The complexity of handling messages that need decoding and contain different characters should be addressed. 

Microsoft should improve networking and programming flexibilities and should be more open to other vendors. There should also be increased tool support from third-party tools.

I have almost four to five years of experience with this solution.

Stability feels 50/50, around five to six on a scale from one to ten. Complex transformations can challenge the system's stability.

I would rate scalability around seven to eight. Azure Stream Analytics, being a SaaS solution, is quite scalable with options like per-usage payment.

Customer service was initially satisfactory. Microsoft's support seems to be dropping in quality due to increased demand.

Neutral

Without Azure Stream Analytics, we used solutions like Event Hub, IoT Hubs, and Databricks with structured streaming. The flexibility provided by structured streaming with Databricks made it preferable. Stream Analytics was lacking some of these flexibilites.

The initial setup is easy for less complex projects. However, as project complexity increases, managing deployments and transformations becomes challenging.

We deployed using a DevOps framework where JSON can be copied or manually pushed to higher resource groups, like production environments.

Stream Analytics is cheaper, especially for small-scale requirements or telemetry needs. However, for enterprise-level data handling, structured streaming with Databricks might be more cost-effective, considering flexibility and availability.

Kafka, Azure Event Hub, IoT Hubs, Databricks with structured streaming.

Depending on your use case, Azure Stream Analytics can be straightforward and easy to learn. For building future-proof solutions, structured streaming is recommended.

I'd rate the solution five out of ten.

I use the solution in my company for real-time analytics on IoT data.

Azure Stream Analytics was not meeting our company's expectations because it was tedious to change the job, write queries, or if I needed to change something, I needed to stop the entire stream processing to change the job so that the changes could take effect. The aforementioned reasons were concerning, but I think that many of the issues related to the product have been resolved with the help of Microsoft Fabric.

The only challenge was that the streaming analytics area in Azure Stream Analytics could not meet our company's expectations, making it a component where improvements are required.

I have been using Azure Stream Analytics for two and a half years. My company has a partnership with Microsoft.

It is a stable solution.

Azure Stream Analytics is a scalable solution.

My company deals with small, medium, and enterprise-sized customers for the product.

The solution's technical support is good. As soon as Microsoft's product team gets involved with the product, the support that our company receives from Microsoft is good. I rate the technical support a ten out of ten.

Positive

I also use Microsoft Fabric.

Azure Stream Analytics is easy to deploy. Other than the streaming analytics part, the rest of the areas in the product were fine. Templates were available in the product for the deployment process. The product's deployment process and connectivity were smooth. Scaling options in the product are good.

The solution can be deployed in a couple of minutes.

The product's price is at par with the other solutions provided by the other cloud service providers in the market.

Against Azure Stream Analytics, I had considered products like Amazon Kinesis and Google Cloud Dataflow.

Azure Stream Analytics for anomaly detection was something that was not meeting our company's expectations, but the new tool within Microsoft Fabric for real-time analytics is really good for even Azure Stream Analytics as it allows me to get alerts and use data activators, so I can take instantaneous actions. Regarding anomaly detection, it is much easier and faster with the availability of an SQL database, which is a real-time database. Within Microsoft Fabric, there is a component called real-time analytics, which consists of multiple tools like Eventstream, KQL database, and data activator.

Speaking about Microsoft Fabirc's features that were valuable for processing large volumes of data in real-time, I would say that our company is able to process a terabyte of data daily in real-time. The scaling part of the is outstanding, and the connectivity between the components is smooth. For the overall experience provided by Microsoft Fabric, I rate the tool a ten on ten if I specifically consider real-time analytics. Within Azure Stream Analytics, real-time analytics was not good, but in Microsoft Fabric, it is.

The product's integration capabilities have always been good since I could integrate multiple sources and ingest data.

Though my company has a maintenance team, the product does not need to be maintained as such. It is when we receive alerts in our company that we check the product. Dedicated maintenance or support is not required for the product.

Learning to use the product is a straightforward and easy process. I find AWS to be a bit confusing compared to Azure Stream Analytics.

Compared to Azure Stream Analytics, Amazon Kinesis, and Google Cloud Dataflow, I find Microsoft Fabric to be the best.

I rate Microsoft Fabric a ten out of ten.

I rate Azure Stream Analytics as seven to seven and a half out of ten.

We use Azure Stream Analytics for simulation and internal activities.

The solution’s customer support could be improved.

I have been using Azure Stream Analytics for more than two years.

Azure Stream Analytics is a stable solution.

I rate Azure Stream Analytics a ten out of ten for scalability.

I rate Azure Stream Analytics a nine out of ten for its ease of initial setup.

Azure Stream Analytics is a little bit expensive.

Overall, I rate Azure Stream Analytics ten out of ten.

We are using Azure Stream Analytics for small to medium size streaming datasets where you would like to flag patterns from the stream. It works well or pairs well with IoT edge scenario use cases that are on Azure. If you have exceptional conditions, such as a sensor being way off the average for the last one to five hours, then you can flag a scenario. It works well with the IoT infrastructure that Azure provides.

We didn't end up using Azure Stream Analytics in production, or for a client, we implemented it. However, Azure Stream Analytics is something that you can use to test out streaming scenarios very quickly in the general sense and it is useful for IoT scenarios. If I was to do a project with IoT and I needed a streaming solution, Azure Stream Analytics would be a top choice.

The most valuable features of Azure Stream Analytics are the ease of provisioning and the interface is not terribly complex.

Azure Stream Analytics could improve by having clearer metrics as to the scale, more metrics around the data set size that is flowing through it, and performance tuning recommendations.

I have been using Azure Stream Analytics for approximately three months.

Azure Stream Analytics is stable.

Azure Stream Analytics can improve the scaling and the connectivity to external datasets.

We are not using this solution extensively and we do not plan to increase usage.

The level of support quality depends on how much you purchased.

I rate the support from Azure Stream Analytics a four out of five.

Positive

The initial setup of Azure Stream Analytics was straightforward. It has a quick startup time and is easy to start.

I did the implementation of Azure Stream Analytics for my client. We have the developer setting the solution up and once it's in production, your infrastructure team can monitor it just like any other solution. Since it's Azure, it has a lot of metrics that allow you to be proactive to flag an issue if there is one.

I have seen a return on investment with Azure Stream Analytics. If you're not doing terribly complex scenarios, this is a quick and fast way to have your streaming pipeline set up. You won't have to invest a lot into its deployment because it's the cloud. You are not paying any upfront capital.

I rate the price of Azure Stream Analytics a four out of five.

I have evaluated other solutions, such as Databricks

Azure Stream Analytics it's good for proofs of concept and for scenarios that are not too complex. It's promising in the future, but if you start to scale out, you might want to consider other scaling solutions, such as Databricks.

Got it. And do you see a return on investment with this one?

I rate Azure Stream Analytics a nine out of ten.