What is our primary use case?
My main use case for CloudAMQP involves using it in various ways because it is a versatile technology, but the two main ones are as a publish-subscribe technology, using stream-type queues, and as FIFO queues for direct and asynchronous communication between two or more microservices.
Between these two use cases, I consider both to be critical for my company's activities, because even if the publish-subscribe, the streaming one, can be considered a secondary use case, it has actually allowed us to cut costs and also reduce the number of microservices that were handwritten, so to speak, basically reinventing the wheel. Perhaps the second is more critical, the FIFO queues, because they made it possible to connect different data pipelines efficiently, particularly using the producer-consumer approach, even more complex ones, avoiding data loss thanks to resilience, and allowing us to scale very effectively even with thousands, almost millions of data points per second.
A real situation where CloudAMQP made a difference in my architecture is in the producer-consumer and subsequent data lake pipeline of our main data ingestion engine. In particular, we did not use RabbitMQ, we used its variant LavinMQ, which is written in Rust, so the compiled language is much faster. In this case, we used the hash-type exchange, which in turn allowed us to distribute the load at the LavinMQ application level across many more data, making it even more scalable. Besides adding resilience, it also made it even more scalable because the producer and consumer were separated. The simpler producer receives the data but must be sure that they are saved and not lost. The consumer, which is obviously heavier, has to be sure that the data it receives is correct and that it is then subsequently handled by the engine that inserts the data into the data lake.
How has it helped my organization?
CloudAMQP has had a positive impact on my organization in terms of efficiency, reliability, and cost reduction. Before working with CloudAMQP we had the AWS solution, specifically we were using SQS only, because streams with SNS were not being used yet. But that had a really high cost. Instead, with AMQP, even without going on-premises yet, we already had benefits in terms of the versatility of the services and in terms of costs.
The versatility and cost reduction have influenced my day-to-day activities and business outcomes significantly. I have seen faster processes and more autonomy for teams, or other tangible benefits. Having a single product to work with was an advantage, because people from various teams only had to know one product rather than, as in this case, SQS and then learn other products to solve tasks that AMQP already solves by itself. This resulted in faster processes, more streamlined operations and, indirectly, also in terms of costs. By reducing costs on one side, we could afford to evaluate other technologies for other purposes that then kept us within budget.
What is most valuable?
The best features offered by CloudAMQP include various aspects that led me to prefer it over other solutions, including managed cloud solutions like SQS, SNS, and Kafka. Above all, the fact that it can be easily managed from the cloud or hosted locally is valuable. At the beginning we started from the cloud, but the price-to-quality ratio was not that high, meaning it is a very good service, of course, but if it has to handle many data points per second, really a lot, we are talking about tens, hundreds of messages per second, then we really have to pay a lot. The same solution is also offered on-premises, so we can, as we did, install it on our Kubernetes cluster and use it easily. Its versatility is also appreciated, because besides this aspect, it also offers many different features. For example, AWS offers you Event Stream, SQS, SNS, so different services that obviously serve their business to earn more. Instead, CloudAMQP offers everything together and so with a single solution, we are able to address many different use cases.
The possibility of implementing CloudAMQP both in the cloud and locally has affected my architectural choices significantly, especially when still talking about the set of microservices that manage data ingestion and data intelligence. Previously, the solution involved using CloudAMQP and microservices that ran on individual servers, so it was more complex to manage, but we did not have to manage the message broker itself. It was actually very straightforward to manage, thanks to the API, thanks to the accounts, virtual hosts, and policies, which are very thoughtfully designed, allowing us to grant access to applications and different people in a secure way. I found this mirrored one-to-one on-premises, and I really appreciated that, because other services heavily restrict on-premises hosting, trying to push the user to work in the cloud. Working on-premises on Kubernetes, I was able to spend much less, the company was able to spend much less while remaining loyal to the product. This is obviously a positive thing, because even if CloudAMQP, or rather the company that offers the service, may not earn directly from us, indirectly it does, because we will always speak positively of this product.
What needs improvement?
CloudAMQP could be improved in some areas, though it seems pretty complete to me, also because there are plugins, so in case of a missing feature, which so far has never happened to me, you can develop them or look for them in this plugin marketplace. Something from a graphical point of view could be updated, which has always remained a bit outdated, but nothing major besides that.
In terms of necessary improvements regarding the user interface or other less obvious aspects, I think it is not very straightforward to take advantage of the charts to understand what happened further back in time, because the visualization really limits the data, showing only the most recent ones. Log management is also not particularly intuitive, which in my opinion should be reviewed.
For how long have I used the solution?
I have been using CloudAMQP for 4 or 5 years, basically since I started working for my current company.
What do I think about the stability of the solution?
CloudAMQP is very stable in my experience.
What do I think about the scalability of the solution?
Regarding scalability of CloudAMQP, I have always had a good experience. The only thing is that if you reach the load limit provided by your subscription, the server stalls and you risk losing data.
How are customer service and support?
In my experience, CloudAMQP's customer support is very fast. We have dealt with them and we have always had quick and precise responses. I rate customer support a 10, since I have never had problems with them.
Which solution did I use previously and why did I switch?
Previously, we were using AWS SQS, but the problem is that it is limited only to producer-consumer architectures and not streaming. Whereas AMQP lets you use both solutions at a very low cost.
What was our ROI?
I have already obtained a return on investment because there has been a time saving by using AMQP, a single solution. I can quantify this time saving as roughly 5 hours out of a 40-hour week.
What's my experience with pricing, setup cost, and licensing?
Regarding my experience with pricing, setup costs, and licenses, I think setup and license costs are not a problem. Regarding pricing, I think it is convenient for managing small amounts of data, but in my opinion it is not convenient as the volume of data increases, because you very easily end up paying hundreds or even thousands of euros per month.
Which other solutions did I evaluate?
Before choosing CloudAMQP, I evaluated other options, looking around at other AWS services in general, but the cost, both in terms of money and the learning curve, was high, and I also looked at Apache Kafka, but that too was not convenient.
What other advice do I have?
I rate CloudAMQP a 9 on a scale from 1 to 10. I give it a 9 and not a higher score because it has many strengths and in my opinion no weaknesses. I would only give a 10 after the graphical aspects are improved in a minor way, but especially when log-related aspects are made a bit more streamlined, without necessarily having to move to other log visualization and aggregation tools such as Splunk.
We use the on-premises solution on a Kubernetes cluster.
For the rest of the architecture, we use both on-premises solutions and public clouds. CloudAMQP, specifically LavinMQ, runs on our Kubernetes cluster.
The cloud provider I mainly use for these public cloud solutions is Amazon Web Services (AWS).
My advice to other people who are considering using CloudAMQP is to first assess why they need it and what volume of messages they have to handle, what the requirements of the application or context are, and then evaluate whether to move forward. I would tell them to definitely go with CloudAMQP.
Which deployment model are you using for this solution?
On-premises
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)