PubSub+ Platform Reviews

We use PubSub+ primarily for long processing messages. For instance, we generate customer forms, which take a longer time. We place these requests in queues, process them, and return them to the queues. This ensures that messages are not lost due to the time-consuming nature of such tasks. We also use PubSub+ for audit trails where we intercept requests, apply them, and store them in MongoDB, placing them in the queue before final processing.

PubSub+ has enhanced our asynchronous mode processing, ensuring that messages are not lost and requests are kept until fully processed. This means that messages do not get lost in the network or time out, thus improving reliability and efficiency.

We do a lot of pricing data through here, market data from the street that we feed onto the event bus and distribute out using permissioning and controls. Some of that external data has to have controls on top of it so we can give access to it. We also have internal pricing information that we generate ourselves and distribute out. So we have both server-based clients connecting and end-user clients from PCs. We have about 25,000 to 30,000 connections to the different appliances globally, from either servers or end-users, including desktop applications or a back-end trading service. These two use cases are direct messaging; fire-and-forget types of scenarios.

We also have what we call post-trade information, which is the guaranteed messaging piece for us. Once we book a trade, for example, that data, obviously, cannot be lost. It's a regulatory obligation to record that information, send it back out to the street, report it to regulators, etc. Those messages are all guaranteed.

We also have app-to-app messaging where, within an application team, they want to be able to send messages from the application servers, sharing data within their application stack. 

Those are the four big use cases that make up a large majority of the data.

But we have about 400 application teams using it. There are varied use cases and, from an API perspective, we're using Java, .NET, C, and we're using WebSockets and their JavaScript. We have quite a variety of connections to the different appliances, using it for slightly different use cases.

It's all on-prem across physical appliances. We have some that live in our DMZ, so external clients can connect to those. But the majority, 95 percent of the stuff, is on-prem and for internal clients. It's deployed across Sydney, Singapore, Hong Kong, Tokyo, London, New York, and Toronto, all connected together. We are currently working on the cloud setup to connect on-prem with cloud based virtual message routers all connected together.

With the old platforms we were coming from, if we wanted to make changes, some of those changes were intrusive to make. For example, to add a new application into the environment, we would have to make a change that might cause some disruptions to the environment. We only have very limited downtime for our environment on a Saturday after midnight and before midnight again on Sunday. That is our only change-window for the week, if we have to do something intrusive. That limited us to when we could truly make changes. On a lot of other vendors' platforms, to add things, you've got to restart components and cause disruption. 

The benefit of Solace is that we can add an application in the middle of the day, with no disruption to anyone. It's purely based on our access-control list and permissioning. We can add an application in with zero disruption. We can onboard applications during the middle of our business day. It's still under change control, but there's zero impact by doing it. For us, that is super-powerful. Whether we're adding users or adding applications, we can do it, without causing any disruption. For a lot of other products, that's not the case. That's been a huge win for us.

In terms of application design, I've seen applications go live in less than a week, from coding the first line of code to putting something into production. It depends on how complex the application is. We have a central team where we support the wrappers on top of the vendor's API and we have some example code bases where we show a simple application built using our wrapper on top of Solace's API. A developer who joins our company knowing nothing about Solace, can walk through our documentation, have a look at our wrappers, take some of our example code, and get up and running and off to the races pretty quickly. Getting up to speed is definitely not difficult.

We might get a new user in our bank who is familiar with other messaging systems and who has preconceived ideas on how they want to do things. They might ask us, "How do I get access to this messaging system that I used to use with my old organization? That's what I'm familiar with." Sometimes we have to do sessions with those people and say, "Okay, we're familiar with the systems you're talking about. We supported them in the past. Talk us through what your use case is, what it is you are trying to achieve." Once they explain their use cases, we can say, "Okay, great. We actually have this and here's some example code and this how to do it." Within a day, that person has gone from knowing nothing about it to saying, "Okay, you're, absolutely meeting my application needs and now I'm educated on how this works." They're off and running very quickly.

We take all kinds of data onto the environment to share. Because the event bus is the place that every application always needs to start, they're no longer building an application now within the capital markets organization without putting their data onto our bus in some way. It's definitely a way of lowering the barrier to sharing data and getting things up and running quickly. Similarly, they can take data from other teams, once they find out what's available. Someone might say, "I need all the FX prices in the bank. Oh, I can just subscribe them from here. I don't even need to talk to the FX team." Teams can get up and running very quickly without having to spend a lot of time working with other groups to make that happen.

By having all of that data together in one place, Event Broker has definitely reduced the amount of time it takes to get a new application onboarded. We came from a place with six or seven different systems, where we might have bridged some of those together in some way, but it wasn't one common environment. Now, we've got application A that comes online and starts putting that data out for application B to get up to speed and to start looking at that data. That is very quick and easy for us to do. All the messaging that we do is self-describing. They can look at the payload of a message and understand it without even needing to talk to the upstream application. We can have applications starting to look at data where they didn't even have to speak to the upstream application. We've gone from 8 x 1 Gig, 10 years ago, to 8 x 10 Gigs today, and the reason for that is because we keep putting more and more data and applications on here. That continues to grow exponentially. If it wasn't easy to do, the data wouldn't be going up and we wouldn't have all these applications now on here. It's hard for me to say it has definitely increased the productivity, because I don't own the application development piece but, anecdotally, I would say it has.

Another area of benefit is that we're in the process of containerizing all of our applications at the moment, whether they'll be run on-prem or in the public cloud. The underlying piece is that these containers, wherever they run, are going to need to share data between the different applications and then back to the users. The Solace event mesh or event brokers are the underlying lifeblood among all of these containers. They need to have some way of communicating with each other and we see solace as being that connection among all of them. All the different cloud environments have their own messaging and we don't want to build applications that are specific to any one cloud; we want to be cloud-agnostic. To do that, we need to have a messaging system that is equally agnostic. Given that we already have a huge investment on-premise for all of our Solace stuff, we see that the future of containerizing our applications goes hand-in-hand with our messaging strategy around Solace, so we can be totally cloud-agnostic.

Technology, in the last 10 years, has probably become a lot more stable generally, but I can say that with the amount of data we put through these appliances, and route globally every day, if our environment was down capital markets wouldn't be operating for the bank. That's how critical it is. We can't afford to have any issues. At the same time, literally no application can run in our front office without this. If I look back 10 years ago, we might have had six or seven different distributed systems, all with their own problems. Now that we've consolidated all that, there's a huge efficiency by sharing all our data between the different groups. It means we can get up to speed very quickly, but also, what we're enabling from a business perspective, by sharing 120 billion messages a day, is hugely valuable to our front office.

We handle publishing/emitting events to the PubSub+ Broker via one protocol, and it seamlessly makes the event available for multiple consumers with different protocols without writing code is one of the primary use cases for us. 

As an architect, I wanted to implement an event-driven architecture, majorly focusing on a cloud solution with a PubSub pattern. 

Out of curiosity, our CoE team explored Solace PubSub+, and after several months of testing with our requirements, this was the kind of solution we needed.

Previously, we used on-prem Kafka for event distribution. The Kafka cluster was self-managed and hence required a dedicated team and infrastructure to manage. We wanted a cloud solution, and Kafka was already available as a SaaS offering, yet we wanted a better and more compatible solution with respect to protocols and security. Hence, we opted for PubSub+ with the support of AMQP, JMS, MQTT, and HTTPS. Without writing code, we were successful in creating event-driven microservices; parallelly, administering the cluster was not our responsibility anymore.

We are using PubSub+ Event Broker for specific application processing. We have used the solution In banking systems.

The speed of the throughout has been helpful for our organization.

PubSub+ Event Broker is used to help manage architectures, such as real-time integrations of business processes and systems.

The most valuable feature of PubSub+ Event Broker is the scaling integration. Prior to using the solution, it was done manually with a file, and it can be done instantly live.

There was a challenge in the market that needed addressing. While some tools could serve as event managers, they were not proper event brokers. For instance, Kafka is referred to as an event broker but not a proper one. If you want to use it as an event broker, you would have to implement your event manager, which could be quite complex. The same can be said for Kafka, a text-based broker that stores events in a text file on disk and then consumes them. In other words, an event is just a message stored in a text file that is not reactive. If you introduce events into the system, you cannot react to them in any way. It creates a problem that needs to be addressed by implementing tools that can react to events or pre-defined topics. The way topics are created is also an issue. For instance, if you want to consume a specific topic, you have to create a new one, and you cannot filter events using the mechanism provided. If you wish to query events, there is no provision.

It is where PubSub+ comes in. It provides the option to query events and route messages from one topic to another, as well as clients, to facilitate this process. We primarily use PubSub+ for event-driven applications, where we react to and process applications based on those events. For instance, when we receive an order, we react to that event and create multiple other events based on it. This reaction is based on events, so we use PubSub+ Event Broker.

We mostly had a few teams, which were prototypes because we were planning for large-scale usage. But now, some domains have around fifty to sixty developers using PubSub+ Event Broker, and that's the teams using it.

The first use case is technology operations tools. We are a best of breed monitoring shop. We have all kinds of tools that monitor things, like storage, network, servers, applications, and all types of stovepipes that do domain specific monitoring. Each one of those tools was sold to us with what they called a single pane of glass for their stovepipe. However, none of the tools are actually publishing or sharing any of the events that they have detected. So, we have been doing a poor job of correlating events to try and figure out what's going on in our operations. 

Our use case was to leverage that existing investment. For about a year, we have been proving that we can build publishing adapters from these legacy monitoring tools which are each valid in their own right, like storage monitoring tools, network monitoring tools, and application monitoring tools (like Dynatrace), and more modern than other ones. We have been building publishing adapters from those things so we can transport those events to an event aggregation and event correlation service. We're still trying to run through our list of candidates for what our event correlation will be, but the popular players are Splunk, Datadog, and Moogsoft, then ServiceNow has its own event management module. 

From an IT systems management perspective, our use case is to have a common event transport fabric that spans multiclouds and is WAN optimized. What is important for me is topic wildcarding and prioritization/QoS. We want to be able to set some priorities on IT events versus real business events. 

The second use case is more of an application focus. I'm only a contributor on the app side. I'm more of an infrastructure cloud architect and don't really lead any of the application modernization programs, but I'm a participant in almost all of them. E.g., we have application A and application B side by side sitting in our on-prem data center, and they happen to use IBM MQ Hub to share our data as an integration. Application A wants to move to Azure. They are willing to make their investment to modernize the app, not a forklift, but some type of transformation event. Their very first question to us is, "I need to bring IBM MQ with me because I need to talk to app B who has no funding and is not going to do anything." Therefore, our opening position is, "Let's not do that. Let's use cloud-native technology where possible when you're replatforming your application. Use whatever capability you have for asynchronous messaging that Azure offers you. Let's get that message onto the Azure Event Hub. Don't worry about it arriving where it needs to arrive because we'll have Solace do some protocol transformation with HybridEdge, essentially building a bridge between the Azure Event Hub and MQ Hub that we have in our data center."

The idea is to build bridges between our asynchronous messaging hubs, and there's only a small handful of them, where Azure Event Hub is the most modern. We have an MQ Hub that runs on a mainframe and IBM DataPower appliances that serve as our enterprise service bus (ESB). Therefore, if we build bridges between those systems, then our app modernization strategy is facilitated by a seamless migration to Azure.

The most recent version is what we installed about three weeks ago.

The solution is deployed on Azure for now. We will be standing up some nodes in our on-prem data centers during next phase, probably in the next six months. 

The plan is to use event mesh. We're not using it as an event mesh yet, as we are only deployed with Azure. We want to position a Solace event mesh for enterprise, but we're just now stretching into Azure. We're a little slow on the cloud adoption thing. We've got 1200 applications at CIBC with about four of them hosted in clouds: one at AWS and three at Azure. So, we're tiptoeing into Azure right now. We're probably going to focus our energy on moving stuff into Azure. However, for now, because the volume is so low on stuff that's outside of our data center, the concept of a mesh has been socialized. There's not a ton of enthusiasm for it, even though I might be shouting from the rooftops saying, "It's a foundational capability in a multicloud world." It looks like we're putting that funding on the back burner for using it as an event mesh.

This solution has increased our application design productivity compared to other solutions. There is a ton of momentum in our application development space for leveraging Dynatrace with Solace's monitoring tool. We have made the investment in getting Dynatrace to publish events that it detects, mostly application performance related events. The app development teams have taken a liking to implementing the application monitoring tool early in their development cycles, maybe not in development, but in their performance testing cycles. We can practice what a code drop stack shift would look like if they're shifting from stack A to stack B or if they're doing rolling reboots on some of their app servers as they're doing upgrades. We get to exercise that and see what the monitoring patterns look like correlated with servers going up and down along with web services coming up and down. That's been helpful to the development community to see that automation occur in mid-environments.

There have been quite a few incidents where a test infrastructure has become unavailable because of some change going on and the app developers aren't on the nut for fixing the problem in the UAT environment because they know the outage was caused by the fact that they did a code drop 20 minutes earlier, which is a legitimate server outage. We are seeing some benefit, but it's more of an optimization of incident management resources. E.g., we have somewhere between five and 10 Internet-facing applications, and when something goes bump on the firewall that's behind them, we have DMZs (or different zones) where we put our web tier and app tier. Therefore, when something goes bump in our network tier, we got 10 application teams that are all fired up, and say, "What's going on?" Then, they all spin up their own tech bridges. Meanwhile, the firewall guys are working on a problem that we just don't know about. So, we have wasted a lot of time and energy trying to figure out things that aren't our problem. The bad scenario hasn't happened to us in production, but in our test environment, it's happens once where a couple of app dev teams have been able to stand down because we were correlating events correctly.

We struggle with mean time to resolution on things. We do have a lot of change control rigor, but the solution hasn't changed our organization yet. The idea is when we're getting the events for our service provider of operating systems, servers, and storage network correlated intelligently together with our application changes, application performance monitors, and application availability monitoring tools, then we'll make more intelligent decisions about root cause, where problems lie, and be able to react more intelligently. This will reduce mean time to resolution, but we're not there yet. 

The division who has been using Solace for years has a mature costing estimator model for internal projects. That model certainly will be leverageable for the technology operations guys. We haven't crossed that bridge yet because we're still in PoC mode. It's very likely that once we hit prod, we'll have ease of solution design when we have a protocol-agnostic message transport in place, and that our solutions will be easier to craft and give cost estimates.

It is easy for architects and developers to extend their design and development investment to new applications using this solution. In our architecture practices, we are always documenting compositions. We care a lot about the data exchanges between applications or the integrations. We have a lot of contractors and other integrations that we care about. Having transmission facilitators definitely makes the architect's life a lot easier when we just put a message on the queue and it's going to get transported by the facilitators to wherever it needs to go. It is definitely easier when we have Solace and an event mesh up and running. Today, when we have integrations that don't leverage those transmission facilitators, like an MQ Hub or Solace event mesh, those integrations are much harder to get approved because we have to dive into the security, access controls, encryption, and all that other stuff. 

One of our use cases at our global company went live recently. We have a lot of goods that move via sea routes. While there are other modes of transport, particularly for the sea route, we wanted to track our shipments, their location, and that type of information and generate some reports. Also, there are multiple applications which need this data.

With Solace, we are bringing information in every minute (almost real-time) from our logistic partners and putting it on Solace. Then, from Solace, the applications that want to consume the information can take it. E.g., we are generating some dashboards in Power BI using this information. We are also pushing this information into our data lakes where more reporting plus slicing and dicing is available. In future, if more subscribers want this information, they will also be able to take it. 

We have both our private cloud and a version completely hosted on SaaS by Solace. 

The base use case is that we wanted the shipment tracking information in multiple applications, like Power BI and data lake, for the more reporting, etc. If we would not have got Solace, then we need to extract this information multiple times from source application. Now, we pull this information only once and then put it on Solace. Anybody can take it from Solace because the information is readily available. We are generating the data only once. 

Every organisation has exposure of their data with their devices being interconnected. We don't want to transfer the same data multiple times. Solutions like Solace can help us in publishing data only once, then anybody can pick it from there. This reduces costs of data transfer. It reduces the load on data sources, because we aren't asking them to generate the same data multiple times.

Our company is an SAP centred company. A lot of our key applications are using the SAP product suite. When we talk about transaction data and master data, that is where the real complexity comes into play. There are a couple of use cases that we have discussed with Solace for topic hierarchy. E.g., a product master might be sold by multiple channels, produced in multiple factories, and sold in multiple geographies, so creating a topic hierarchy for these could be challenging. When we started, we discussed this complexity with Solace. They helped us arrive at an initial topic hierarchy based on some similar use cases which have been implemented for other customers, sharing their insights.

Another point is their overall approach to topic building. They have very good documentation. It will be our own internal complexity that will drive the topic hierarchy. We are currently in the early stages, and so far journey has been good. Right now, we are comfortable with information and help we are getting from Solace along with the overall approach recommended for topic building. However, time will tell, especially after we generate very complex cases with Solace, how this topic hierarchy functions.