What is our primary use case?
It supports business applications for performance testing across various domains, such as retail, financial services, and banking. They had already procured a tool, LoadRunner, for use in their environment, whether on-prem or in the cloud. I've worked on numerous use cases that vary depending on the sector, like banking or insurance, as each sector has its requirements.
What is most valuable?
This tool has specific components, like LoadRunner Virtual User Generator for scripting, and execution windows, such as LoadRunner Cloud or Performance Center. It’s divided into three parts: scripting and code development are handled by one component, execution by another, and analysis by a separate file. This makes it user-friendly and easy to understand. In comparison, JMeter, which is open-source, can feel a bit clumsy. NeoLoad, on the other hand, is a good alternative and performs well, as I’ve worked with both. However, due to customer requirements, I procured LoadRunner, and we continue to use it. So, there was no option but to go with LoadRunner. In terms of setup and usability, it’s straightforward. If I were to rate the leading tools in the market, LoadRunner comes first, followed by NeoLoad. Based on my experience and colleague feedback, these are the most commonly used project tools.
What needs improvement?
The main difference is the interface; the look and feel have changed, but the background setup and configuration remain the same. The project admin team had already set up LoadRunner Cloud in our environment.
Since AI plays a major role in today's world, many tools are expected to integrate with it. If LoadRunner has AI integration, that would be a great feature.
In past projects, including those with LoadRunner and NeoLoad, clients often asked about integrating CI/CD pipelines, such as using Jenkins to automate the triggering process. I’ve done POCs on this, and it’s possible. Once set up, the pipeline can automatically execute tests without manual intervention.
For how long have I used the solution?
I have been using OpenText LoadRunner Cloud for ten years.
How are customer service and support?
From your experience, there are typically two options when purchasing a license: with or without support. Opting for support has been beneficial, especially when issues arise.
For instance, you mentioned a colleague facing a problem where they couldn't capture an application's network due to some background package being installed. After raising a request, the vendor support team from OpenText connected within 24 hours, set up a meeting, and provided helpful suggestions. Although it took a few days to understand and resolve the issue fully, the support was proactive and responsive.
Both LoadRunner and NeoLoad seem to offer efficient and timely support, ensuring users can get assistance when needed, which adds value to the tools themselves.
How would you rate customer service and support?
How was the initial setup?
It's quite simple and user-friendly. Anyone with some LoadRunner experience can easily adapt.
The cloud configuration setup and server access can be tricky, as we don't usually get direct access. Sometimes, even after the configuration is supposedly done, problems arise when we start testing. For example, after developing a script locally, I occasionally face issues executing it on the Cloud LoadRunner machine. The test doesn't start, and I can't review the results. This usually happens if the configuration isn't done properly. I have to check the cloud-based machines and verify the IP range of the virtual systems because Cloud LoadRunner doesn't use static IPs—it has a dynamic range. The IP range can vary, and it picks one by one where the problem arises.
When that happens, I need to check whether my application can access the cloud machines. We need to request firewall access, specifying the source and destination, to enable communication between the cloud machine and our application. On-prem setups are different; there’s a physical machine, and I have direct access. I can check the hostname of my application and verify access to the load generator. I can ping the machine, use Telnet, and quickly resolve the issue. The process feels more cumbersome in the cloud, especially when facing execution challenges.
I transitioned to the cloud just six months ago. I've worked on cloud-based environments for only six months after migrating from on-prem applications. This involved migrating existing scripts to the cloud and re-executing previous tests to compare performance between on-prem and cloud. The goal was to assess the differences between the two environments, as the client expects improved response times and overall performance due to the cloud's higher configurations. I’ve shared the results of these tests, which reflect those improvements
What's my experience with pricing, setup cost, and licensing?
The pricing is high compared to other licensing tools like NeoLoad. It's not excessively expensive but higher than NeoLoad. However, in my experience, clients often weigh NeoLoad and LoadRunner equally. LoadRunner has been around for a long time, and while some companies have moved away from it, the higher pricing can be attributed to its support for multiple protocols. For instance, it supports over 40 protocols, including SAP, Citrix, Oracle databases, and batch testing. This wide range of protocol support is one reason for the higher cost. Depending on the applications being tested, whether desktop, Citrix or something else, you may need to purchase specific protocol-based licenses.
LoadRunner offers a free license for up to fifty users, which may not be adequate for most projects. Projects require more load to simulate real business scenarios, so a more comprehensive pricing model may be necessary.
Two main licensing budgets are to consider: one for LoadRunner and another for NeoLoad. If a project has no budget for purchasing a product and is looking for open-source tools, I highly recommend starting with Apache JMeter. However, it's important to note that JMeter may not be flexible enough for all requirements, especially for desktop applications, as it has limitations.
JMeter operates in a single interface and lacks commercial tools' structured reporting and usability features, making it less user-friendly. While it’s a good starting point due to being free, users may need extra effort to organize and interpret results.
We recently evaluated OpenText's LoadRunner and NeoLoad for a project. We engaged with both vendors to take demos and understand their licensing models. Generally, NeoLoad's pricing was comparable to LoadRunne
What other advice do I have?
If it's in the cloud, there's a separate version called LoadRunner Cloud. We have both options on-prem and cloud. Some applications only support on-prem, while others are cloud-based, so we must test them in the cloud environment. In this account, we work with both environments. It's worth noting that around 90% of applications are moving to the cloud these days. In the cloud version of LoadRunner, the concept of manually adding load generators doesn't exist. You don't need to worry about adding load generators to the tool, a common challenge with on-prem setups. In on-prem projects, if there are ten load generators and others are using them, you must wait for availability before running your tests. LoadRunner Cloud automatically assigns the load generators.
If someone in my network is looking for a performance testing tool, I recommend LoadRunner, especially if the client has a sufficient budget. For clients using cloud-based applications, it's essential to consider LoadRunner's cloud environment, which requires specific infrastructure from OpenText. They can opt for the Performance Center without worrying about cloud infrastructure if their applications are on-premises.
The recommendation ultimately depends on whether their applications are cloud-based or on-premises. Additionally, pricing and supporting the required protocol play significant roles in decision-making. It’s important to assess which types of applications they are working with, such as web, Citrix, or Windows applications, to determine the necessary protocol licenses. Buying unnecessary protocols can lead to wasted expenses.
Overall, I rate the solution a nine-point five out of ten.
