Qualibrate Reviews

We are trying to convert a large majority of our regression testing from being manual to using this automated solution. We want to basically reduce the number of manhours that we have to spend three or four times a year to execute our regression test cycles. We want Qualibrate to offload those hours.

It is a web-based application, but my understanding is that it is hosted in the cloud on Qualibrate's server. So, I would say it is in the cloud.

It will alleviate some of the stress in terms of enabling us to do more frequent testing, and it will help us with our timelines. I certainly think that the potential is there, but we're not there yet because we're still kind of in the project rollout phase. So, we're not quite reaping the rewards or the dividends of deploying Qualibrate because of this upfront effort we're having to put into it, but myself, my manager, and my colleagues see the value. We do see the long-term value in terms of time savings and the stress it is going to alleviate for the team. That's the whole reason we wanted Qualibrate. It is just unfortunate that we're not far enough to have reaped the rewards just yet. There might also be some additional savings or benefits, but we haven't gone off to explore those in any sort of capacity since it is not really our focus.

It is the principle functionality that we're leveraging, which really can be defined as recordings and playbacks. So, you record the scripts that you want to execute and you also want to be able to playback. So, these are the features that we are largely leveraging. There are flows and scenarios, and they are the design aspects that fit within the playback and the recording solution. For me, they are the core of Qualibrate, and that's what we're using. 

There is a module that we would like to have. We would like Qualibrate to design a requirements module so that we can design our testing, our flows, and our scenarios based on our actual requirements. Right now, we're doing that, but we're having to do it outside of Qualibrate. For example, in Excel, we might have a list of 50, 60, or 70 different requirements and combinations of tests that need to be executed, and since that module doesn't exist in Qualibrate, we're doing it offline. We have already vocalized that wishlist to them, and they have acknowledged it, but I have no idea when they're going to get around to deploying something like that. It is probably number one on our list. 

We're using the testing schedule and execution a little bit. We're not really thrilled with the testing schedules just yet. They just recently enhanced the schedules module. So, I haven't got to play with those new enhancements just yet, but we have noted to the Qualibrate team that we were hoping for some expanded functionality in that space. I think they're working on that, and some of it might've been delivered recently, but there are still some things in their backlog that we're hoping they will deliver.

Their support has been quite good overall, but there have been some frustrations recently where we just feel like we're not always being heard. They are very inconsistent. In some cases, their support is very immediate, and then in other cases, there is absolute silence.

We just started this year. We got training in early February, so we have been using it basically for seven or eight months.

Its stability is pretty good. We had some issues early on, probably back in February or March. We raised some concerns with the Qualibrate team, and they deployed some development changes that created some improved stability. I still run into a stability type of issue once in a while, but it is 80% to 90% less than what I was running into six or seven months ago. So, it seems like it's a much more stable solution. There is still room for improvement, but I'm quite happy with it at this time.

It is a very scalable solution. We started out with one component, and we are already working on our second and our most important component. I don't see why Qualibrate couldn't be used basically for 90% to 95% of our testing. I do think there are going to be some test cases for which it just makes sense to leave them as manual because they may be highly complex, or there may be some nuances about it. So, it might make more sense that a person does those test cases on his or her keyboard and monitor, but we're going to scale it and really maximize it from a regression testing perspective. Our team is really looking forward to having all this completely done and be able to have the time savings. So, its scalability is great.

In terms of usage, I don't think we're going to use it on a day-to-day basis. This is going to be largely used for automated testing that's going to occur on a virtual machine. We want to basically schedule it and have these tests run hands-off. I and my two colleagues who are hands-on in these projects are basically administrators of the Qualibrate solution within the team. So, we're making the decisions and governing who's using it, etc. We do most of our regression testing on a quarterly basis or near quarterly basis, so there are three to four cycles a year. That will be the primary use case. We will probably run some of the core tests on a more frequent basis. I can see us doing it maybe weekly. I don't know if we'll ever do any sort of daily testing with it, and I'm not sure that we would ever do any sort of manual testing. So, I don't see any of my colleagues sitting at their desks and going, "Oh, you know what? I think I'll go run some tests in Qualibrate today." We're going to schedule these, we're going to offload these, and we're just going to check on what the outcomes are. If something goes wrong, a test fails or something else is amiss, then we'll address that. Otherwise, it will be kind of out of sight, out of mind as per our long-term plan.

I am going to give their support a seven out of 10 because I feel that the quality of the support is quite high, but we are a little bit frustrated, especially in recent weeks, with the timeliness of the support.

When we were a customer who was just starting out, they were always available for an email or for a quick call. They would be willing to jump on a call on short notice. They're probably assuming that we're much more self-sufficient now, which is largely true, but we've had three or four topics come up probably since the 1st of July. I know that my colleague has submitted three or four support cases, and he hasn't even got an acknowledgment that they were received by them.

The first thing you want to do with a customer is to acknowledge that we've got your ticket, and we're looking into it. We'll get back to you, even if it takes a while to get back. They're not even acknowledging his cases. I submitted one this past Friday, and I got a response the following Monday. So, it seems they are very inconsistent. In some cases, their support is very immediate, and then in other cases, there is absolute silence. 

This is the first functional testing tool that we've implemented. I have been harping on this to my management staff for years and years. We never even considered and evaluated any tools, but finally, management saw a need and said, "Yeah, okay, Steve, you're right." It wasn't really so much that use case that clinched the purchase or procurement of Qualibrate; it was the fact that we actually have a second use case on which we haven't really spent a lot of time, and one of our higher up managers wanted to use Qualibrate to generate data in our SAP systems.

Our solution sits on top of SAP or works within the SAP environments. So, we do a lot of demos and proof of concepts. The systems that SAP provides are actually quite old, and there is no more-recent data. We're talking data from 2013, '14, '15. So, this manager wanted to have one of my colleagues write up some scripts that would actually run in SAP and generate, for example, sales order data, purchase order data, and other business-like data that would look like more real meaningful data from a demo perspective and from the customer's perspective. Because that takes on a sales or pre-sales approach, he could go in and justify the expense. So, even though I've been screaming for it for 10 years and has basically been ignored, it was another use case that actually clinched it. My colleague is working on it, and he is having some success with that use case, but from a testing perspective, this is the first time we did any sort of evaluation or actual purchasing of an automated testing or an automated scripting solution, whatever you want to call Qualibrate.

I found it a bit more on the complex side. There is some flexibility within the solution, and that's fine. Flexibility can be useful. I like flexibility, but if you're brand new to the solution and you don't really know what you're doing, then it's hard to understand how to design what you want to do within that flexible solution. So, we had to scramble a little bit. We're now on our third phase of this project, and each time, I and the two colleagues who are heavily invested in this have learned what not to do versus what to do. We're improving each time, but we're having to figure that out kind of on our own with some guidance from the Qualibrate team. Most people learn by doing and by making mistakes, and that's where I feel like we're at. So, initially, we struggled with the design and how we wanted to design our testing within the Qualibrate solution. We struggled a little bit like that. We did reach out to the support team at Qualibrate. We got some insight from them, but I also feel like we were largely left on our own to kind of figure it out. That's not necessarily a problem, but some customers may not like that approach. They're going to want more direct answers.

We're still deploying it. The initial phase was three months. We then took a break, and then we went on for two months. We just started our third phase about two weeks ago. So, when we're reaching a milestone, we are putting that phase off, looking back at lessons learned, and then taking a small and kind of a mental break from it. We then come back to it a short time later. The phases are getting smaller, and that's also representative of the fact that we are getting better and more efficient. We have a better understanding of the solution and how our requirements stick within the Qualibrate solution. I actually expect this third phase to be as little as maybe six weeks. So, it started out initially as three months of learning the hard way and making mistakes. We've now whittled down to perhaps a month and a half for each of our phases. We still probably have another one or two phases to go. This will be something that would roll around probably through the new year. I hope to be done by January, and then we can effectively start using Qualibrate on a near-daily or perhaps a weekly basis. 

In terms of the implementation strategy, our software is fairly mature. It has been around for 20 years. We kind of felt like using Qualibrate to effectively transition all of our manual testing to automated testing. It was like an elephant. How do you eat an elephant? It is one bite at a time. So, we tried to look at our software from a component level, and we said, "We've got this really big component, and we have a couple of other smaller components." We then basically tried to choose one that didn't have a massive footprint. It was important, but it was not so large that we felt like we were going to try and drown getting it integrated into Qualibrate. So, that was phase one that we focused on. Once we got that done, we moved to phase two.

We had to break down phase two and phase three into two sets. So, our last phase and the current phase we're on will cover our most critical component within the solution. So, this will be a big milestone. When we're done with phase three, we'll be able to look back and say 90% of this component can now be tested through Qualibrate. Phase four, and probably phase five, is going to touch smaller and less important modules that ideally we'll be able to roll out and get done in a pretty compressed timeframe. So, we might be done sometime in December, but because of the upcoming holidays at that time of the year, it probably is going to roll over into January.

For the project that I'm closer to, we're not realizing the dividends, but we know that it's on the horizon. We're just not quite there yet, but we're confident that we're going to have it. So, we are looking forward to that day.

For the parallel use case, we are seeing some early returns. The reason is that it is a bit of a project. My colleague was able to stand it up in a fairly limited number of hours. It pretty quickly and easily started creating or generating data in the SAP demo systems that were already being used by our pre-sales team. So, we are seeing some benefit from that. 

We probably have 10 licenses, but I don't know what are the costs or anything like that.

We did not evaluate other options. The reason was that we had contacts on each company who already had some sort of relationship. I think it was a rather informal relationship, but there had been some dialogue and some interaction in terms of people shaking hands and knowing each other by name, probably in trade shows in Europe. So, they knew each other. Because we already had kind of that rapport, we just looked at Qualibrate.

They did a short PoC with us. We met with them over a period of three or four days, and it looked like a really good solution. It was young but mature enough to do what we needed to do, and we were sold. Perhaps, we should have done some due diligence. We would have done that if we hadn't had that relationship already there, but I think that's what kind of clinched it for us.

I would advise making sure that you understand the tool. You don't have to understand every aspect of it, but you need a pretty thorough understanding of its components, the playbacks, the designs, and the technical pieces that make up the solution. Our pain point was the very first phase. We were brand new to the solution. We went through some knowledge-sharing sessions with their support staff, but we really felt like we were struggling at first. Most of that was probably because we simply did not understand Qualibrate to the depth that we did in subsequent phases. So, it would be good if you could really spend a little bit of time upfront, maybe just from a sandbox perspective, and you test it, mess around with it, and get familiar with it. After that, you can start to decide and design how your solution and how your scripting should look within Qualibrate. This way, you'll avoid some of the pitfalls that we ran into. We found ourselves having to do rework over and over again, and that was simply because we learned a better way of doing it than we did initially. 

You can maybe test it on a smaller scale. We were so motivated to get started that we tried to run before we could crawl. We should have kind of spent a little bit more time doing that initial build-out testing. We should have made sure we were happy with the results and that we were taking the right approach. We did all this work and then went, "Oh, that's not really going to work." We then had to go back and do some redesigning. So, some upfront commitment would help there.

It is fairly advanced. It has only been around for three or four years, but it does seem like a pretty mature solution. Most of it is pretty user-friendly. There are some components that I have found to be a little bit more technical and a little bit more daunting to use, but overall, it is quite easy to use as far as this kind of scripting software goes. I'm quite happy with it. I'm not overly technical, and I can understand it fairly easily. That must make it pretty good.

We're not using it on a daily basis. We're still kind of in a project phase where we're still ramping up and getting all of our scripting and recording done. So, right now, everything is more in kind of a batched process or phased process. We do hope to get to the point of being able to use it for daily or near-daily testing, but we're still a little away from that. Since we've only had it in-house for seven or eight months, we're just not using it on that frequency just yet, but that is our long-term plan. There are a couple of components that we are not really using at this time. That's because they are still immature, and they have not been around very long as part of the Qualibrate solution. We'll get to those perhaps another day. We're focused on our primary use case right now.

We are quite happy with the flows and scenarios and those building blocks that really represent the core of the solution for scripting and playbacks. So, we're overall quite happy with this solution at this point.

Based on how it operates today, I would give it a nine out of 10.

We are using Qualibrate for our testing strategy, as well as for standard change requests.

We began working with Qualibrate for testing our on-premises SAP solution. After that, we started implementing the cloud application, SAP SuccessFactors, so we started using it to test that.

Regression testing is very important for this application, and we use Qualibrate for this, as well as to perform user acceptance testing.

Another activity that we use Qualibrate for is the automation of standard changes. These types of changes are pre-approved, and via Qualibrate, the configuration is done. The flow receives requests for changing an item in our ticketing system, ServiceNow, via our middleware system, which is connected to Qualibrate. In Qualibrate, existing recorded scripts will be started and then Transport in the SAP development system will be created automatically.

In the SAP landscape itself, the Transport will be moved automatically with all checks through the landscape. Of course, at the end when the testing is completed in the production environment, there will be a message sent back to the ticketing system via the middleware and the ticket will be closed.

I recently created a business case for Qualibrate and it has to do with regression testing. We have approximately 300 to 400 SAP end-users on-premises in the organization, across different business units. If you have to organize a regression test, it is because you have changed something and you want to know if it will have an impact on the processes in the other regions. In cases like this, you want to check to ensure that nothing was touched.

Normally you invite maybe 50 key users to test in the different functional domains. This will perhaps cost an average of four hours per person. Instead, you are running automated test scripts that have been pre-recorded. Even if the length of time per test remains at four hours, you are saving that much in terms of what it would cost to pay testers for the same work.

It is not possible to test everything, especially if you have interfaces with other external systems, but you do win a lot of the time. Also, it's not only for testing when your project is going live, or when you're bringing changes to an environment but also checking your daily operational support. It brings a really big benefit in terms of keeping your system stabilized. In the end, you have fewer disruptions for the business.

The low-code approach enables users to onboard quickly and then use it in their daily test creation, but I would always suggest centralizing it. The users have the scripts and you only need one or two people on a central level to keep control of it. 

People can make use of Qualibrate for training because that's also a benefit. If you have a new employee, you can show them in the system how one of the end-to-end processes looks. You can also explain the UFT in the same way.

The most valuable feature is that it's user-friendly. It is really easy to use and if you have your adapter downloaded, you can remotely explain to your key users, the testers, how to use it.

We have all of the biggest parts centralized, which means that we can run all of the scripts at the moment we bring changes into the UFT environment. They can easily be run in the background. 

You have the ability to reduce all of the recorded transactions so that you can build your own consolidated end-to-end process. If something in the end-to-end process changes then it is very simple to insert a new part into it.

We have just finished all of the scripts for SuccessFactors, which is an HR application. Every quarter, you have a new release, and we are dealing with scrambled data that needs to be reduced. Using Qualibrate makes it easy.

What I would really like to see is if you are running scripts in Qualibrate, and there is a defect, then you can have it automatically raise a defect in your own ticketing system. We do not use the defect system in Qualibrate, which is why this would be helpful. We won't want to have to use the ticketing system in Qualibrate in addition to the one we use now.

The reporting can be improved. For example, the reports can be better and show details like how many of your scripts have been executed, what the history is, which scripts have not been executed for more than a month, and other types of statistics.

It would be nice if when you add or change some specific script in your SAP system, it would automatically check to see what it implied with respect to Qualibrate. For example, it would link to the appropriate Qualibrate script if it needed to be changed. In the context of an end-to-end process, which has already been recorded, if next week I change something in the process then I would like to get an alert that we need to make the relevant change in the process.

My company has been working with Qualibrate for two years.

I worked with Qualibrate for another company approximately six years ago.

Stability-wise, so far so good. Last week, there were two times that the system was unavailable but that really does not happen very often. In case it does happen, the communication from their support desk is very fast. They inform you directly.

Scalability-wise, this solution is fine.

We don't have very many people who are working on it. There are two external, my test manager, and myself, so four in total. 

We have no issues with the technical support. When I have a problem and the system is not available, I want to know how soon I can get it back up and running. I do not run these kinds of systems every hour; even when your SAP is running 24/7, your testing does not. However, when you want to test something, it needs to be available, and support takes good care of this. When I have called they pick up the problem and they resolve it.

Approximately six years ago, I was with another company and I used several different test systems. I was involved in a pilot at the time and helped to define the requirements. One of the important ones was to keep it fair user-friendly, which Qualibrate is.

Over the period of a few years, I looked at test automation. At one point, in 2018, I compared the features in Qualibrate with what it had four years prior, and there had been new guidelines for things like data introduced during that time. A lot of improvements had been made to the product.

I have experience with HP QTP, but it is an expensive tool and I explained that I didn't need a Rolls Royce for testing. Rather, I just wanted a simple tool. I didn't want to have a very expensive tool where I only use 50% of the functionality when instead I can have one that's user-friendly.

Comparing Qualibrate with my other tool in terms of manual testing and the process of documentation, I would rate Qualibrate a nine and the other tool a five. The other tool was very technical and not user-friendly. Every time you need to change something, you have to contact technical support. It is also very complex when you want to explain it to your key users.

I have my own test manager who is responsible for the SAP on-premises solution, which includes testing it with Qualibrate. We had an external team help us with supporting the functionality of our cloud system, and we also asked them to take care of Qualibrate. Unfortunately, it took a really long time and the person responsible for it has now been replaced. Together, the new person, working with my test manager, has it under control.

If you have it under control, you just have to do the maintenance.

We have seen a return on our investment. There are a lot fewer people involved in our testing and the system is more stable. In addition, more people compliment you and less complain if the system is not available, which is a really big benefit.

We signed a three-year contract and the pricing is in line with our expectations.

If you look at the cost of your license and what is required, you will see that it is better to keep everything centralized and pay for perhaps five user licenses. This is more cost-effective than giving one to everybody in the business.

When I proposed to use Qualibrate, they were hesitant because it was not in the top four or five of Gartner. That necessitated some more evaluation which included a tool by IBM and at least one other one.

My advice is that if you know how an application is working then reduce it on a weekly or perhaps daily basis. If you are doing this then you will very quickly understand how the system is working.

The first time you record a regression test, I would advise doing it in a classroom session. In this scenario, people are doing the regression test manually and you ask them to execute the same script, but this time, start the Qualibrate recording function. People can make use of the backspace functionality and with it being recorded, it can be used the next time a regression test is needed.

I suggest that you not only look at the price but also the quality. This vendor listens to you and if you have requirements or need help, they are always available and not only by phone. If required, they will visit the customer from time to time. The application is user-friendly and there is documentation available. Overall, we are really happy with it.

I would rate this solution a nine out of ten.

We're using it for testing, mostly regression testing.

Before Qualibrate, we did a release in about three months, and the regression testing would take us about three weeks with 15 people. Now, we do it in one night. The time savings are really important because we can now release things more quickly.

When it comes to problems that require changes, we can now do them in one week. Before, we would save all such changes and do them together over three months. The business is very pleased with the way we work now.

Qualibrate's low-code approach also enables users to get on board quickly and use it. We work with SAP and we know everything about SAP, but when we use Qualibrate, we can easily explain to our new people also how it works. Even business users can create automated test scripts without coding knowledge. Generally, getting them to do so would require about a month of training but now it is done in a day.

We're also able to do more frequent testing, without putting too much pressure on timelines and resources. In previous times, we didn't test security notes because there were too many. Now, every time we have a security note in the system, we do a test, and that's great. That has helped make our risk very low.

The most valuable feature is the way it works, the usability. From the first setup, the user interface is the most beautiful. It's very nice the way you can recover things and make it work.

We use the solution’s Test Planning & test Execution Scheduling features, and they are very important. They are easy to work with. We use SAP Solution Manager, and Qualibrate works with it, enabling us to manage all our tests, taking them from Solution Manager directly into Qualibrate. Because everything we do is in SAP Solution Manager, we are now able to do test automatization, combined with Qualibrate. All the defects are available in SAP Solution Manager and all the changes will be in Qualibrate, immediately.

From Qualibrate, we see whether tests are good or not. We can create a test-related incident in Solution Manager for those that are not good, and Test Suite in Solution Manager will notify the person who has to rebuild it. That works very nicely.

Not everything in SAP works well with Qualibrate. There is a development tool called xpath and you have to program it. We always thought it wouldn't be necessary to program it with Qualibrate, that everything could be solved by Qualibrate without programming, but you have to program some things. Using xpath is more complicated, and not easy for everybody. It would be helpful if there were a no-code solution for this.

We have been using Qualibrate for more than three years now. Our first version was version 2.0, and now we are moving to version 3.0.

It's very stable. We never have problems with it.

We haven't had any problems with scalability. Currently, we have more than 100 scripts and some are big scripts with 400 to 500 steps.

We have plans to increase our usage of Qualibrate because we are going to S/4HANA, and that's a big step for us. 

Qualibrate's technical support is great. They always work together with us. There are people from India who help us and there is no problem with support when there is a problem we need to resolve.

Positive

We did not have a previous solution. We did everything manually.

The initial setup of Qualibrate was not complex at all.

The group we bought it from, Fiori, was very willing to work with us. It was very good because there was very little waiting. I would call the manager and he would call me back and we would immediately have someone on the line who knows the business. That was really great. And they come onsite, making it easy, and explain the solution to you.

Our deployment took about two weeks.

We had 32 big scripts already and we wanted to put them in Qualibrate. The consultant gave us lessons and, in about two weeks, we were ready. It was easy. We had 15 people on the floor, and two of them were from the consultant, and they helped us with the scripts so that everybody understood everything. That was great.

We have seen ROI in our ability to release things a lot faster.

I compared the prices of the 15 solutions we looked at. Qualibrate was the most valuable because it could be integrated with SAP Solution Manager. Tosca had a big price, but it does everything, including defects. But because we already had defects taken care of with SAP Solution Manager, we just needed one module to work with and that was Qualibrate. It was easy to pick it and not so expensive.

We looked at Tosca and about 15 other solutions. Because we have a lot of things in SAP Solution Manager we tried an SAP solution, but that didn't work for us, so we looked further. Qualibrate was the best for Solution Manager. The big pro was the user interface. That was really the most important thing I saw.

If we didn't have SAP Solution Manager, I probably would have bought Tosca, but our other tools already covered a lot of its functionality.

When you are considering buying something, you have to see the user interface. If you buy something that is complex, people won't work with it. Qualibrate is so easy to work with.

I would probably rate the solution higher than eight out of 10 if we weren't using the on-premises version. With the cloud version, Qualibrate can look at things with you, but with the on-premises version they can't. That makes some things a little bit difficult, but because we work with defense material, it's not possible. The on-premises version is more difficult than the cloud version.