What is our primary use case?
We are trying to convert a large majority of our regression testing from being manual to using this automated solution. We want to basically reduce the number of manhours that we have to spend three or four times a year to execute our regression test cycles. We want Qualibrate to offload those hours.
It is a web-based application, but my understanding is that it is hosted in the cloud on Qualibrate's server. So, I would say it is in the cloud.
How has it helped my organization?
It will alleviate some of the stress in terms of enabling us to do more frequent testing, and it will help us with our timelines. I certainly think that the potential is there, but we're not there yet because we're still kind of in the project rollout phase. So, we're not quite reaping the rewards or the dividends of deploying Qualibrate because of this upfront effort we're having to put into it, but myself, my manager, and my colleagues see the value. We do see the long-term value in terms of time savings and the stress it is going to alleviate for the team. That's the whole reason we wanted Qualibrate. It is just unfortunate that we're not far enough to have reaped the rewards just yet. There might also be some additional savings or benefits, but we haven't gone off to explore those in any sort of capacity since it is not really our focus.
What is most valuable?
It is the principle functionality that we're leveraging, which really can be defined as recordings and playbacks. So, you record the scripts that you want to execute and you also want to be able to playback. So, these are the features that we are largely leveraging. There are flows and scenarios, and they are the design aspects that fit within the playback and the recording solution. For me, they are the core of Qualibrate, and that's what we're using.
What needs improvement?
There is a module that we would like to have. We would like Qualibrate to design a requirements module so that we can design our testing, our flows, and our scenarios based on our actual requirements. Right now, we're doing that, but we're having to do it outside of Qualibrate. For example, in Excel, we might have a list of 50, 60, or 70 different requirements and combinations of tests that need to be executed, and since that module doesn't exist in Qualibrate, we're doing it offline. We have already vocalized that wishlist to them, and they have acknowledged it, but I have no idea when they're going to get around to deploying something like that. It is probably number one on our list.
We're using the testing schedule and execution a little bit. We're not really thrilled with the testing schedules just yet. They just recently enhanced the schedules module. So, I haven't got to play with those new enhancements just yet, but we have noted to the Qualibrate team that we were hoping for some expanded functionality in that space. I think they're working on that, and some of it might've been delivered recently, but there are still some things in their backlog that we're hoping they will deliver.
Their support has been quite good overall, but there have been some frustrations recently where we just feel like we're not always being heard. They are very inconsistent. In some cases, their support is very immediate, and then in other cases, there is absolute silence.
Buyer's Guide
Qualibrate
March 2023
Learn what your peers think about Qualibrate. Get advice and tips from experienced pros sharing their opinions. Updated: March 2023.
687,256 professionals have used our research since 2012.
For how long have I used the solution?
We just started this year. We got training in early February, so we have been using it basically for seven or eight months.
What do I think about the stability of the solution?
Its stability is pretty good. We had some issues early on, probably back in February or March. We raised some concerns with the Qualibrate team, and they deployed some development changes that created some improved stability. I still run into a stability type of issue once in a while, but it is 80% to 90% less than what I was running into six or seven months ago. So, it seems like it's a much more stable solution. There is still room for improvement, but I'm quite happy with it at this time.
What do I think about the scalability of the solution?
It is a very scalable solution. We started out with one component, and we are already working on our second and our most important component. I don't see why Qualibrate couldn't be used basically for 90% to 95% of our testing. I do think there are going to be some test cases for which it just makes sense to leave them as manual because they may be highly complex, or there may be some nuances about it. So, it might make more sense that a person does those test cases on his or her keyboard and monitor, but we're going to scale it and really maximize it from a regression testing perspective. Our team is really looking forward to having all this completely done and be able to have the time savings. So, its scalability is great.
In terms of usage, I don't think we're going to use it on a day-to-day basis. This is going to be largely used for automated testing that's going to occur on a virtual machine. We want to basically schedule it and have these tests run hands-off. I and my two colleagues who are hands-on in these projects are basically administrators of the Qualibrate solution within the team. So, we're making the decisions and governing who's using it, etc. We do most of our regression testing on a quarterly basis or near quarterly basis, so there are three to four cycles a year. That will be the primary use case. We will probably run some of the core tests on a more frequent basis. I can see us doing it maybe weekly. I don't know if we'll ever do any sort of daily testing with it, and I'm not sure that we would ever do any sort of manual testing. So, I don't see any of my colleagues sitting at their desks and going, "Oh, you know what? I think I'll go run some tests in Qualibrate today." We're going to schedule these, we're going to offload these, and we're just going to check on what the outcomes are. If something goes wrong, a test fails or something else is amiss, then we'll address that. Otherwise, it will be kind of out of sight, out of mind as per our long-term plan.
How are customer service and support?
I am going to give their support a seven out of 10 because I feel that the quality of the support is quite high, but we are a little bit frustrated, especially in recent weeks, with the timeliness of the support.
When we were a customer who was just starting out, they were always available for an email or for a quick call. They would be willing to jump on a call on short notice. They're probably assuming that we're much more self-sufficient now, which is largely true, but we've had three or four topics come up probably since the 1st of July. I know that my colleague has submitted three or four support cases, and he hasn't even got an acknowledgment that they were received by them.
The first thing you want to do with a customer is to acknowledge that we've got your ticket, and we're looking into it. We'll get back to you, even if it takes a while to get back. They're not even acknowledging his cases. I submitted one this past Friday, and I got a response the following Monday. So, it seems they are very inconsistent. In some cases, their support is very immediate, and then in other cases, there is absolute silence.
Which solution did I use previously and why did I switch?
This is the first functional testing tool that we've implemented. I have been harping on this to my management staff for years and years. We never even considered and evaluated any tools, but finally, management saw a need and said, "Yeah, okay, Steve, you're right." It wasn't really so much that use case that clinched the purchase or procurement of Qualibrate; it was the fact that we actually have a second use case on which we haven't really spent a lot of time, and one of our higher up managers wanted to use Qualibrate to generate data in our SAP systems.
Our solution sits on top of SAP or works within the SAP environments. So, we do a lot of demos and proof of concepts. The systems that SAP provides are actually quite old, and there is no more-recent data. We're talking data from 2013, '14, '15. So, this manager wanted to have one of my colleagues write up some scripts that would actually run in SAP and generate, for example, sales order data, purchase order data, and other business-like data that would look like more real meaningful data from a demo perspective and from the customer's perspective. Because that takes on a sales or pre-sales approach, he could go in and justify the expense. So, even though I've been screaming for it for 10 years and has basically been ignored, it was another use case that actually clinched it. My colleague is working on it, and he is having some success with that use case, but from a testing perspective, this is the first time we did any sort of evaluation or actual purchasing of an automated testing or an automated scripting solution, whatever you want to call Qualibrate.
How was the initial setup?
I found it a bit more on the complex side. There is some flexibility within the solution, and that's fine. Flexibility can be useful. I like flexibility, but if you're brand new to the solution and you don't really know what you're doing, then it's hard to understand how to design what you want to do within that flexible solution. So, we had to scramble a little bit. We're now on our third phase of this project, and each time, I and the two colleagues who are heavily invested in this have learned what not to do versus what to do. We're improving each time, but we're having to figure that out kind of on our own with some guidance from the Qualibrate team. Most people learn by doing and by making mistakes, and that's where I feel like we're at. So, initially, we struggled with the design and how we wanted to design our testing within the Qualibrate solution. We struggled a little bit like that. We did reach out to the support team at Qualibrate. We got some insight from them, but I also feel like we were largely left on our own to kind of figure it out. That's not necessarily a problem, but some customers may not like that approach. They're going to want more direct answers.
We're still deploying it. The initial phase was three months. We then took a break, and then we went on for two months. We just started our third phase about two weeks ago. So, when we're reaching a milestone, we are putting that phase off, looking back at lessons learned, and then taking a small and kind of a mental break from it. We then come back to it a short time later. The phases are getting smaller, and that's also representative of the fact that we are getting better and more efficient. We have a better understanding of the solution and how our requirements stick within the Qualibrate solution. I actually expect this third phase to be as little as maybe six weeks. So, it started out initially as three months of learning the hard way and making mistakes. We've now whittled down to perhaps a month and a half for each of our phases. We still probably have another one or two phases to go. This will be something that would roll around probably through the new year. I hope to be done by January, and then we can effectively start using Qualibrate on a near-daily or perhaps a weekly basis.
In terms of the implementation strategy, our software is fairly mature. It has been around for 20 years. We kind of felt like using Qualibrate to effectively transition all of our manual testing to automated testing. It was like an elephant. How do you eat an elephant? It is one bite at a time. So, we tried to look at our software from a component level, and we said, "We've got this really big component, and we have a couple of other smaller components." We then basically tried to choose one that didn't have a massive footprint. It was important, but it was not so large that we felt like we were going to try and drown getting it integrated into Qualibrate. So, that was phase one that we focused on. Once we got that done, we moved to phase two.
We had to break down phase two and phase three into two sets. So, our last phase and the current phase we're on will cover our most critical component within the solution. So, this will be a big milestone. When we're done with phase three, we'll be able to look back and say 90% of this component can now be tested through Qualibrate. Phase four, and probably phase five, is going to touch smaller and less important modules that ideally we'll be able to roll out and get done in a pretty compressed timeframe. So, we might be done sometime in December, but because of the upcoming holidays at that time of the year, it probably is going to roll over into January.
What was our ROI?
For the project that I'm closer to, we're not realizing the dividends, but we know that it's on the horizon. We're just not quite there yet, but we're confident that we're going to have it. So, we are looking forward to that day.
For the parallel use case, we are seeing some early returns. The reason is that it is a bit of a project. My colleague was able to stand it up in a fairly limited number of hours. It pretty quickly and easily started creating or generating data in the SAP demo systems that were already being used by our pre-sales team. So, we are seeing some benefit from that.
What's my experience with pricing, setup cost, and licensing?
We probably have 10 licenses, but I don't know what are the costs or anything like that.
Which other solutions did I evaluate?
We did not evaluate other options. The reason was that we had contacts on each company who already had some sort of relationship. I think it was a rather informal relationship, but there had been some dialogue and some interaction in terms of people shaking hands and knowing each other by name, probably in trade shows in Europe. So, they knew each other. Because we already had kind of that rapport, we just looked at Qualibrate.
They did a short PoC with us. We met with them over a period of three or four days, and it looked like a really good solution. It was young but mature enough to do what we needed to do, and we were sold. Perhaps, we should have done some due diligence. We would have done that if we hadn't had that relationship already there, but I think that's what kind of clinched it for us.
What other advice do I have?
I would advise making sure that you understand the tool. You don't have to understand every aspect of it, but you need a pretty thorough understanding of its components, the playbacks, the designs, and the technical pieces that make up the solution. Our pain point was the very first phase. We were brand new to the solution. We went through some knowledge-sharing sessions with their support staff, but we really felt like we were struggling at first. Most of that was probably because we simply did not understand Qualibrate to the depth that we did in subsequent phases. So, it would be good if you could really spend a little bit of time upfront, maybe just from a sandbox perspective, and you test it, mess around with it, and get familiar with it. After that, you can start to decide and design how your solution and how your scripting should look within Qualibrate. This way, you'll avoid some of the pitfalls that we ran into. We found ourselves having to do rework over and over again, and that was simply because we learned a better way of doing it than we did initially.
You can maybe test it on a smaller scale. We were so motivated to get started that we tried to run before we could crawl. We should have kind of spent a little bit more time doing that initial build-out testing. We should have made sure we were happy with the results and that we were taking the right approach. We did all this work and then went, "Oh, that's not really going to work." We then had to go back and do some redesigning. So, some upfront commitment would help there.
It is fairly advanced. It has only been around for three or four years, but it does seem like a pretty mature solution. Most of it is pretty user-friendly. There are some components that I have found to be a little bit more technical and a little bit more daunting to use, but overall, it is quite easy to use as far as this kind of scripting software goes. I'm quite happy with it. I'm not overly technical, and I can understand it fairly easily. That must make it pretty good.
We're not using it on a daily basis. We're still kind of in a project phase where we're still ramping up and getting all of our scripting and recording done. So, right now, everything is more in kind of a batched process or phased process. We do hope to get to the point of being able to use it for daily or near-daily testing, but we're still a little away from that. Since we've only had it in-house for seven or eight months, we're just not using it on that frequency just yet, but that is our long-term plan. There are a couple of components that we are not really using at this time. That's because they are still immature, and they have not been around very long as part of the Qualibrate solution. We'll get to those perhaps another day. We're focused on our primary use case right now.
We are quite happy with the flows and scenarios and those building blocks that really represent the core of the solution for scripting and playbacks. So, we're overall quite happy with this solution at this point.
Based on how it operates today, I would give it a nine out of 10.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Other
Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.