What is our primary use case?
Most of our automation is for REST API, and it involves requested response validation, database validation, and some level of UI automation/validation. The UI is done using Selenium, and SoapUI Pro is used for database validation and file-based validation.
How has it helped my organization?
We run around 10,000 different validations, including database validations, and we can complete the entire aggregation in less than one hour. This includes file validations, database validations, and response validations. All 10,000 plus the test conditions include all these three types of validations.
What is most valuable?
The out-of-the-box support for the database is a valuable feature. For example, SoapUI open-source does not have any out-of-the-box support for doing database validations. However, the commercial version of SoapUI Pro has out-of-the-box support for the database.
Along with that, creating an export or import is very much easier in SoapUI Pro compared to that with the open-source version.
SoapUI Pro is an easy option for non-technical staff as well. I'm currently leading a team in a front project, and I've seen that even a non-technical person with pre-guidance can easily pick up the automation. It is very easy for any person to pick up the automation, and that is something that I like about the tool.
What needs improvement?
It is limited to scope and risk services only. It does have some support for JMS, but it is not out-of-the-box; you have to do some tweaks here and there. They should improve the scope, and the message should not be limited to SOAP and REST services but also include other protocols and message formats so that people who have other requirements, other than SOAP and REST services, can also think of the solution.
Otherwise, if you have 15 to 20% of automation other than SOAP and REST services, you would have to go to some other tools because it is not fully covered by SoapUI Pro.
As for pain points, reporting is one. It's a bit difficult to filter out or to customize the report. It will be an either everything or nothing kind of report. Reporting also should have a flexible customization option.
When I do a 10,000 testing execution, I don't have the option to filter out the report that is generated. This makes it even impossible on a Windows system to open that report because of the size of the file. It will capture each and every request and each and every SQL period, as a result of the SQL period. So, it is too heavy for Windows to handle it.
For how long have I used the solution?
I have been using it for several years.
We have two kinds of setup: one is for the automation expert, who does the scripting using his workstation, and the other is for aggregation, for which we have it on the cloud. It is on the AWS server (private, specific to a project), which is used for CICB, a tool from the script using Jenkins. So for DevOps, we are using it on AWS.
What do I think about the stability of the solution?
I don't see any issue with the setup or the working of the tool. We have never faced any major hiccups.
What do I think about the scalability of the solution?
It is customizable with some scripting. We have to have an expert in Groovy or JavaScripting to do the customization. The users of the tool, any team member, with a little bit of guidance and training can do the automation. However, when it comes to customization a team member with scripting experience is required.
We have both junior and senior team members, and associate level staff who are using it, including senior associates and a couple of team leads. This includes both solutions, SoapUI Pro and ServiceV Pro. Altogether, we have around 15 to 17 members in the team who are using the solutions.
How are customer service and support?
I found that there are lots of communities, so it's very easy to get help. You can go to the forum, and you can also reach out to the technical support team from SmartBear. They will respond maybe in a day or two; they don't want to take that much time. We post it on the community forum as well. Often, someone has already faced the same problem and has a solution that was implemented. So, we just have to get the solution from them. It's very easy to address those issues with API.
Which solution did I use previously and why did I switch?
We started with the open-source version, but later on as vendor services increased, the scope increased. So, at that time we migrated from open-source to the commercial version of SoapUI Pro.
We are not just using SoapUI Pro; we are also using ServiceV Pro, which is part of the ReadyAPI umbrella. So we are doing virtualization and also API automation using the same tool. For us, it was easy to have a single tool or tool set from a single vendor with all those options included with virtualization or API automation.
How was the initial setup?
Initial setup wise, the local one was straightforward, but doing it on AWS took some time for us. We were doing it for the first time on AWS, so it took some time for us to understand and complete the setup.
The setup was quick and easy, but integration with other servers, for example, the Jenkins server with the AWS server via SoapUI Pro, took some time. Otherwise, installation on just the AWS server took just a couple of minutes, but integrating it and having a working setup took almost a month for us, including all the approvals and the delays around the approvals.
What was our ROI?
We initially started with the open-source tool, and the automation which we have covered using the commercial version is far better than that by the open-source version or any other tool, that is, 10,000 test cases in less than one hour. That is very impressive.
What's my experience with pricing, setup cost, and licensing?
The licensing is one area with which we have a problem. To give an example, we have team members who are working in shifts. For example, team member A, works the morning shift and team member B works the afternoon shift. It is not possible for us to utilize a single license on a single piece of hardware so that both team members can use it. We have to take out two different licenses for each of the members.
We have found this to be problematic for us because if we have 20 members and 10 members in each shift, it is not possible. The cost also increases when we try to have 20 different licenses, whereas the hardware or the machine is the same machine, which is shared in two different shifts.
Which other solutions did I evaluate?
I used IBM Green Hat, Datasoft, etc.
We also use ServiceV Pro, which is from the same vendor, SmartBear. ServiceV Pro is used for stubbing, mocking, or virtualization of the services, whereas SoapUI is used to automate the services or the API.
So, if you want to do automation, you can use SoapUI Pro, but if you want to remove any dependencies related to other services and you want to mock or stop it, then you can use ServiceV Pro.
What other advice do I have?
Your choice totally depends on what kind of scope you have. If it is just limited to SOAP and REST services, then SoapUI Pro is the best option. Also, the cost is very much less compared to those of other solutions, for example, DevTest, which is the most expensive tool on the market.
I would recommend that if you have any SOAP or REST service automation requirements that you should try out ReadyAPI, which has an evaluation version of SoapUI for 30 days. You could try it, and if you feel that it is a good fit for your automation requirements, then you should definitely think of SoapUI Pro.
I would rate this solution at nine on a scale from one to ten.
*Disclosure: My company has a business relationship with this vendor other than being a customer: Partner