Parasoft SOAtest Reviews

Repeatable automated regression tests in a SOA architecture IT landscape, for several SOAP/REST webservices including Web Gui end-to-end testing, all scenario-based for 10+ applications. Also, scheduled for test runs at night, using Jenkins.

It provides us with fast(er) test results, more predictable results and automatic (regression) test runs outside the employee workstations. This tooling made us more agile and more ready for continuous integration. We use the night hours to run different regression tests over systems, for which we only need to check the results in the morning.

• Fast and easy automated tests for UI
• Interfaces (SOAP, REST)
• Additional database queries
• Easy validations on responses (soap, rest, json, sql, Gui)

Most valuable for us is the combination of all these items in one solution for us. Broad user story scenario's from A to Z, though a chain of systems, including test data prerequisites.
The next most valuable benefit over the past year is the possibility to use the tooling for continuous testing purposes, using the server-side run option 'as a web service' for (nightly) batch runs.

In general, its user-friendliness needs a little improvement since there are a few bugs that still remain. It needs to be (possibly) more lightweight and have more validation options. The bulk changes in the test cases are not always easy. But all in all, these are minor details compared to the benefits that we have.

Very seldom, there were some stability issues but they do appear only from time to time. Since the 9.10 release we haven't seen this any more. Stable!

We only encountered scalability issues with large test scenarios, combining interfaces with a lot of browser GUI tests. Work-around is cutting up the test cases into smaller scenarios.

The technical support is fairly okay. Technical tool-wise they help a lot if needed, but when it comes to complex test scenarios or weird issues, they do not always have a solution.

An exchange of examples is difficult, since we use tooling for internal applications. So, it is not accessible from the outside world, which makes combined debugging impossible.

Initially, we were using some lightweight open-source (free) tools, but none met with our requirements fully; so we switched solutions.

The setup was fairly easy. They provide good instructions. The workstation install is a piece of cake and some additional (simple) config is needed, for Git/SVN etc. The license server install can be tricky, but it turned out to be fairly easy compared to the earlier versions.

We did it our self, with a little help from the seller.

Use floating licenses, so it’s not attached to one workstation. Furthermore, the pricing is something for the staff to handle, not for us. ;)

We mainly looked into the SoapUI solution, along with some Selenium software and besides that, other stand-alone database tooling options. We also evaluated manual testing.

Start off simple. Build your test setup from there. Do think about setting up some specific test cases as building blocks (reference material) for other test scenarios to use, so that the maintenance in the future is more easily done.

Also be sure to read the Help function, might help you on your way when you're facing some difficulties.

Automated execution and chaining of test validators is the most valuable feature; it is easily extensible via the built-in custom scripting API.

It has allowed us to build a large library of repeatable automated tests that are able to be quickly executed on demand. This builds a solid foundation for future initiatives expanding into a more continuous integration and the agile methodology adoption.

Memory Management suffers when using solutions other than the data repository for managing the test cases.

The platform can become unstable as the memory usage increases. For example, we use Excel spreadsheets for managing test cases, so the size of these test cases must be tightly controlled.

Scaling tests beyond a certain point can trigger issues with Memory Management, but good design practices usually indicate a need to break up a test before it would reach that point.

Technical support from Parasoft has been excellent.

The setup was straightforward. It was very easy to set up a test scenario that consumed assets in our web service layer and to run validation against the results. The tool itself allows for some more complex chaining of components within a test scenario, so familiarizing yourself with the tool enough to get more creative in your test setups can be more complex.

If you plan on using a floating-license model, plan on a buffer of at least 10 user tokens between the maximum allocated by your license and the expected usage for the tool.

I looked at SoapUI NG Pro.

Keep your test scenarios small and focused rather than trying to cover an entire web service with one scenario.

The facilities for testing API’s (services, REST API’s, messaging clients, etc..). Especially the features for parametrizing requests and checking responses are very powerful. But the product really shines in end-to-end testing, not only because of all the built-in tools supporting this, but also because of the great extensibility of the tool.

If you want something that’s not provided out of the box, then you can write it yourself and integrate it with SOAtest.

As an expert user, you can accomplish a lot with this tool.

Using this tool, we were able to set up a nightly, still growing, company-wide regression test -- dramatically decreasing time being spent on (manual) regression testing.

• Web-testing facilities. This was not very good when we were using version 9.2, so we built our own Selenium framework and integrated this with SOAtest (this being possible at all is a very positive point in my opinion ;-).
• The user-interface is not very lightweight or friendly, sometimes buggy and cumbersome. The product has a steep learning curve, but very well worth the effort.
• There is virtually no (online) user community, except for an outdated forum. When having issues you have to find everything out yourself (or call in help from Parasoft).
• Reporting facilities can be better.
• Last but not least: using the tool in current continuous integration/delivery practices is not very easy or straightforward.

Five years.

Yes, the product can be buggy or crash sometimes. But not to the extent that it hinders working with it.

We had some issues with stability and performance -- with a growing testbase and/or ‘big’ tests. But these also were minor issues.

8 out of 10.

Technical staff living nearby are always prepared to come by and help with issues. Communicating with developers abroad is also satisfactory.

Not really, QTP was used a little before but was not the right tool for our purposes.

It was complex, but that had more to do with our own complex infrastructure than with the product itself. But it took some time to get the setup of the product right.

It is an expensive product, so think carefully about whether it fits your purposes and is the right tool for you. Support pack is helpful. Due to high license costs, the tool is not easy to use by many people in your organization (like with open-source tools) and so to make test automation something for the whole team.

Yes, but before I was involved, so I don’t know more than that.

Take your time to get to know SOAtest very well, don’t get put off by the steep learning curve, it’s very well worth the effort.

Automatic testing is the most valuable feature.

Compatibility with HTTP 1.1 and TLS 1.2 needs to be improved.

I have used SOAtest for one year.

We get a lot of time-outs and after we start the run again it goes well.

We have had scalability issues.

I rate technical support 8/10.

It was our customer’s choice to use SOAtest.

The tool is complex.

We followed our customer’s choice.

Think twice before you start with it.

The most valuable features are the Browser Playback tool for recording and playing web scenarios (Web UI testing), and the messaging client for sending SOAP calls and for validating responses.

The testing time is shortened because we generate test data automatically with SOAtest.

During the process of working with SOAtest and building test cases, the .TST files will grow. A negative side effect is that saving your changes takes more time. The bigger the .TST file, the more time it costs. Sometimes a few seconds! An option is to split up the file into separate files, but often that’s no realistic option because you want to keep the test suites together.

TST-file is the container file in which all testsuites and teststeps are stored. It often happens that you have to change a teststep and want that change to be saved. Saving your teststep in an almost empty container (with only one testsuite and a few teststeps) will go very fast (within 1 second). Saving your step in a container with multiple testsuites and teststeps takes a lot more time. In my case sometimes five seconds.

Sometimes SOAtest hangs, but that is very sporadic. It could be caused by the stability of the environment of the customer in general.

We haven’t had any scalability issues.

The technical support in the Netherlands is very good. Questions about the tool are almost answered within a day. When needed, a Parasoft consultant joins the project for one day or more.

We used SoapUI for testing web services. In our case, the advantage of using Parasoft SOAtest is the use of the datasheet; with one message, you can loop through your datasheet and check all your data.

The setup was done by another team of the company and was out of our scope.

The products of Parasoft have the name; they’re expensive. I think it would be a great step to decrease the price of the licenses. A possible consequence could be getting more users and more familiarity in the Netherlands and around the world.

I have experience with a lot of other products, but when I started with this job (Dutch Government) three years ago, the product, SOAtest, had already been bought by the company.

Invest a lot of research time into what your exact needs are. After that, select some test tools to compare all the options. In my opinion, Parasoft has so many possibilities that it must have a place on your shortlist!

SOAtest is, at its base, a collection of tools that can be combined to generate test scenarios.

Input data can be gathered from any possible source from web GUIs, databases, Excel, and files and is used in SOAP/REST tools and web browser tools. The output of those tests can be saved in files and databases.

Every imaginable source in the entire world of information technology can be accessed and used.

The following features are particularly valuable to us:

• Generating SOAP/REST requests
• Validating responses against WSDL
• Scripting parts of an XML request
• Accessing DBs with JDBC
• Jenkins Continuous Integration
• Regression test automation
• Modular scenario setup

Our tests are run nightly on every environment using Jenkins CI.

We moved from the Waterfall/V-model to Agile a couple of years back. With SOAtest, we are able to test features in every sprint, quickly and efficiently.

We use SOAtest to test the interfaces of maybe a dozen back-end services, simulating multiple, different points of sale. For example, a customer web shop, a sales agent application, sales machines, or mobile apps.

Service interfaces are continually integrated and updated within SOAtest. We test most front-end features first with manual tests, and then integrate them with validations into SOAtest scenarios.

Before using SOAtest, we had a huge delay in acceptance testing due to the amount of manual regression tests that we had to do.

Sadly, almost everything needs improving. Don’t get me wrong, the tool is very powerful and immensely useful for day-to-day usage. But certain things are missing or are set up in a way that makes it unnecessarily complicated. Here are some examples:

• Replacing part of an XML request by script requires a variable in the scenario folder. This variable needs to be prepared before the SOAP request tool with the partial XML to be replaced in the request. This variable can then be set to replace a subtree of the XML request. It would be easier if we could directly set a script in the SOAP request tool as a replacement.
• Enabling/disabling an optional element of an XML request is only possible if a data source (e.g., Excel sheet) is connected to the test. Otherwise, the option is not available at all in the drop-down menu.
• True parallel testing is currently impossible.
• There is a separate product ‘Load Test’ that has parallelism built in. But this is only to multiply a scenario with all test cases by a number of users executing all test cases. In a future release, there should be the option of running all test cases of a single test run in parallel to speed up testing.
• To build a modular scenario, you can reference test suite files and execute the test tools contained therein. But every test contained in the referenced test suite file will be executed, as it is set up in the test flow logic of this scenario.
• Sadly, there is no way to setup single operations of a web service and then combine those operations into a scenario. This is how SoapUI does it.
• SOAtest is a plug-in for Eclipse. But actually, it comes bundled with its own Eclipse and several other Eclipse plug-ins like PyDev. Although this ensures compatibility of all contained tools, it also means that every part is a little out of date. For example, PyDev is available as v4.5.5, which means you get Python v2.5.2.
• You can manually upgrade to a newer version, but this means every installation using your scenarios, and especially scripts, must be updated as well.

• We have been using v9.10 for about six months.
• We have used SOAtest itself for around five years (since v9.3).

We had a stability issue, but Parasoft support was always very helpful in finding a solution or a workaround.

I would give the technical support a rating of 10/10, or even 20/10!

No matter the problem, from simple user errors to complicated Eclipse issues, the support team is always quick to respond and is very good at what they do.

If you actually encounter some issue in which the first or second level support really can’t fix in a couple of minutes, then the third level support will dig deep and come up with amazing ways to help you.

There is a user driven support forum where you can get help and examples from other users.

We did not use another solution previously. We do use SOAtest and SoapUI in parallel:

• SoapUI is used as freeware by developers for single/simple SOAP requests.
• SOAtest is used by the testing team to build extremely complex scenarios.

Installation is easy and straightforward, as is the initial scenario building. SOAtest gives you the ability to choose how complex you want to build your scenarios.

The simplest level of usage is a full graphical view of requests and responses, storing values in data bank tools for use in later SOAP/REST request and SQL tools.

The more intricate levels consist of scripting in several available languages and really revealing the beauty of SOAtest.

The documentation guides you through several examples and helps enormously with the initial steps of building your test scenarios.

We evaluated SmartBear SoapUI.

• Learn XML as well as XPath.
• Learn Jython, a variation of Python running in the JRE. It gives you the option to import Java classes into your scripts and use the latest Java improvements, but without the type safe limitations and language complexity of Java.
• Implement your scenarios from the beginning with continuous integration in mind. For relative paths everywhere, abstraction in SOAtest environments etc., use a version control system to save and version your scenarios and scripts.
• Take a short course (less than one day) from the Parasoft support team to familiarize your test team members with the product very quickly.

We do a lot of web services testing and REST services testing. That is the focus of this product.

It has also enabled easy automation and concentration.

Working with the Parasoft team was very easy for us. They helped us get some of the solution customized, in terms of the things we needed, such as LDAP coding. We needed a special coding for that and we worked along with Parasoft to get that done successfully.

The primary use for us is the Web Services layer.

It enabled an easy integration of our adjacent platform that was for full automation. We did reach over 97% automation with the use of this tool. It was extremely lightweight.

The only unfortunate part is that as the organization no longer exists. Now there are different groups managing the tool.

Parasoft is not becoming the tool of choice for the next generation of software that we're building.

The feedback that we received from the DevOps of our organization was that the tool was a little heavy from the transformation perspective.

However, Parasoft seems to be working on a solution to make the tool slightly lighter. That is one of the things that could be improved.

Of course, it is a technical tool, so perhaps the interface could be better. We all have our struggles with any tools that we learn.

We implemented Parasoft over five years ago.

In the beginning of the implementation, we did have some concerns in regards to the stability. Primarily, the concerns were about the licensing. We were not concerned that the application itself would crash. We never really experienced that and that did not become a problem for us.

There were no issues with the scalability. It's a user-based licensed product. As long as you have enough licenses, you can just do whatever you need in order to scale it.

In the beginning, we worked very closely with their technical support. They had great tech support and they did road shows once a year for us.

They would definitely listen to the concerns that we had and provided relevant feedback. The tech support is not just great, it is absolutely awesome!

We were with HPE and they failed with their new product. It was not compatible with what we already had scripted.

The decision to switch was made in order to create more flexibility within the groups who were using the tools.

The product is great. Sometimes companies make decisions based on the profit and not based on efficiency. The cost of Parasoft seems to have gotten higher with a projection that wasn't really stipulated for our company. They've done a tremendous job at negotiating those deals. We'll see what happens next.

We did look into a couple of open source tools. We also have an array of other suites that we could use. For example, we looked at the SoapUI NG Pro suite and a couple of other tools out there.

We felt that Parasoft would be a mutual gain with our team, and with our performance team, because they would use a lot of the performance-center items.

That was primarily why the company agreed on making a switch, so that it would benefit both areas in which we had concerns with the HPE products.

My advice to anyone implementing this solution:

• Work closely with the tech support so that you don't get stuck on the little things.
• Since Parasoft provides tremendous tech support, work closely with them in order to get the first implementation and proof of concept done.
• You can then export from there based on your needs.

• Has the ability to add regression controls and scripts in different languages for parts of the test script.

• Regression controls help to create a baseline response and ignore the dynamic values, so we can compare against any future builds. We can tell the impact of any change right away. We can tell which APIs we need to focus on for the API backward compatibility. This feature was not found in any other product that I have used.
• The scripting support provided for Java scripts and languages that rely on Java is very helpful. This skill is easy to find in the industry.

It used to take a couple of days to run several scenarios of the billing project at AOL. I was able to automate a collection of 72 APIs and the back-end batch jobs, so as to run the whole thing in under 30 minutes.

I was having trouble with the browser automation on my Mac laptop. They need to make it easier, so as to configure and run this tool on Mac.

I have tried to automate purchasing QuickBooks from our corporate intranet. I could not signup for the Free Trial from the below URL, due to security certificate issues:

https://quickbooks.intuit.com/...

I got busy on other projects and did not revisit this. Apparently, I did not have similar kind of issues while using the Windows OS laptop, in the past.

There were no stability issues.

There were no scalability issues.

A few years ago, I would have given technical support a rating of 10/10. However, I would now give them a rating of 7/10.

Its ease of use and the variety of protocols that it supports are the main reasons why we switched solutions.

The setup was straightforward.

We evaluated several home-grown and open-source tools such as JMeter, Postman, and SoapUI.

This is the best tool for API and back-end automation.