Worksoft Certify Reviews

Our eCommerce platform is Hybris. We run end-to-end tests where we place orders in Hybris, then we validate the order in ECC. Additionally, when an order is placed on Hybris, our QA environment has a lot of things which the SAP analysts have to prepare to get an order ready, so it doesn't clutter up the system, such as creating deliveries. Worksoft can do this for us as well.

Hybris is out most modern application. Our point of sale system is web-based, and it is in web form. We are on Azure. One of the things that we've been able to do is use Jenkins to put our Azure machines on business hours. We tell them to turn it off at 5 PM, then we tell them to turn it on at 7 AM. This has saved us about 62 percent of computer operations.

Prior to Worksoft, there were three different individuals within the company who worked collectively for about three years trying to automate just one of our smoke tests for our point of sale system. A lot of them got pretty far, but they weren't able to finish. However, with Worksoft, we have been able to automate six of our smoke tests in four months.

The most valuable feature is having a no-code solution for automation, so our QA team and some of our business users can work on automation. Then, they don't have to be developers.

Most of our SAP analysts use LiveTouch. They use LiveTouch along with prebuilt components. Our QA team uses LiveTouch when they need to add things.

I would like to learn how to get better logs for their support team.

It is fairly stable. We have run into some intermittent bugs off and on that we can't explain. Since they are typical Window's stuff, you just kill them.

When it is unstable, there will be times when a test that we are running in Certify will just stop, and it will say, "Aborted." There will be errors. There will be no explanation as to why it happens. It has now happened maybe one out of 20 times. When it happens, I just tell our QA team to stop Certify and restart it, hoping we don't see it again.

Scalability is throttled in part by the system that you are testing. So, how much testing can your system handle? 

I think they came out with a different type of licensing specifically for testing. Therefore, you don't have to use a more expensive user license, you can use an automation license. So potentially, if we had 100 use cases, we could spin up a 100 different machines, have them all run and be done in five minutes. That would be the goal, but I don't know if that would actually succeed or not.

For the couple of the issues that we were really scratching our heads over, we were in communication with the technical support several times, but they never got back to us. The issues are not critical because they're not really blocking anything. They're just annoying.

We needed a first step in order to get into DevOps. The first step was being able to automate our smoke tests and regression tests. They are tests that we use to make sure that our SAP environments are viable and our point of sale system. We chose Worksoft because they were the only people who we could find which were capable of automating SAP right out-of-the-box.

We needed a faster feedback loop. We have a third-party who develops our Hybris application for us and wanted to be able to hook into their Git repository, so when they push a new version, it would automatically deploy and run our smoke tests. Then, I can know within ten minutes if it works.

The new environment was pretty straightforward to set up. There were four servers, and maybe a fifth one, if you wanted to have a separate server for automation testing.

Some of the integration depends on the subject matter expertise on your team. How well do they know ECC and their processes? Then teaching them how to use Certify to build out their processes. So, on a scale of one to ten, it is probably a seven if you are not familiar with some of the development principles, like looping. If you are not familiar with them, then it will become more difficult to build out processes needed. This is just understanding the methodology of doing certain things, not Worksoft specifically.

The IT department and I worked with a Worksoft deployment engineer because of all of our Worksoft infrastructure. She walked us through setting up the database in SQL and the MongoDB with Worksoft Analyze. It was a pleasant experience. Most of the issues that we ran into were because I did not know something.

Because we haven't built out our suite of tests yet, we haven't saved that much time. However, we know that it will allow us to save a lot of time and money, because once we are fully DevOps, we'll be able to spin up and spin down our systems on demand. Then, we will know within 30 minutes whether the system deployed successfully or not.

We did not evaluate any other vendors. We didn't find anything else that did what we wanted.

if you can use Azure or AWS for your Worksoft infrastructure, then use that for ease of deployment. Once you have your environment, then you can save it using Infrastructure as Code. Thus, if you needed to rebuild or repurpose it, you would be able to do it.

We haven't taken advantage of all the current functionality.

We hardly use the Capture 2.0 feature at all.

We have been using Worksoft Certify for ten years.

We do automation of both SAP and other applications: One of them is our new grants management system. Originally, they wanted to do everything manually. Now, they highly rely on us to do testing in very short periods. 

We do web UI testing of modern applications. In our environment, we have SuccessFactors, plus SRM through Fiori. 

We also do performance testing and end-to-end testing of packaged applications, like SAP, InfoEd, and OnCore.

The most valuable features right now are the Capture capability and versioning. The Capture 2.0 has a lot of flexibility.

The easy of use and ease of integration of Worksoft Certify are very good.

We are looking for some enhancements on the Capture 2.0 tool. This would give us the ability to control it directly, like we could with Capture 1.0. Right now, Capture 2.0 doesn't really work for our Business Analysts. Because they don't work with it all the time, it's a little complicated for them to stay up to speed on it. With Capture 1.0, we wrote a wrapper to make it easier for them to use, but we can't use that wrapper with Capture 2.0. So, if Capture 2.0 gets enhanced, we'll start using it. For an enhancement, we want to be able to start and stop recording through an API. Then, we want to see how many steps have been recorded through the API.

We do a lot of test maintenance because they are constantly changing the applications. This is one of our biggest problems that are constantly making changes and switching products. For example, we used to use the Supply Relationship Manager. Now, all those tests that we build there will be replaced when we go to Ariba. All the old SAP GUI stuff for HCM, when we got SuccessFactors was thrown out, then had to be redone with SuccessFactors.

It gives you the ability to have a more structured environment for tests. It is not just recording of key strokes. It is more systematized, more like a programming language. That is the biggest advantage for us. Because of its consistency, once the developers know one skill set, they can use it to automate any application.

It scales well. We have eight offshore people and four developers in New York. We can add developers as needed. With the offshore team, we've have at least 20 people trained on Worksoft, but eight people actively on the team now.

We have about a 1000 active users with about 5000 users total. However, this includes all of the employees and their self-service.

The technical support is very good. They have always been a great help. With a lot of companies, you can't even really talk to the developers that someones provides you. With Worksoft, you can get a hold of the developers and work with them. It is all very timely.

The initial setup was very straightforward. We just build the database and installed the client application on the system. There was nothing out of the ordinary about it. It is not like setting up a lot of the IBM products or the Quality Manager, where there were so many different modules. There was just one module to set up.

During our yearly upgrades, we have now gotten them down to ten days or less. We have Worksoft run all our integration tests, where it used to take probably six weeks to do that manually.

As far as maintenance testing, it has saved us time. We find problems a lot of the time that they aren't aware of. This is because we run the tests even though they don't ask us.

We evaluated Micro Focus QuickTest Professional (QTP).

Properly staff the testing team before they attempt to do automation. Be aware that this will not be a one-time overnight process. We tried to automate everything in two months with eight people, and it was impossible.

We have been using it for so many years that we are really very happy with it.

We will be converting to Ariba for purchasing, so that will have to be automated.

Worksoft Certify is being used to run automated weekly regression tests across some of our primary SAP systems in line with our Change and Release management strategy. These tests run every weekend without fail. The results are reviewed on every Monday morning to check for failures and to analyse if any failures are associated with the changes scheduled to be transported to the production environment that week. Failures (if any) are fixed and the tests re-run before transporting the associated changes into the Production environment.

We also utilize it for projects that need extensive business-user testing and functional testing. There can be testing requirements which come at short notice which can take three to four weeks of manual testing effort. By using Certify, we have been able to bring timescales down to a few hours of automated testing effort.

Our final goal is to utilise this for 'Unify', our new global solution which is currently being deployed, which will deliver common processes and systems to all sites and sectors, replacing all our existing legacy systems which will demand extensive regression testing. 

We have never had any systematic regression testing regime in the organisation. This has helped in building an automation framework across our SAP application landscape, thereby introducing mandatory regression testing across all our key systems and improving the overall quality across our production systems.

From an audit perspective, results generated from Certify (BPP reports) provides detailed test evidence which is also being utilized for internal training purposes/training guides, etc. The BPP reports also provide details on failures along with screenshots.

We have a variety of complex systems in our landscape, one of them being the Openlink Endur which is a commodity trading and risk management system. We are currently building an automated regression test suite to support application testing for Endur.

Our weekend regression tests are performed in 'lights-out; mode. Tests are scheduled to run at a certain time over the weekend using the Execution manager functionality. Usage of Certify has also prevented some major defects going into Production and we have seen significant savings in all manual testing activities as the business users/functional teams are getting more time to perform 'value- adding' activities.

Post our recent upgrade to Solution Manager 7.2, we are currently in the process of implementing the Test suite functionality and the integration of the same with Certify. We expect Solution manager to be the single source of truth bringing out all the results from Certify which is going to be extremely beneficial from an audit perspective. We have already implemented the integration of Certify with HP ALM in our landscape.

Moreover, we have this reusable asset now which can be run frequently to support all our projects and change requests across our legacy SAP systems. Even last-minute testing requests are being accommodated by utilising the automated regression suite without any dependency on business users/functional users for their efforts. We use it across the multiple projects which need immediate assistance and for our weekly regression cycles. To give an example of a recent project which was a major platform migration from a Data Centre in Asia to Europe which needed extensive Disaster recovery testing and Functional testing/User acceptance testing. The initial testing estimate was approximately five to six weeks, however with the use of Certify we could do extensive testing in less than three hours saving many weeks of manual testing effort.

It is very user-friendly with an appealing UI, unlike a lot of other automation tools that we have evaluated. With sufficient training and adoption of best practices, the tool will certainly help organisations to successfully implement an automated testing framework and eliminate manual testing activities.

The fact that it can be used across SAP and non-SAP applications (including web-based apps like Web Dynpro) is a big advantage for us because we have a variety of SAP and non-SAP applications across the Johnson Matthey IT landscape. Being a 200-year-old organisation, our variety of legacy systems have a lot to benefit from the use of automated testing.

Certify has many interesting features, e.g.: 'PRIMO' which is the image recognition functionality is a life saver in instances where Certify standard functionality cannot identify and learn objects within certain legacy applications.

Regarding end-to-end testing of packaged applications, Certify is primarily used across our SAP application landscape and the Openlink Endur (commodity trading and risk management system). We hope to realise more benefits by implementing Certify across our wider application landscape over the next few months.

We have been using the Capture feature, although not the latest version, the initial version, for process captures was used to create our test designs. It has been a life saver in many instances, without the need to spend any extra effort to create test designs and captures. The test steps get captured in the background which generates an XML file which can be easily imported into Certify, creating the basic test structure which can be improvised/modified to make it a repeatable reusable test. In terms of the amount of time it takes users to create documentation automation using this feature, it is the same amount one would spend to do a manual test. While a person is performing a manual test, Captures are automatically generated in the background. We have used it extensively to build our test designs.

We have requested for some minor new features which Worksoft is considering.

The PRIMO image recognition functionality has room for improvement, especially around its ability to work with java interfaces, Execution manager scheduling, etc. as we have observed. As we explore more of our legacy systems, I am certain there will be a need to use more of the PRIMO features to learn the objects.

Overall from a SAP perspective, it works almost seamlessly.

One to three years.

The version of Certify that we are using has been mostly stable and we have rarely encountered any problems. Our weekend regression test failures are often associated with environmental/system performance issues and not related to the stability of Certify. I have been happy with the overall performance of Certify and how it has helped to optimise our tests.

I am confident that Certify can scale to fit our automation testing needs as we expand the current automation testing framework across the wider Johnson Matthey application landscape. We are also exploring options to identify potential areas where Certify can help support mass data uploads, etc. to benefit other teams in their day to day operations.

We have several concurrent users accessing Certify in our environment, primarily automation engineers, test engineers and tech managers.

Our interactions with technical support has not been the best always and there is room for improvement especially with respect to the time taken to respond to cases. However, with the right contacts and reasonable escalations we have always managed to get quick attention on our issues.

We have always been heavily reliant on manual testing and as a result, regression testing was not systematic and we could never think of implementing frequent weekly regression test cycles which was challenging. We decided to go ahead with automated testing and use Certify because:

1. Manual regression testing takes a lot of time and resourcing is always a challenge.
2. Regression testing not being systematic, the quality was very difficult to measure as we did not have a standard set of manual regression test scripts/sufficient documentation.
3. There was a delay in our time-to-market because all the testing was being done manually and there was no way we could accommodate frequent, weekly, regression test cycles. That meant high business risk, that we would have more defects in the production environment/ more associated costs.

We had all these challenges and we started exploring options to mitigate these risks and automation was identified as the way forward, nearly two years ago. We evaluated various automation tools in the market. It was critical that we had to identify a strategic tool which would cater to our SAP and non-SAP application landscape. Worksoft Certify came in as a big winner ticking most of our requirements.

We went through a lot of initial challenges, mostly around internal resourcing issues. Looking back, I am happy to say that we could overcome these challenges and have managed to successfully implement an automation framework using Certify.

Early in 2017, we decided to go ahead with Worksoft Certify post evaluation of multiple automation tools. Our initial engagements with Worksoft consisted of several onsite workshops to explore the tool in detail along with technical feasibility assessments across our application landscape. These engagements were extremely beneficial and it gave us the overall confidence to adopt Worksoft Certify as our strategic test automation tool.  

We did a pilot implementation with Worksoft to see if we could take this ahead on a large scale before embarking on the major project to build the automated tests. Some key processes across our critical SAP systems were identified as candidates for this exercise. Test designs were created with support from the functional teams and taken ahead for automation build with Senior Worksoft consultants and our internal resources. This 7-week Automation Roadmap Engagement exercise was extremely successful and we learned a lot of lessons from it which helped us plan the next big phase of the automation roll out. It gave us overall confidence across the functional and management teams which subsequently led to securing the appropriate budget, etc.

One of the biggest lessons learned from this engagement was around the ways to structure our teams. This led to us going ahead with a Managed Services model with Worksoft. We have an offshore based Worksoft Automation Services Factory team who helps build our automated tests. The team can scale up/down based on our automation forecasts.

The automation deployment is still ongoing. The initial phase was completed across a five-month span. Currently we are rolling out the second phase of the automation build focusing primarily on our global Unify solution and the Openlink Endur application.

Regarding implementation strategy, we followed an agile two-week sprint approach. Our functional teams continuously created test designs and these were fed to the Automation Factory every two weeks, who in turn developed the automated tests. This was the most practical model, which worked well in our environment.

At its maximum capacity, we have had approx. 10 to 12 automation engineers in the Factory team. Our functional teams are spread across multiple global locations and we had between 3 to 6 resources working on test designs liaising with the business users as required.

From a script maintenance perspective, we spend an average of 4 to 5 hours every week with the current asset of nearly 800+ tests.

We have always worked directly with Worksoft, along with support from our internal resources. Worksoft has been delivering excellent services through their managed services model.

We have a res-usable re-runnable asset built which is saving a lot of time across the functional teams/business user community.

Our final goal is to utilise this for 'Unify', our new global solution which is currently being deployed, which will deliver common processes and systems to all sites and sectors, replacing all our existing legacy systems, which will demand extensive regression testing. 

It is expensive compared to some of the other automation tools in the market. However, the benefits and ROI has proved that it has been a good investment.

We have concurrent licenses.

SAP TAO and Micro Focus UFT.

It was critical to identify a strategic tool which would cater to the testing requirements across our SAP and non-SAP application (including web based apps like Web Dynpro) landscape. Worksoft Certify came in as a big winner ticking most of our requirements.

It is a great product and we have not seen anything which cannot be automated till date in our application landscape.

It is important to do sufficient technical feasibility assessments before deciding to go ahead with Certify and equally important to determine the best implementation approach which will work for your organisation. Functional teams/business users' buy in is critical as the test designs cannot be created without their continued support. Adoption of best practices around naming conventions/folder structures etc. will help in easy overall maintenance of the test assets, which will also help with the generation of development and execution dashboards/overall reporting.

I would rate Certify at eight out of ten. Worksoft has always been very supportive and responsive to our needs and this has certainly helped us achieve our initial milestones successfully. I am extremely proud of what has been achieved so far and looking forward to expanding the automation framework across our wider IT application landscape over the months ahead.

We have developed some end-to-end regression testing scenarios that we've found pretty valuable, so we have created a bunch of processes in Certify, linked them together, and we use them every week - sometimes more than once a week - in regression testing.

We used another tool for many years. It became unworkable because of the length of our scripts and how many of them there were, and how they were linked together. They became cumbersome in the other tool. It's much easier in Certify, and Certify can handle them, no problem.

In addition, our organization is implementing Agile, we're moving towards continuous development, and I don't see how we could do that, in any imaginable way, without Certify. We're able to import our changes weekly, based on the results in Certify. And we're confident that because of having tested the main business processes, fairly rapidly, within one day, we can tell whether the imports are going to break anything.

It has absolutely enabled us to automate and save time. The weekly imports of the changes allow the developers to plan on a weekly cycle, which increases the speed of their development. They don't have to wait for a release or anything else, they can test their changes quickly and get the results the next day. They know that they're able to import with no problem.

Finally, it has helped us cut test maintenance time. 

The idea that it's not language-specific is really nice. Keywords and the drag-and-drop functionality are great. The ease of use is superior to anything on the market. It's very easy to integrate. We've been very impressed with the tool. Because we primarily use the configuration with SAP, the integration is pretty seamless. But we have used our own in-house VB app as well, and it's worked very well with that. We've really not seen any problems whatsoever with integrating.

We love the Capture 2.0 feature. It seems to work very well. As for how long it takes to create documentation using it, we do not get into the documentation so much. That end is not as useful to us. But it's built-in if we ever needed it. We're not USDA or anything like that so we don't have a super need for documentation right now.

Also, the ability to work with the data, with recordsets, and plug those into the scripts is very easy and very powerful. We use it extensively.

In the past, when we've tried to automate some of our web apps, it has not been as robust. If there were one thing that could be improved, it's interaction with web applications. The issue we were running into is that it was harder to identify the objects than it is with some of the other architectured applications. That's the nature of the beast with the web as well.

It's very stable.

We have had some issues. We would be unable to log in, in certain situations. But they've all been self-inflicted, changes that we've made on our side that have prevented us from being able to use the tool at times. Once we got those resolved, we were fine.

It's fairly easy to scale, which is a nice thing. Once you create what I will call a sub-process, if you want to use that sub-process in many other processes, it's really easy to use. For us, that's what makes it scalable. You can use that same process wherever you need it. The use of the recordsets just allows us to be able to change the data that make it unique and that make it easily maintained. It's very easy to scale. It simplifies our workflow.

Worksoft technical support is very helpful, very knowledgeable. Whenever we've had an issue, they've responded very quickly. We don't actually have very many tickets, but whenever we've had them in the past, I've just gone into the portal and I get an email back, usually the very next day. I've never had to escalate an issue.

We were using Micro Focus UFT. It became completely unworkable for us. Our end-to-end processes were just too cumbersome for the tool to handle. It got worse and worse to the point where we had to say, "You know what? We have to change tools, this is not helping us." That's when we investigated Worksoft, and we were very pleased with how it worked.

The initial setup was pretty straightforward.

The last time we did it, for the upgrade into version 10, it just required me and one other person on the database side, and then the technical person from Worksoft. It was fairly easy. It took just a couple of hours. At that time, we were just upgrading. The basic architecture was already there so it didn't really require a project plan or anything like that. Once we got it set up, it was just a matter of migrating what we already had in UFT.

There have been several times where it has highlighted an important issue. Some of the defects we've found have been high-impact defects that would've really been costly had they made it to production. There are other times where, because we were able to test with Certify, we knew within a day whether there were gaps in the way we configured a change, things that we had missed that we wouldn't have been able to find if we didn't have the ability to test quickly.

That one defect we found easily saved us $1,000,000. That was just one. Over the years, the amount of money that it has saved us is certainly in that range.

The ability to test quickly has enabled us to develop quickly. We've been able to capture lots of savings in terms of projects that have been delivered faster because we can test faster.

There are savings on a lot of fronts because of this solution.

We would purchase more licenses right now if they were cheaper. Pricing is a little bit of a hindrance.

We didn't look into any alternatives.

It's a highly powerful tool. It's very customizable. It's not a cure-all for everything, but if you want to do end-to-end testing, regression testing, it's a great investment.

We use Certify for end-to-end testing of packaged applications. We have implemented almost anything that touches SAP, using Certify. When C4C came out, the customer application, we regression tested our existing suite to make sure that nothing would break. We anticipate doing the same thing with Success Factor. At the moment, we don't use Certify for web-UI testing, but we're planning on implementing some of that, coming up.

Since it has been up and running, we've had three people maintain it: Myself, I'm the principal QA person, and we have two offshore partners whom I've trained on Certify and they are now helping us execute and maintain the tests. It requires full-time maintenance. We have plans to expand the reach of our automated testing, so we plan on adding more people. We are the only three using Certify in our organization at the moment.

It tests our core business processes but we still have many core business processes that we would like to add to that, to validate if they work, before we send changes through every week. And we would also like to increase the speed at which we can add changes; not just once a week, but eventually daily. We plan on increasing our resources from a manpower standpoint and also from a technological standpoint. We're just going to try to do that as fast as we can. There are a lot of business processes that we would like to add, a lot of apps that we would like to add. The business side has continual, increased demand in terms of things that they are working on and they would like to automate and not test manually, so there's a lot of demand on us right now.

I would rate Certify at nine out of ten. I rely on it every day. It's a great tool, and any problems that we have are hardly ever attributable to the tool itself. It's always some other factor; the way we're using it, or some external factor, which is the problem. It's nice not to have to worry about the tool being the issue. We're very enthusiastic users.

We are using Worksoft Certify to enable our stakeholders to do test automation on the UI level.

We use it for end-to-end testing of packaged applications. We are part of the internal IT department within our company. Most of the time we are using it on our own products. The products and systems that we get are usually preconfigured and prepackaged and we do additional testing, not just for the functionality for the coding that we add to the product, but also on the prepackaged solutions.

We use it for all kinds of applications. Our focus is the web area, including web-UI testing of modern applications. We have two tools in place: our own internal corporate test automation tool, and Worksoft Certify. The latter is a complementary tool, especially in the web area where there are some white spots for our corporate tool which it cannot cover. That was the main reason why we brought in another tool. And for that, it fits perfectly.

We are able to run our test phases faster. Once the scripts, the test cases, are ready and automated, not only are we able to check our systems or landscapes during the test phases, but we can proactively monitor our development and test systems. Proactive monitoring of our systems is very important for us and was not possible before because manual testing is just too time intensive.

Worksoft Certify helped us to increase time savings. We didn't start test automation in general with Worksoft Certify. We did automation before with our own tool, but it helped us to increase the coverage of test automation and to increase the time savings.

We had a success story with two teams. For the execution of the scripts, we had time savings of 82, 88, 95, and 90 percent. And for the speed, it was between nine and 21 times faster than manual execution.

It's not necessarily saving us money, but it's helping us to free up the capacities of our end users to work on other stuff. Instead of doing testing for two days, they can work on bug fixing, developing new features, etc. That person still gets the same paycheck at the end of the month, so it's not saving us money, but it increases the value of our products. It increases the quality of our products. The reason for that is we are not customer-facing. We are dealing with internal teams and internal products. We are not selling anything to the outside. We are with the internal IT department. For the development teams and the sales team or the consulting team it might be different. But we are not really going out, selling our products and getting the revenue for the company. This is done by other teams.

It's script-free, which is really important for our end users because we are usually dealing with colleagues who are not developers and who do not always have the technical background of developing and scripting. It's very useful that there is a nice UI and the tool is script-free.

It covers all of the technologies we need to cover.

And one big advantage of Worksoft Certify is its integration with SAP Solution Manager, the test suite of Solution Manager, with the certification. That is crucial for us since Solution Manager is our test management tool of choice.

There are a couple of small things, technically, that could be improved.

Features we have asked for include single sign-on. It's a bigger project to make sure that our end users do not have to store passwords, usernames, and the like, for the different tools we have. 

We are also working on an additional integration with another tool that we have in place for lights-out testing. That's ongoing at the moment.

Another idea we brought is that the definitions for the objects need to be automated. They need to be recognized automatically by Worksoft Certify instead of changing them back and forth manually. This is also something that Worksoft is currently working on. 

Updates, in general, is a topic that we are working on with Worksoft on a regular basis. For new products, for new UI technologies when they come out, the test-automation providers need to update their definitions to make sure that the objects are recognized correctly.

It is pretty stable. After upgrades, we usually don't experience any big issues. Of course, it's software, so here and there we find bugs, but nothing crazy, to be honest. The availability of the system is pretty good, almost 100 percent. I don't see an issue here.

At the moment we don't have any issues with scalability. We have about 300 end users working with Worksoft. On infrastructure, it's split. We have a WTS environment, a Citrix environment, for those colleagues who want to use the prepared environment. We have other colleagues who are using the client on their own machines, on their own laptops or desktops. The only thing that we saw at the beginning which we need to change in the future is that, due to the latency, we cannot use clients in the US, for example, while having the server in Germany.

If the latency is over a certain number of milliseconds then it is basically impossible to do automation. That was one of the main reasons why we set up the Citrix environment at the very beginning.

We are still in the phase within our company, or within IT services, of training and spreading the topic of test automation, overall. So our coverage, at the moment, is not the entire organization, it's only the IT department. Once we have done this - and it will take at least another year - we will see if we spread using Worksoft and our internal corporate tool as a combination, or tool ecosystem, further into the organization. But this is not our not our team's responsibility so it's not really in focus at the moment. We are pushing for test automation in our teams and there is still a huge demand for training and new teams coming into the topic of test automation.

In general, technical support is good. They are collaborative and responsive. The only thing I don't like - and this is the only complaint I usually have for Worksoft - is that the first-level support is not always the best for working on topics. We sometimes need to escalate to second-level support and then we know that we are getting a colleague who is aware of the issue and is not just playing for time.

We already reported this to Worksoft and asked them to find another way or to educate the first-level support or to make sure that the tickets go directly to second-level support if they come from us. The guys on our end who are reporting the issues sometimes know more than the first-level support.

When it comes to second-level support, we are happy. There, we know we will get the help that we need. The colleagues are responsive and very helpful and, from a quality perspective, they are very good.

In the beginning, there were some issues with the integration, it didn't work the way we wanted. We spent some time with the Worksoft team, with the support and engineering team, in adding some enhancements to adapt the solution to our three-tier Solution Manager landscape. But that worked very well.

We have a very good collaboration and relationship to Worksoft. For example, every two weeks we have calls with them. We'll provide feedback and they take it seriously. They usually provide us with updates, with enhancements, with new functionalities that we need. That's working pretty well.

We started with manual testing and then we started the test-automation initiative. We started with our internal corporate tool and then, as I mentioned earlier, we figured out that we could not cover everything with that. At the end of 2015, we started to check the market. We did some PoCs and we decided to go forward with Worksoft Certify.

There were a number of reasons we went with Worksoft Certify. The Worksoft team did a great job. They came to our headquarters and did the PoC, showing that the tool is suitable for our needs. They did another PoC with our operations colleagues who were running the regressions testing in Singapore. And then there were the technical requirements that we had. Worksoft Certify was able to cover all of them, some of which I have mentioned already: Being script-free, being fully integrated in Solution Manager, and being able to script in a modular way. And finally, the integration between our own internal tool and Worksoft Certify was also important.

The initial setup was pretty much straightforward. I cannot give you too many details because I did not take care of the implementation. But I know that it took us about two weeks to set up the whole infrastructure. It was not really difficult. And we had very good support from Worksoft, from the support and engineering team. They helped us in setting up the database, setting up the connections. It was not a big deal. In total, within two to four weeks, everything was working fine.

On our side, we had a couple of staff members involved in the implementation because our team is the application owner. We had to involve two more colleagues from the database team because we don't have all the authorization stuff, for the databases area, for the servers etc. In total we had about three staff members involved, but not full-time. It was about one work-week for each of them.

Maintenance is done by myself and one of my colleagues, with the help of our database and server/infrastructure team. We don't have authorizations for everything and we are not database experts. There are three or four staff members taking care of maintenance, as part of our job; it's not a full-time job, obviously.

Whenever we need to do a full upgrade, when we need to plan the downtime for the production system, we try to make it on the weekend. I also already recommended to Worksoft that it would be nice to have something like an offline update where the system can be upgraded or smaller changes and fixes can be included without having full downtime. For an upgrade we usually need two to three hours. Afterward, we do a bit of testing, so upgrading takes about half a day.

It was just one or two people from Worksoft and three guys on our side.

We get feedback from all areas that the return on investment is there. Not just regarding time savings, but also cost-reduction. The return on investment in one case was reached at something between five and six runs, which is pretty fast, especially in an Agile environment.

What is also very important for us here is the avoidance of human error during the execution of tests. Usually, if someone is sitting in front of a laptop and doing testing eight hours a day, he or she will make some mistakes. This does not happen with a tool. Another important factor for us is the availability for testing. Usually, it's pretty hard to plan a test phase to bring all the testers to the table and get the time blocked off for the test phases. For the tools, we just need the systems up and running and then it's a matter of minutes to set up the test plans and to run the tests.

At the moment we are rolling out Execution Manager.

At the time we decided to bring in Worksoft Certify, we looked into two other tools. The key difference was that Worksoft was script-free. That was not the case for other tools. And the full integration to Solution Manager was one of the key differentiators between the tools.

If you have done a market evaluation and have decided to go for Worksoft, my advice is to go for it. I would definitely recommend Worksoft Certify as a test automation tool.

The feedback that I get from our stakeholders is that the tool is pretty simple to use. What we usually do is a two-week training, not full time, where the total is about three to four business days, 20 to 25 work hours. From there, most of our colleagues can start working with the tool. Of course, they have questions later on, some difficulties when it gets into special activities. But overall, the tool is easy to use. It's generally found to be intuitive.

In terms of cutting test maintenance time with respect to the scripts, that has not happened. If you need to adapt your scripts, automated test scripts are much more complex and more effort-intensive than manual test cases. But this is the nature of the beast. It will happen with every tool. If a screen changes, if a system changes, then you have to adapt your script for manual testing. For a manual script, you just adapt a Word document or an Excel sheet or the like. But if the process flow changes, you have new windows, new options, then you have to adjust your script for each and every provider that you're selecting. The maintenance of scripts is something that I always discuss with my end users and should never be underestimated.

We are not using the Capture 2.0 feature at the moment. We are planning to use it in the future. But at the moment, due to the heavy workload on our plate, we haven't had the chance to look into and to roll it out. We are familiar with the concept of Capture and it's a very nice feature because it makes the collaboration between business and IT much easier, and business can be involved in test-automation topics and activities as well.

We have three roles in our environment. We have the key players, who are the project managers, the persons responsible for test automation overall in the respective teams.
Then we have the test automation engineers who are responsible for creating test scripts and to maintain them; sometimes they run them as well. And finally, we have the executors, the ones who are running the scripts, checking the details and, if something is not working fine, going back to the test-automation engineers and asking for support and help.

I rate Worksoft Certify at nine out of ten. I'm happy with the tool, I'm happy with our colleagues at Worksoft. We have a very good relationship, we can bring up everything. There isn't much I can complain about. I'm happy at the moment with Worksoft.

We use it for SAP and ERP. I think from the SAP and ERP point of view, it is very good.

When we used it for web based application, so it was quite poor.

One, it's module based and it's giving functionality. That is, first of all we can really implement it as an ERP (Enterprise Resource Planning). We can utilize our concept and our functional people to work with the tool and without having a lot of technical knowledge, so they can really work with the tool with small technical knowledge. So that's really a big thing for us. We can use the tool with our existing people so that we don't need to have a new technical expert using the tool.

It is poor for a web based application. We are living in very integrated organization are most of the companies out there. There's the big companies we have these organizations that we work with, where we have plenty of different services which are very much connected. So, we are really looking solution which really can support all the different services. We really need to focus end to end instead of stand alone case. 

Whatever reportings are there, so they're really not very user-friendly. So there is a lot of technical data instead of user-friendly data. It would be wonderful to add web support, I would really like to see this in the future.

If I would evaluate it on a scale of one to seven, I would give it a seven. 

It's not very scalable. 

They struggle to provide an accurate solution.

I think it was not really very difficult. It was quite easy to set up the tool. The architectural tool was really easy.

We define certain different criteria. We check if the vendor is giving support for the tool, and we also evaluate the price. Of course, we also take note of the ROI of the solution.

This product is a good one, if you are really focusing as an ERP. We really need to have a solution which is really end to end rather than just focusing on key product. It's good to be aware if there something that can support you end to end rather than one specific product.

When talking about continuous testing, so then the tool is not really good at all because you need to purchase some additional tools. For example, the Execution Manager and  other tools, which are an additional cost. 

Another added cost is the payment necessity for reporting.

We are using it so far for regression testing. We have reduced 5% to 10% of regression related issues. This is a huge impact on our organization. So that's really quite good improvement in our eyes.

Speed at which it processes our testing scripts, detail of testing results for audit validation, documentation can serve as training document for new employees.

Prior to implementing Worksoft for our testing, the scenarios required eight employees over two weeks to complete (waiting for hand off of next steps, employee availability due to meetings, etc.

After implementation of Worksoft, the same test scenarios currently run in approximately 45 minutes – hands free! We have also implemented Worksoft into our SOX Control documentation processes.

Approximately seven years.

No.

No.

Excellent. They always went above and beyond to timely support our needs

Setup of the software was extremely easy. Setting up the scripts was complicated at times (due to our customization of our SAP clients).

However, most routine transactions were handled with ease.

I am not familiar with the pricing or licensing fees.

No.

Make sure to be prepared for the amount of time it will take you to complete the testing scripts.

Have a dedicated team that performs your test script setup and execution of these tests.

Include your business users to ensure the processes are clear and accurately reflect their business processes, not what you think their processes are.

Well, obviously it's the test automation piece of it. Creating our scripts and being able to execute our SAP scripts in order for us to be able to do comprehensive regression testing of our SAP system.

Well, it's definitely reduced the number of defects that we have. Before, when we were manually testing everything, it was unclear to us where changes that we were making, where the impacts were in our system. I mean, we've got very smart people that work here, but they were not 100% on where the impacts were going to occur with their changes.

By automating our scripts, we were able to do more comprehensive testing and therefore eliminate the number of defects that were be introduced into our environment.

There is an impact assessment piece to Worksoft. I think that's one of the pieces that we still struggle to use.

What the tool does is, you can analyze a transport to know where your impacts are. I think that's one of the challenges that we still have -- is just getting a better handle on how to use the Impact Assessment better for us.

In our regression testing, all that, we've got a pretty good handle on that. We identified things. When we run the analyze of a transport, we're not...we've found other ways to do things better.

Four years now.

No. It's been very, very reliable. The only issue, we've virtualized everything. I think the only thing, the only real issue we encountered was with our virtual machines. It was not a Worksoft issue.

It was on our side -- once we worked with our sys-admins to investigate what was going on, they were able to resurrect the issue and we were fine. It was not a Worksoft issue at all. It was something with our virtual environment.

It's been very scalable for us;

Depending on how many licenses, it's all seat licensed. It's not a concurrent license. If you don't have people logged in, more people can't use your licenses from your pool.

You assign those licenses out to people and as you continue to grow, like I was talking about with our virtual machines, we do everything in a virtual environment for all of our Worksoft work.

As we've needed more virtual machines, we've got an image of our Worksoft clients and we just add them into our environment.

3 out of 10 -- You submit an email, so there's not really a hotline that you can call to talk to somebody on the other end.

I had familiarity with several tools out there. Turnkey is probably one that you guys have looked at. I've done a lot with the quality center products in the past...Quicktest Pro.

There's other test automation tools out there. I've got a background with test automation and Worksoft...really it was between Worksoft and Turnkey. Worksoft, we just found ,was going to better meet our needs.

Yeah. That was actually pretty straightforward. When we set up our environment, they gave us a technical support person that really walked through with our system administration team to get everything set up in our environment.

As far as training, they have a very comprehensive training program. I'm sure it's all ... I'm sure it's very good. What we did and what helped us to be very successful is we went through and did live test cases as part of our training.

Instead of spending all the time going through the books and learning the tool in and out, we went through and we got right into creating our test automation with the trainer that they sent out. That hands on experience was, it was priceless. It helped us to be successful and get going with creating all of our test automation.

Make sure...before you start any of your test automation, make sure you have the appropriate support from your business.

What I mean by that is, you have all of your business processes documented. There is a lot of prep work that goes into creating your test automation to make sure you're creating...you're creating your scripts in the most efficient manner. A lot of this testing is done based on your business process. You need to understand your business processes.