Worksoft Certify Reviews

We have automated quite a number of test cases in last year.

We have a weekly release. A weekly release means every week that we have testing going on. The particular year where we started is called the asset management area. We could never run our Intuit testing, and we have a whole lot of Intuit testing. This one product helped us pass the testing with the Nighthawk testing, which is working on the Nighthawk manager. That's the one that we use. We can switch it on in the night and run the testing, then come back in the morning and see what has been completed. If there is any fail, we can even analyze it. We can use the evidence document to pass it on to development team to tell them, "This is where we failed."

It has improved on our defect management time. It has improved our test execution time. I don't need to manage these things, just sit somebody down to look at how the script runs. There are a lot of ways that it has helped us.

It is very easy to maintain. With scripts, I can change one line and in one step. Whatever I want, I can do. I don't need to be an expert to use it. Being a manager without a whole lot of technical knowledge, like an automation person, I can change Worksoft using what I learned during the training. That's what I like about it. Anybody can do automation.

I love the Capture 2.0 feature. When you are doing a normal manual testing, go ahead and switch on the Capture 2.0 feature, then capture everything and pass it on to your teams who can convert them quickly into test automation. With this feature, it is saving our automation creation time by about 60 to 70 percent. It is also helping our manual testing time in terms of catching all the evidence documents. 20 to 25 percent time is being saved because of this product and Capture 2.0 feature and what we are receiving with the good documentation.

I would expect more opportunities to automate Java.

I would like it to analyze what we are not using. 

I would like to see the impact analysis integrated with the performance testing tool. We have multiple tools doing multiple items. I would like to have one common tool. 

I'm also interested in load testing automation and whether we can create a script for it, then can we use the same script for my performance testing? 

Stability is quite good when compared to other things. I don't want to say it gets 100 percent rate in terms of stability, because I'm using this for about ten to ten and a half months right now. It has been close to a year, but I'm really amazed when comparing it to any other tool that I have used in the past.

I don't have the words to express its scalability besides awesome. The amount of changes that we have seen are tremendous.

We don't use them frequently, just when we have an issue. I would rate the technical support a seven out of ten. Most of the team is good and helpful. However, I would like them to evaluate the issues a little more sometimes before escalating them to engineering.

We were previously using Micro Focus UFT. The tool was good, and we did not have a lot of problems with it. The only problem was SAP changes a lot of things every time. The frequent changes were causing a lot of issues for us in terms of automation. We were able to automate many things, but the maintenance was a big problem for us. 

1. You needed to have a person who had the coding knowledge to do it.
2. The frequent changes made the scripts useless. Then, we would have to come back and redo a lot of things. 

This is where we were looking for a product where we could have minimum maintenance that anybody can automate. This is the concept why we came into Worksoft.

We used Worksoft team initially. That helped us through the setup and other things. They did a great job. We probably automated about 275 tests in less than five weeks, including the setup. 

I suggest people go head and use Worksoft, along with their services, when you buy the tool. They will help you to onboard it quickly and set it up for you. They will do lot of the automations. They will help you with lot of these practices, then you can take it over from there.

We have seen ROI.

By using automation, it reduced about 75 percent of the time when compared to any other tool. The changes for Worksoft used the same script as the script automating UFT. Manually, running our tool takes about 4 hours, but with Worksoft, we were able to do it in less than 30 minutes. Whereas, the same thing that you had to do would take you almost 55 minutes to an hour. There is now a 50 percent savings in terms of other automation tools and an 85 percent savings in terms of manual to automation.

We have seen more than 40 to 50 percent reduction, in terms of all around time, where we were doing five days a week for a major maintenance testing of our first cycle. We have reduced it down to three days now. So, that is a 40 percent savings which we are seeing. We are not completely automated because we are still doing sampling. We have just automated a good 40 percent of our things. When we go to 80 to 100 percent, it will reduce 60 to 80 percent of our time, which is what we are looking for.

We looked at Panaya, but Panaya is not a desk automation tool. We are using Panaya for some of our impact analysis.

We did try with Selenium and many other vendors. A lot of other tools have a small test director that we tried. The ease in using this tool is very good because our business team can go in and use it. So, anybody can record for me and capture. Then, we have a very small team of automation testers who can convert the information immediately into a reusable component, parameterize it, and do the records sets. In that way, with a very small set of test automated guys, we can do much more.

I recommend this solution already to my colleagues worldwide.

We run this on seven different multiple applications. It starts from SAP, goes to the UI, comes back to SAP for violations, and then goes to mainframe for validation. Then, we use Java Web as a Java. After that, there is another HDM which we try to validate. Also, we are trying to validate a third-party application using it, because we have used a lot of their components trying to do a mock type of filing import/export option with the tool. 

We have used this solution for web UI testing, as we are on SAP Web UI 5.0 right now. We use this very heavily right now in our asset management area. It is very easy to use. The Capture 2.0 together with it is helping us, because we are now able to recognize some objects through Capture 2.0. We also have LiveTouch. This is another advantage where you can use this to capture multiple items at a time.

We use Worksoft Certify to test our SAP System. We have a global instance of SAP, which we started implementing in 2012, and we are still in the process of implementing. We have rolled out SAP to about 80 percent of our manufacturing and distribution. Right now, the remaining projects are a small distribution center and sales offices. We have ongoing projects, and three times a year, we release a new version of SAP. We rolled out SAP to a new geography, and we also added new features for our business users. Thus, as part of those projects, we use Certify to do regression testing of our existing business processes, and we also use it in the project to test new functionality.

When we are rolling out in a new country, we do a configuration for that new country. We use the automation test to test the business processes and prices of that new country. It is sort of semi-automated. Our business analysts generate sales orders from the new country, and we will run them through the shipping orders to cash, the shipping steps, and the concrete steps. Then, we get a set of documents to review. The business analysts review those documents to make sure the order is processed correctly. So, it's not fully automated, but it does help cut down on testing a lot when we roll out to new countries. 

For regression testing, that is fully automated. We have tests where the software checks the results and either returns a pass or fail. These are run as a regression suite anytime we push a change to production.

We do use it for the end-to-end testing of packaged applications, primarily SAP. We do have some plugin applications that we use it to test, which are part of the business process. We use Salesforce for CRM, and we have a custom built eBusiness application. While we don't do extensive testing of those applications using Certify, when the business processes touch one of those applications, we do cover those application with another certified test.

It has cut down on the amount of low level, grunt work that business analysts have to do and can free them up to do more critical thinking. Before we had test automation, we were running tests and relied on people, which was very time consuming. A business process test might have 100 to 150 steps across different applications, and we don't have a single person who has expertise in all those applications. When executing a manual test, we have to balance the test between different people to do their steps. In a typical project, we might have 100 to 150 of these types of tests running. The coordination of the testing process where you have to have different people available at different times is very time consuming and inefficient. What automation has done is cut that cycle dramatically because automation does not have to worry about having to find the right order management or warehouse person to do their steps. The automation just runs through, then the business analyst can review the results afterwards. Therefore, it has been more efficient, cutting our testing part down by almost two-thirds to 75 percent.

Our automation tests are more robust than our manual tests. We found our test lab would grow over time because we didn't have a lot of discipline within the team for manual testing to have a master test which could be used repeatedly and revise as necessary. So, they were creating a new test for every specific little thing that they wanted to test. They were  setting up these manual tests where they had ten to 15 tests which tested the same thing, but not quite. Therefore, it became a bear to manage. Whereas, with automation, because it is more controlled, we have a core set of about 125 automation tests entered into our library. That's in our change control. Therefore, we know exactly what the state of our tests are. 

If there is a new business process or new wrinkle in a business process, we didn't have a defined process, so now we are updating automation tests. The quality of the data that we're getting out of test from automation is much higher than we received out of manual testing. If we know the automation suite is parsing, then the application is working properly. With automation, we have more confidence that if the test is parsing than the application under test is working correctly.

It is fairly straightforward. We have some deep expertise after using it for five years. We have some people who know it now very well. 

This type of marginalization of the code inside Worksoft Certify has been very valuable to us. The ability to capture documentation. We are a technology company and are regulated, so we have pretty stringent requirements. We use Certify to capture screenshots and evidence during testing. We can capture every screenshot during the business process including a document and hand it off to the auditors. It makes defending an audit very simple. We can, if they ask for evidence, produce a document that shows the business process of every step and the screenshots showing all the pertinent data, which has been pretty useful as well. This is the report feature with Certify. When you run a test, you can either have it generate documentation or run it in the background. Most of the time when we were running regression tests, we just run them in the background. 

We don't generate documentation, but we could turn Capture 2.0 on, where it creates a screenshot. As we're starting the test, it creates a screenshot of the application it is testing every step of the way and produces a word document or a pdf at the end that you can then hand off to auditors and show them the actual flow of the process that you're testing. However, we do not use this feature.

One feature that we have been asking for has been to treat tests as code and store the source code for tests in a configuration management tool. Right now, for version control of testing, it's all internally within the tool. If we have a test of a business process and want to revive that test, our methodology now is purely manual work. We go into the tool, create a copy of the existing test, and call the next one: v2. Now, we have two of them and the only way you can tell them apart is by its naming convention.

This is not an efficient way in terms of how modern applications do version control. If this was code, we could plug it into a tool like Git or GitHub to manage of our versioning and branching. The reason why we want to do this is that the application which we are testing branches. When we branch the code, we put a bunch of new functionality on the new version while our production version stays unchanged. Then, at the end, we merge the two together.

From an automation testing perspective, we have to run tests on both. Then, we have two current versions of our test. So, it's a bit hard to manage in the tool right now because you can only have this manual approach where we are tracking it via the name convention. Whereas, a modern way of doing it would be to have our application plug it into a version management tool, like GitHub, where we would store the code and could just pull in the version of the test that was applicable to the version of the software that we were testing.

This is something we have been asking for for a while now. I understand that it's in the pipeline, and it may be in their latest version (version 11). This is something that we will be looking into this quarter.

The challenge that we face everyday for test automation are more internal (people issues). We need change management and getting people to accept automation instead of the technical limitations of the tool. The tool does what we need it to do from an SAP testing perspective.

We have no stability issues. 

Maintenance-wise, we have one system administrator who is not full-time, since it has been pretty stable. We don't change much compared to other applications. This application is pretty hands-off.

We should be upgrading to the latest version in next couple of months.

At our peak usage, we have had seven people working on it, and have had no issues with that. Now, with our current work load, we only have three people. We only run our test suite. It was one of my goals on this project that we had the infrastructure setup, so we could always run our entire test suite overnight. As we built out our library, this meant expanding our infrastructure. Right now, we have 100 to 150 integration tests, and some of them can take ten to 20 minutes to run. A single instance of Certify can only run one at a time. Thus, we have had to think about how we set up our infrastructure in such a way that we can run the entire suite of 150 tests in six hours.

The way that we have done this is to split it up amongst servers. Therefore, we still have extra servers for execution. We have four servers now and run the tests in batches of about six queued up at a time. In this way, we can run our suite of 150 in parallel across four machines and get it done in about six hours. Right now, we do this manually. We do the manual breaking up and monitoring. I know Worksoft has some tools which automate this. This is something that is on our radar to look at as we grow. However, right now, we just manually manage the process.

We have three test developers using it. These are the people actually building tests. In terms of consumers of the test automation, we have probably 35 to 40 business analyst.

The technical support is pretty responsive. We haven't had many issues with it. When we were doing an investigation into doing web testing, we ran into some roadblocks. The team at Worksoft was very responsive. At the end of the day, it came back to technical limitations of a tool. I have been pretty impressed with how responsive the team. They were always able to answer our questions to the extent that the tool was able to do what we needed to do.

It was all manual. For convenience, we used Micro Focus ALM for tracking our manual tests. We still use that as our central hub for our test documentation. We weren't using any test automation tools in IT. Within the organization, we have R&D groups that develop software for various systems and medical devices. Those teams are running tests and code. They are in automation test suites, and I was part of one of those teams before joining IT. However, in IT, before we started using Certify, we weren't using any test automation.

Manual testing was costing us a huge amount of money. We did a double rollout of SAP. We split it over three deployments:

• With deployment one, it was just one division in North America. We had over a 120 people doing manual testing for a period of about sixteen weeks. Add up the cost of that. 
• As we moved into deployment two, we were going to have to test new functionality and also regression test what we'd already booked. If you took the amount of testing that we'd done in deployment one, even if we weren't going to redo all of that, we're going to have to do 50 percent of that. It was going to be a huge manual effort and a sunk cost. We'd put all that money into manual testing and wouldn't have an asset. It would be money that we are basically suspending with no reuseability.

It was a pretty easy decision to convince the team to move to automation because it would be an asset that we could reuse again. Over the last five years, we've shown that we've had a positive ROI on it. The initial upfront cost in terms of licenses, plus all the money that we spent developing tests, has proven it's worth. Now, we can do a regression test suite in ten days as opposed to sixteen weeks.

The setup was very straightforward. We did a proof of concept with Worksoft. They came in and had an engineer onsite. We set them up on a server and pointed them at our test SAP system. They built a couple of prototype tests for us. When it came to implementation, we had an existing prototype that we looked back on. I have a systems administrator on my team, and he was able to pick it up pretty quickly. 

The documentation was good. We did the install on our production system, copying over our prototype tests. We used that as our starting point for building out our library. We also sent out a couple of guys for training.

We were up and running with a functional system within a couple of weeks. The challenge, at that point, came down to training our business analysts on how to use the tool. This took longer than getting the system up and running, which was pretty straightforward.

We did the deployment ourselves. It took less than a week. Internally, we had one system administrator do the bulk of the work.

We ran the deployment on Windows Server. We have two machines: a database server and an application server. Our test developers can logon via Windows Remote Desktop to access those machines. They built all their tests out on that system. Architecture-wise, it is hosted all behind our firewall, but it is all server-based. No one is building tests on their local desktops. It's all server-based, and we can share some of our scripts amongst our team members.

My primary team is offshore. They are in India and Bangalore. Therefore, all of the test development is done there. However, we can access the central test library seamlessly, and  the test strategy for setting up and standing up servers and installing the software was pretty straightforward.

Our ROI is primarily a reduction in testing time. The testing, when we were doing it manually, was 30 to 40 percent of the project's cost. This was a $450 million USD deployment of SAP, and testing is 30 to 40 percent of that cost. We spent probably about a million and a half in test automation, but managed to reduce our testing times from weeks to days. There is a clear cut return.

If we write a new test that's 80 percent the same as an existing test, it is pretty straightforward to reuse the steps from existing tests for our new tests and build upon them. We found that there has been increasing ROI automation as we built up our library. When we write new tests now very seldom is a new test build from scratch. It is normally a variation of something that we already have, so we can turn those around pretty quickly within a couple of days to two weeks.

We ended up buying too many licenses. They were very good at selling it to us, and probably oversold it a little. We bought 45 licenses and have never used more than twenty. However, they gave us a pretty significant discount on the bigger license, so it made sense for us to buy enough that we wouldn't have to go back and ask for more. 

At that time, we had budget to do that. The licensing is pretty straightforward. We have considered using them to do robotic process automation and may still do that. Initially, we were worried that our license might preclude us from using the tool for something other than testing, but when we checked into that, there is no limitation. 

We could use Certify to do robotic process automation, which is basically running a process on your correction system instead of your test system. Therefore, we may do that in the future. 

We also looked at HPE UFT (the HPE automated testing tools) and SAP TAO (SAP's own internal test automation). The reason we pretty quickly went with Worksoft was primarily the responsiveness of the team. The evaluation happened between deployment one and deployment two.

When implementing SAP, we had IBM as our system integrator. We went to both SAP and HPE asked them to show us what they could do for test automation. We also looked around and found Certify as a third candidate. The response from the Worksoft team was far higher than the other two. IBM wasn't able to produce sufficient expertise to demonstrate the SAP test automation tool and same with HPE. I also didn't have a good response from them. We felt, " If this is the level of support that we were getting during the sales cycle, how will it be after the sale has occurred and we have to go to them for support?" Whereas, Worksoft was very responsive. They sent people onsite. They did a proof of concept using our system and data. There was a pretty clear cut night and day difference in teams and companies involved. I didn't get a chance to evaluate the technology of the SAP or HPE solution because their sales teams weren't responsive.

We have a dedicated team of what we call test developers who are specialists in this application. While I don't use the application myself, but they're pretty productive with it. We have a team using Certify for SAP Test Automation and a team using Selenium for web application development. The SAP test development is more efficient than the web test development. For a similar sized test development project where they have to test and develop five automated tests of a certain method, we can turn them around in SAP faster than we can turn them around in Selenium.

Now, it might be Selenium has a higher learning curve than Certify. Or, it is easier for test developers to get good at developing test units using Certify. Selenium is far more technical. Of the two tools that we use, Worksoft is more user-friendly than Selenium.

The technical instrumentation was pretty straightforward. The tool does what we need it to do. The primary challenges that we have had with test automation have been change management, getting the old, greater IT organization to accept automation as a substitute for manual testing. Culturally, within our organization, we put a lot of pressure on our business analysts to thoroughly test the application, and if they have never used automation before, there is a fear factor there saying, "I'm responsible. Then, I want to see it with my own two eyes." 

I recommend expanding, training, and coaching people that automation is just as good, if not better, than manual testing in terms of finding bugs and proving that the system is working correctly. It is far faster, and you will get a lot of your life back. That has been the biggest challenge for us: Telling that story and expanding the use of automation throughout our organization. Now, automation is pretty mainstream and accepted, but that was the biggest challenge for us. It certainly wasn't technical challenges.

We don't use Capture 2.0. We found it easier because we have a large pool of business analysts who are not certified users. Our process for capturing the business process which needs to be automated, therefore we use Zoom Recorders. It is like a WebEx tool. It has a screen sharing device and a record feature with audio. We find the audio is quite beneficial. When we capture the business process, we will have people record in Zoom, annotating with their voice (doing a voice over of what they're doing). Then, we handed it off to the test engineers to build up the automation. We look at Capture some time ago and felt it wasn't as efficient. Capture 2.0 is the newest version, and we haven't really looked at it in-depth. We will certainly reconsider it, but right now, we are not using Capture 2.0 to do business processing.

We use web UI testing to a smaller extent as part of the SAP business process. For a business process which incorporate Salesforce, a field service engineer might order a spare part. This is a post process that spans both Salesforce and SAP. For the first half of the processes, we use Certify. We did attempt to use an in-depth testing of web applications sometime ago. At that point, we felt there were some technical limitations. The project was to use Certify to do comprehensive testing of our Salesforce application. However, we found when we did a deep dive that there were some aspects of Salesforce and proprietary screens which Certify already struggle with. At that point, we decided to switch to Selenium which is the industry standard for web testing. Now, we do most of our tests on Salesforce in Selenium. While Certify has become a lot more capable with web testing since then and the newer versions are better at it, at the time we investigated it, we felt that Certify probably wasn't up to scratch as a web testing application.

Going forward, we will look at Certify again as a web testing application tool since it is more efficient than Selenium. We are finding that it's costing us more to develop a test for a web application than it does to develop a test for a SAP based application. We want to take a look at them again as a solution because it might help increase our efficiency as most our applications from this point forward will probably be web applications. So, there's a lot of work to do in that arena.

With our eBusiness and Salesforce suite, we are not even close to full test automation coverage. We still have a lot of work to do. So, it's worth us looking at Certify again. We're expanding into big data and big data analytics. There are a whole slew of terms around that with regard to testing. E.g., how do you verify that your data's accurate? We are just dipping our toes into it, as we haven't done any model testing yet. That is something that we have to look into. There are a lot of areas where we could use it.

In the last couple of years, we have become an established and accepted part of the SAP testing in the organization. We are a fairly conservative group. Now that we've done the SAP testing, we need to start looking at different horizons of mobile, big data, and web testing where we still have a lot of work to do in terms of building up our automation.

Worksoft Certify is being used to run automated weekly regression tests across some of our primary SAP systems in line with our Change and Release management strategy. These tests run every weekend without fail. The results are reviewed on every Monday morning to check for failures and to analyse if any failures are associated with the changes scheduled to be transported to the production environment that week. Failures (if any) are fixed and the tests re-run before transporting the associated changes into the Production environment.

We also utilize it for projects that need extensive business-user testing and functional testing. There can be testing requirements which come at short notice which can take three to four weeks of manual testing effort. By using Certify, we have been able to bring timescales down to a few hours of automated testing effort.

Our final goal is to utilise this for 'Unify', our new global solution which is currently being deployed, which will deliver common processes and systems to all sites and sectors, replacing all our existing legacy systems which will demand extensive regression testing. 

We have never had any systematic regression testing regime in the organisation. This has helped in building an automation framework across our SAP application landscape, thereby introducing mandatory regression testing across all our key systems and improving the overall quality across our production systems.

From an audit perspective, results generated from Certify (BPP reports) provides detailed test evidence which is also being utilized for internal training purposes/training guides, etc. The BPP reports also provide details on failures along with screenshots.

We have a variety of complex systems in our landscape, one of them being the Openlink Endur which is a commodity trading and risk management system. We are currently building an automated regression test suite to support application testing for Endur.

Our weekend regression tests are performed in 'lights-out; mode. Tests are scheduled to run at a certain time over the weekend using the Execution manager functionality. Usage of Certify has also prevented some major defects going into Production and we have seen significant savings in all manual testing activities as the business users/functional teams are getting more time to perform 'value- adding' activities.

Post our recent upgrade to Solution Manager 7.2, we are currently in the process of implementing the Test suite functionality and the integration of the same with Certify. We expect Solution manager to be the single source of truth bringing out all the results from Certify which is going to be extremely beneficial from an audit perspective. We have already implemented the integration of Certify with HP ALM in our landscape.

Moreover, we have this reusable asset now which can be run frequently to support all our projects and change requests across our legacy SAP systems. Even last-minute testing requests are being accommodated by utilising the automated regression suite without any dependency on business users/functional users for their efforts. We use it across the multiple projects which need immediate assistance and for our weekly regression cycles. To give an example of a recent project which was a major platform migration from a Data Centre in Asia to Europe which needed extensive Disaster recovery testing and Functional testing/User acceptance testing. The initial testing estimate was approximately five to six weeks, however with the use of Certify we could do extensive testing in less than three hours saving many weeks of manual testing effort.

It is very user-friendly with an appealing UI, unlike a lot of other automation tools that we have evaluated. With sufficient training and adoption of best practices, the tool will certainly help organisations to successfully implement an automated testing framework and eliminate manual testing activities.

The fact that it can be used across SAP and non-SAP applications (including web-based apps like Web Dynpro) is a big advantage for us because we have a variety of SAP and non-SAP applications across the Johnson Matthey IT landscape. Being a 200-year-old organisation, our variety of legacy systems have a lot to benefit from the use of automated testing.

Certify has many interesting features, e.g.: 'PRIMO' which is the image recognition functionality is a life saver in instances where Certify standard functionality cannot identify and learn objects within certain legacy applications.

Regarding end-to-end testing of packaged applications, Certify is primarily used across our SAP application landscape and the Openlink Endur (commodity trading and risk management system). We hope to realise more benefits by implementing Certify across our wider application landscape over the next few months.

We have been using the Capture feature, although not the latest version, the initial version, for process captures was used to create our test designs. It has been a life saver in many instances, without the need to spend any extra effort to create test designs and captures. The test steps get captured in the background which generates an XML file which can be easily imported into Certify, creating the basic test structure which can be improvised/modified to make it a repeatable reusable test. In terms of the amount of time it takes users to create documentation automation using this feature, it is the same amount one would spend to do a manual test. While a person is performing a manual test, Captures are automatically generated in the background. We have used it extensively to build our test designs.

We have requested for some minor new features which Worksoft is considering.

The PRIMO image recognition functionality has room for improvement, especially around its ability to work with java interfaces, Execution manager scheduling, etc. as we have observed. As we explore more of our legacy systems, I am certain there will be a need to use more of the PRIMO features to learn the objects.

Overall from a SAP perspective, it works almost seamlessly.

One to three years.

The version of Certify that we are using has been mostly stable and we have rarely encountered any problems. Our weekend regression test failures are often associated with environmental/system performance issues and not related to the stability of Certify. I have been happy with the overall performance of Certify and how it has helped to optimise our tests.

I am confident that Certify can scale to fit our automation testing needs as we expand the current automation testing framework across the wider Johnson Matthey application landscape. We are also exploring options to identify potential areas where Certify can help support mass data uploads, etc. to benefit other teams in their day to day operations.

We have several concurrent users accessing Certify in our environment, primarily automation engineers, test engineers and tech managers.

Our interactions with technical support has not been the best always and there is room for improvement especially with respect to the time taken to respond to cases. However, with the right contacts and reasonable escalations we have always managed to get quick attention on our issues.

We have always been heavily reliant on manual testing and as a result, regression testing was not systematic and we could never think of implementing frequent weekly regression test cycles which was challenging. We decided to go ahead with automated testing and use Certify because:

1. Manual regression testing takes a lot of time and resourcing is always a challenge.
2. Regression testing not being systematic, the quality was very difficult to measure as we did not have a standard set of manual regression test scripts/sufficient documentation.
3. There was a delay in our time-to-market because all the testing was being done manually and there was no way we could accommodate frequent, weekly, regression test cycles. That meant high business risk, that we would have more defects in the production environment/ more associated costs.

We had all these challenges and we started exploring options to mitigate these risks and automation was identified as the way forward, nearly two years ago. We evaluated various automation tools in the market. It was critical that we had to identify a strategic tool which would cater to our SAP and non-SAP application landscape. Worksoft Certify came in as a big winner ticking most of our requirements.

We went through a lot of initial challenges, mostly around internal resourcing issues. Looking back, I am happy to say that we could overcome these challenges and have managed to successfully implement an automation framework using Certify.

Early in 2017, we decided to go ahead with Worksoft Certify post evaluation of multiple automation tools. Our initial engagements with Worksoft consisted of several onsite workshops to explore the tool in detail along with technical feasibility assessments across our application landscape. These engagements were extremely beneficial and it gave us the overall confidence to adopt Worksoft Certify as our strategic test automation tool.  

We did a pilot implementation with Worksoft to see if we could take this ahead on a large scale before embarking on the major project to build the automated tests. Some key processes across our critical SAP systems were identified as candidates for this exercise. Test designs were created with support from the functional teams and taken ahead for automation build with Senior Worksoft consultants and our internal resources. This 7-week Automation Roadmap Engagement exercise was extremely successful and we learned a lot of lessons from it which helped us plan the next big phase of the automation roll out. It gave us overall confidence across the functional and management teams which subsequently led to securing the appropriate budget, etc.

One of the biggest lessons learned from this engagement was around the ways to structure our teams. This led to us going ahead with a Managed Services model with Worksoft. We have an offshore based Worksoft Automation Services Factory team who helps build our automated tests. The team can scale up/down based on our automation forecasts.

The automation deployment is still ongoing. The initial phase was completed across a five-month span. Currently we are rolling out the second phase of the automation build focusing primarily on our global Unify solution and the Openlink Endur application.

Regarding implementation strategy, we followed an agile two-week sprint approach. Our functional teams continuously created test designs and these were fed to the Automation Factory every two weeks, who in turn developed the automated tests. This was the most practical model, which worked well in our environment.

At its maximum capacity, we have had approx. 10 to 12 automation engineers in the Factory team. Our functional teams are spread across multiple global locations and we had between 3 to 6 resources working on test designs liaising with the business users as required.

From a script maintenance perspective, we spend an average of 4 to 5 hours every week with the current asset of nearly 800+ tests.

We have always worked directly with Worksoft, along with support from our internal resources. Worksoft has been delivering excellent services through their managed services model.

We have a res-usable re-runnable asset built which is saving a lot of time across the functional teams/business user community.

Our final goal is to utilise this for 'Unify', our new global solution which is currently being deployed, which will deliver common processes and systems to all sites and sectors, replacing all our existing legacy systems, which will demand extensive regression testing. 

It is expensive compared to some of the other automation tools in the market. However, the benefits and ROI has proved that it has been a good investment.

We have concurrent licenses.

SAP TAO and Micro Focus UFT.

It was critical to identify a strategic tool which would cater to the testing requirements across our SAP and non-SAP application (including web based apps like Web Dynpro) landscape. Worksoft Certify came in as a big winner ticking most of our requirements.

It is a great product and we have not seen anything which cannot be automated till date in our application landscape.

It is important to do sufficient technical feasibility assessments before deciding to go ahead with Certify and equally important to determine the best implementation approach which will work for your organisation. Functional teams/business users' buy in is critical as the test designs cannot be created without their continued support. Adoption of best practices around naming conventions/folder structures etc. will help in easy overall maintenance of the test assets, which will also help with the generation of development and execution dashboards/overall reporting.

I would rate Certify at eight out of ten. Worksoft has always been very supportive and responsive to our needs and this has certainly helped us achieve our initial milestones successfully. I am extremely proud of what has been achieved so far and looking forward to expanding the automation framework across our wider IT application landscape over the months ahead.

We have developed some end-to-end regression testing scenarios that we've found pretty valuable, so we have created a bunch of processes in Certify, linked them together, and we use them every week - sometimes more than once a week - in regression testing.

We used another tool for many years. It became unworkable because of the length of our scripts and how many of them there were, and how they were linked together. They became cumbersome in the other tool. It's much easier in Certify, and Certify can handle them, no problem.

In addition, our organization is implementing Agile, we're moving towards continuous development, and I don't see how we could do that, in any imaginable way, without Certify. We're able to import our changes weekly, based on the results in Certify. And we're confident that because of having tested the main business processes, fairly rapidly, within one day, we can tell whether the imports are going to break anything.

It has absolutely enabled us to automate and save time. The weekly imports of the changes allow the developers to plan on a weekly cycle, which increases the speed of their development. They don't have to wait for a release or anything else, they can test their changes quickly and get the results the next day. They know that they're able to import with no problem.

Finally, it has helped us cut test maintenance time. 

The idea that it's not language-specific is really nice. Keywords and the drag-and-drop functionality are great. The ease of use is superior to anything on the market. It's very easy to integrate. We've been very impressed with the tool. Because we primarily use the configuration with SAP, the integration is pretty seamless. But we have used our own in-house VB app as well, and it's worked very well with that. We've really not seen any problems whatsoever with integrating.

We love the Capture 2.0 feature. It seems to work very well. As for how long it takes to create documentation using it, we do not get into the documentation so much. That end is not as useful to us. But it's built-in if we ever needed it. We're not USDA or anything like that so we don't have a super need for documentation right now.

Also, the ability to work with the data, with recordsets, and plug those into the scripts is very easy and very powerful. We use it extensively.

In the past, when we've tried to automate some of our web apps, it has not been as robust. If there were one thing that could be improved, it's interaction with web applications. The issue we were running into is that it was harder to identify the objects than it is with some of the other architectured applications. That's the nature of the beast with the web as well.

It's very stable.

We have had some issues. We would be unable to log in, in certain situations. But they've all been self-inflicted, changes that we've made on our side that have prevented us from being able to use the tool at times. Once we got those resolved, we were fine.

It's fairly easy to scale, which is a nice thing. Once you create what I will call a sub-process, if you want to use that sub-process in many other processes, it's really easy to use. For us, that's what makes it scalable. You can use that same process wherever you need it. The use of the recordsets just allows us to be able to change the data that make it unique and that make it easily maintained. It's very easy to scale. It simplifies our workflow.

Worksoft technical support is very helpful, very knowledgeable. Whenever we've had an issue, they've responded very quickly. We don't actually have very many tickets, but whenever we've had them in the past, I've just gone into the portal and I get an email back, usually the very next day. I've never had to escalate an issue.

We were using Micro Focus UFT. It became completely unworkable for us. Our end-to-end processes were just too cumbersome for the tool to handle. It got worse and worse to the point where we had to say, "You know what? We have to change tools, this is not helping us." That's when we investigated Worksoft, and we were very pleased with how it worked.

The initial setup was pretty straightforward.

The last time we did it, for the upgrade into version 10, it just required me and one other person on the database side, and then the technical person from Worksoft. It was fairly easy. It took just a couple of hours. At that time, we were just upgrading. The basic architecture was already there so it didn't really require a project plan or anything like that. Once we got it set up, it was just a matter of migrating what we already had in UFT.

There have been several times where it has highlighted an important issue. Some of the defects we've found have been high-impact defects that would've really been costly had they made it to production. There are other times where, because we were able to test with Certify, we knew within a day whether there were gaps in the way we configured a change, things that we had missed that we wouldn't have been able to find if we didn't have the ability to test quickly.

That one defect we found easily saved us $1,000,000. That was just one. Over the years, the amount of money that it has saved us is certainly in that range.

The ability to test quickly has enabled us to develop quickly. We've been able to capture lots of savings in terms of projects that have been delivered faster because we can test faster.

There are savings on a lot of fronts because of this solution.

We would purchase more licenses right now if they were cheaper. Pricing is a little bit of a hindrance.

We didn't look into any alternatives.

It's a highly powerful tool. It's very customizable. It's not a cure-all for everything, but if you want to do end-to-end testing, regression testing, it's a great investment.

We use Certify for end-to-end testing of packaged applications. We have implemented almost anything that touches SAP, using Certify. When C4C came out, the customer application, we regression tested our existing suite to make sure that nothing would break. We anticipate doing the same thing with Success Factor. At the moment, we don't use Certify for web-UI testing, but we're planning on implementing some of that, coming up.

Since it has been up and running, we've had three people maintain it: Myself, I'm the principal QA person, and we have two offshore partners whom I've trained on Certify and they are now helping us execute and maintain the tests. It requires full-time maintenance. We have plans to expand the reach of our automated testing, so we plan on adding more people. We are the only three using Certify in our organization at the moment.

It tests our core business processes but we still have many core business processes that we would like to add to that, to validate if they work, before we send changes through every week. And we would also like to increase the speed at which we can add changes; not just once a week, but eventually daily. We plan on increasing our resources from a manpower standpoint and also from a technological standpoint. We're just going to try to do that as fast as we can. There are a lot of business processes that we would like to add, a lot of apps that we would like to add. The business side has continual, increased demand in terms of things that they are working on and they would like to automate and not test manually, so there's a lot of demand on us right now.

I would rate Certify at nine out of ten. I rely on it every day. It's a great tool, and any problems that we have are hardly ever attributable to the tool itself. It's always some other factor; the way we're using it, or some external factor, which is the problem. It's nice not to have to worry about the tool being the issue. We're very enthusiastic users.

We are using Worksoft Certify to enable our stakeholders to do test automation on the UI level.

We use it for end-to-end testing of packaged applications. We are part of the internal IT department within our company. Most of the time we are using it on our own products. The products and systems that we get are usually preconfigured and prepackaged and we do additional testing, not just for the functionality for the coding that we add to the product, but also on the prepackaged solutions.

We use it for all kinds of applications. Our focus is the web area, including web-UI testing of modern applications. We have two tools in place: our own internal corporate test automation tool, and Worksoft Certify. The latter is a complementary tool, especially in the web area where there are some white spots for our corporate tool which it cannot cover. That was the main reason why we brought in another tool. And for that, it fits perfectly.

We are able to run our test phases faster. Once the scripts, the test cases, are ready and automated, not only are we able to check our systems or landscapes during the test phases, but we can proactively monitor our development and test systems. Proactive monitoring of our systems is very important for us and was not possible before because manual testing is just too time intensive.

Worksoft Certify helped us to increase time savings. We didn't start test automation in general with Worksoft Certify. We did automation before with our own tool, but it helped us to increase the coverage of test automation and to increase the time savings.

We had a success story with two teams. For the execution of the scripts, we had time savings of 82, 88, 95, and 90 percent. And for the speed, it was between nine and 21 times faster than manual execution.

It's not necessarily saving us money, but it's helping us to free up the capacities of our end users to work on other stuff. Instead of doing testing for two days, they can work on bug fixing, developing new features, etc. That person still gets the same paycheck at the end of the month, so it's not saving us money, but it increases the value of our products. It increases the quality of our products. The reason for that is we are not customer-facing. We are dealing with internal teams and internal products. We are not selling anything to the outside. We are with the internal IT department. For the development teams and the sales team or the consulting team it might be different. But we are not really going out, selling our products and getting the revenue for the company. This is done by other teams.

It's script-free, which is really important for our end users because we are usually dealing with colleagues who are not developers and who do not always have the technical background of developing and scripting. It's very useful that there is a nice UI and the tool is script-free.

It covers all of the technologies we need to cover.

And one big advantage of Worksoft Certify is its integration with SAP Solution Manager, the test suite of Solution Manager, with the certification. That is crucial for us since Solution Manager is our test management tool of choice.

There are a couple of small things, technically, that could be improved.

Features we have asked for include single sign-on. It's a bigger project to make sure that our end users do not have to store passwords, usernames, and the like, for the different tools we have. 

We are also working on an additional integration with another tool that we have in place for lights-out testing. That's ongoing at the moment.

Another idea we brought is that the definitions for the objects need to be automated. They need to be recognized automatically by Worksoft Certify instead of changing them back and forth manually. This is also something that Worksoft is currently working on. 

Updates, in general, is a topic that we are working on with Worksoft on a regular basis. For new products, for new UI technologies when they come out, the test-automation providers need to update their definitions to make sure that the objects are recognized correctly.

It is pretty stable. After upgrades, we usually don't experience any big issues. Of course, it's software, so here and there we find bugs, but nothing crazy, to be honest. The availability of the system is pretty good, almost 100 percent. I don't see an issue here.

At the moment we don't have any issues with scalability. We have about 300 end users working with Worksoft. On infrastructure, it's split. We have a WTS environment, a Citrix environment, for those colleagues who want to use the prepared environment. We have other colleagues who are using the client on their own machines, on their own laptops or desktops. The only thing that we saw at the beginning which we need to change in the future is that, due to the latency, we cannot use clients in the US, for example, while having the server in Germany.

If the latency is over a certain number of milliseconds then it is basically impossible to do automation. That was one of the main reasons why we set up the Citrix environment at the very beginning.

We are still in the phase within our company, or within IT services, of training and spreading the topic of test automation, overall. So our coverage, at the moment, is not the entire organization, it's only the IT department. Once we have done this - and it will take at least another year - we will see if we spread using Worksoft and our internal corporate tool as a combination, or tool ecosystem, further into the organization. But this is not our not our team's responsibility so it's not really in focus at the moment. We are pushing for test automation in our teams and there is still a huge demand for training and new teams coming into the topic of test automation.

In general, technical support is good. They are collaborative and responsive. The only thing I don't like - and this is the only complaint I usually have for Worksoft - is that the first-level support is not always the best for working on topics. We sometimes need to escalate to second-level support and then we know that we are getting a colleague who is aware of the issue and is not just playing for time.

We already reported this to Worksoft and asked them to find another way or to educate the first-level support or to make sure that the tickets go directly to second-level support if they come from us. The guys on our end who are reporting the issues sometimes know more than the first-level support.

When it comes to second-level support, we are happy. There, we know we will get the help that we need. The colleagues are responsive and very helpful and, from a quality perspective, they are very good.

In the beginning, there were some issues with the integration, it didn't work the way we wanted. We spent some time with the Worksoft team, with the support and engineering team, in adding some enhancements to adapt the solution to our three-tier Solution Manager landscape. But that worked very well.

We have a very good collaboration and relationship to Worksoft. For example, every two weeks we have calls with them. We'll provide feedback and they take it seriously. They usually provide us with updates, with enhancements, with new functionalities that we need. That's working pretty well.

We started with manual testing and then we started the test-automation initiative. We started with our internal corporate tool and then, as I mentioned earlier, we figured out that we could not cover everything with that. At the end of 2015, we started to check the market. We did some PoCs and we decided to go forward with Worksoft Certify.

There were a number of reasons we went with Worksoft Certify. The Worksoft team did a great job. They came to our headquarters and did the PoC, showing that the tool is suitable for our needs. They did another PoC with our operations colleagues who were running the regressions testing in Singapore. And then there were the technical requirements that we had. Worksoft Certify was able to cover all of them, some of which I have mentioned already: Being script-free, being fully integrated in Solution Manager, and being able to script in a modular way. And finally, the integration between our own internal tool and Worksoft Certify was also important.

The initial setup was pretty much straightforward. I cannot give you too many details because I did not take care of the implementation. But I know that it took us about two weeks to set up the whole infrastructure. It was not really difficult. And we had very good support from Worksoft, from the support and engineering team. They helped us in setting up the database, setting up the connections. It was not a big deal. In total, within two to four weeks, everything was working fine.

On our side, we had a couple of staff members involved in the implementation because our team is the application owner. We had to involve two more colleagues from the database team because we don't have all the authorization stuff, for the databases area, for the servers etc. In total we had about three staff members involved, but not full-time. It was about one work-week for each of them.

Maintenance is done by myself and one of my colleagues, with the help of our database and server/infrastructure team. We don't have authorizations for everything and we are not database experts. There are three or four staff members taking care of maintenance, as part of our job; it's not a full-time job, obviously.

Whenever we need to do a full upgrade, when we need to plan the downtime for the production system, we try to make it on the weekend. I also already recommended to Worksoft that it would be nice to have something like an offline update where the system can be upgraded or smaller changes and fixes can be included without having full downtime. For an upgrade we usually need two to three hours. Afterward, we do a bit of testing, so upgrading takes about half a day.

It was just one or two people from Worksoft and three guys on our side.

We get feedback from all areas that the return on investment is there. Not just regarding time savings, but also cost-reduction. The return on investment in one case was reached at something between five and six runs, which is pretty fast, especially in an Agile environment.

What is also very important for us here is the avoidance of human error during the execution of tests. Usually, if someone is sitting in front of a laptop and doing testing eight hours a day, he or she will make some mistakes. This does not happen with a tool. Another important factor for us is the availability for testing. Usually, it's pretty hard to plan a test phase to bring all the testers to the table and get the time blocked off for the test phases. For the tools, we just need the systems up and running and then it's a matter of minutes to set up the test plans and to run the tests.

At the moment we are rolling out Execution Manager.

At the time we decided to bring in Worksoft Certify, we looked into two other tools. The key difference was that Worksoft was script-free. That was not the case for other tools. And the full integration to Solution Manager was one of the key differentiators between the tools.

If you have done a market evaluation and have decided to go for Worksoft, my advice is to go for it. I would definitely recommend Worksoft Certify as a test automation tool.

The feedback that I get from our stakeholders is that the tool is pretty simple to use. What we usually do is a two-week training, not full time, where the total is about three to four business days, 20 to 25 work hours. From there, most of our colleagues can start working with the tool. Of course, they have questions later on, some difficulties when it gets into special activities. But overall, the tool is easy to use. It's generally found to be intuitive.

In terms of cutting test maintenance time with respect to the scripts, that has not happened. If you need to adapt your scripts, automated test scripts are much more complex and more effort-intensive than manual test cases. But this is the nature of the beast. It will happen with every tool. If a screen changes, if a system changes, then you have to adapt your script for manual testing. For a manual script, you just adapt a Word document or an Excel sheet or the like. But if the process flow changes, you have new windows, new options, then you have to adjust your script for each and every provider that you're selecting. The maintenance of scripts is something that I always discuss with my end users and should never be underestimated.

We are not using the Capture 2.0 feature at the moment. We are planning to use it in the future. But at the moment, due to the heavy workload on our plate, we haven't had the chance to look into and to roll it out. We are familiar with the concept of Capture and it's a very nice feature because it makes the collaboration between business and IT much easier, and business can be involved in test-automation topics and activities as well.

We have three roles in our environment. We have the key players, who are the project managers, the persons responsible for test automation overall in the respective teams.
Then we have the test automation engineers who are responsible for creating test scripts and to maintain them; sometimes they run them as well. And finally, we have the executors, the ones who are running the scripts, checking the details and, if something is not working fine, going back to the test-automation engineers and asking for support and help.

I rate Worksoft Certify at nine out of ten. I'm happy with the tool, I'm happy with our colleagues at Worksoft. We have a very good relationship, we can bring up everything. There isn't much I can complain about. I'm happy at the moment with Worksoft.

We are in the retail business, so our primary use case is automated regression testing on our supply chain. It helps us oversee our internal economy, including the warehouse, HR, and the stores. 

We prefer Worksoft over other platforms because it's a low-code solution

We can't get the process intelligence module to work properly. We can't get the impact comment that analyzes the incoming development code to run, either. We've also had bugs in the CTM and execution manager in the past year. It took technical support a long time to resolve this issue. We escalated it so that the vice president of the company was included as well. 

I have used Certify for a few years, but the company has been working with it for five years. 

I rate Worksoft support five out of 10. They have some general issues that they're they're not testing, but when we open a ticket, they give us a hot-patch fix. It seems like they're not testing their products enough. On the other hand, Worksoft support has been responsive to our requests. They've addressed the whole list of issues, so we have no more open tickets. Once they escalate something, they're efficient. 

Neutral

The vendor installed it for us, but there were still many issues. We've reported over a hundred bugs and defects. In that time, we could run stuff, but sometimes it has been standing still. 

Worksoft is reasonably priced. 

I rate Worksoft Certify six out of 10. Before implementing the solution, you need to think about how you will use it because it has so many modules. You should consider how you will use those and do a proof of concept. Worksoft hyped up the process intelligence and impact modules for two years, and they still aren't working for us. Before you buy it, make sure it works in your environment. In the future, I hope they have a more stable product with more internal testing and quality control. 

Our primary use case for Worksoft Certify is the automation of test cases. Performance-wise Worksoft Certify is very good. 

The most valuable feature of Worksoft Certify is that it has a codeless environment, so you do not have to be a programmer to use it. 

Worksoft Certify supports multiple interfaces and applications like SAP, Web, or Silverlight Java, and Mainframe. It is easily integrated. 

I have used the tool for a long time and much has improved over the years. Many new features have been added and the tool improves on a daily basis. The UI is much better now and it looks much nicer as well. 

Worksoft Certify's tech support's response time could be improved. 

I have been using Worksoft Certify for nine years. I use the product on a daily basis. 

Worksoft Certify is easy to scale. We have somewhere between 60 and 80 people using it at the moment. 

We do have technical support from Worksoft Certify. When we do run into issues, we first see if we can sort them out in-house. If we are not able to, we reach out to them and they look into the issue and sort it out. 

Positive

I previously used Selenium. Comparing these two products, I can say that you need much more IT knowledge to use Selenium. Worksoft Certify is quite easy to debug. 

The initial setup was easy. On a scale of one to five, with one being very complex and five being very easy, I would give the Worksoft Certify deployment process a five. It also does not need very much maintenance at all. 

We deployed Worksoft Certify in-house. The deployment did not take much time at all. It took just a few hours. 

Worksoft Certify is a bit costly. It is a good product so the cost is a bit high. But I'm not sure about how much we pay for it exactly. 

The advice I would give to someone looking to implement Worksoft Certify is this: contact the provider and request what you need to be installed. Once the solution is installed, you can open up the user manuals and start learning it. It is easy to learn and implement a project in Worksoft Certify.

On a scale from one to ten, with one being the worst and ten the best, I would rate this product a ten overall. 

The primary use is developing automation on SAP at a medical device company moving from Micro Focus UFT scripts.

We were able to increase the percentage of automation from between 40% and 50% to 80% within a year.

The most valuable feature is the ability to automate quickly and to maintain and update scripts.

We would like this to be able to be used outside of SAP applications, as it would be good for other types of products.

I have been using Worksoft Certify for three years.

We expanded licenses after a year.

Prior to this product, we used Micro Focus UFT.

We evaluated an SAP escalator from SAP in UFT.

My advice for anybody who is implementing this product is to get consulting and on-site training.