Its primary function is test automation. We record the test procedures and execute them in an automatic way. It gives us reports, in a centralized way, of all the information that we need to manage test automation for the ERP system and a few other systems as well.
We moved from manual regression testing, once a month, to continuous testing every second day, with the same scope or even an increased scope now. We don't need to focus on the manual regression tests, so we've freed up hands for support activities and new developments. I don't think that it has changed how our organization works directly, but we have become more efficient and more flexible.
In our estimation now, we save about 160 hours of manpower every month because we use Certify. It's 80 hours of robots executing the tests, and they're doing it roughly three times faster than humans. To do the same manually, and keep up with the general pace of development, we would spend 240 hours on testing every month.
In terms of codeless, end-to-end process automation across packaged applications, technically speaking there is some programming or development, like you would learn in middle school. But you don't need to be a professional in any programming language to run it. That's how we've done it. We don't deploy any additional software packages with it. We haven't developed anything else, other than using Certify steps to run end-to-end test automation.
Nevertheless you still have to have a mindset of a developer or an engineer. You need to have understand cycles and other primitive functional blocks. Still marketing-wise, I would call it codeless due to its simplicity.
SAP is our core and the people involved with it are not always developers. Some of them are analytics or system experts. They don't know how to write C++ or Python or Java. But they can do testing automation in Certify themselves without help from a centre of competence (CoC). The only time CoC will participate is when truly complicated logic required such as runtime variables staking.
In usual situations, the skill set to start doing test automation needed is much lower.
There are a few aspects that are valuable. First of all, there is the screen-grabbing where you can pick up objects and it automatically recognizes them on the screen. Worksoft call it Certify Capture or LiveTouch. It's automatic capturing properties of the UI objects on the screen or on a mobile device for use in test automation scripts.
Other important feature is the modularization of the tests. That's an important feature not only of Certify but of IT tools overall where you create and reuse the components. Our test scripts are done with a "Russian doll" approach. That's standard for modern IT, however sometimes you see IT products without such functionality. Worksoft Certify's ability to build tests and reuse them is done pretty well and balances simplicity with flexibility.
Due to it, the learning curve improved even further allowing non professional test developers to plug-and-play test scripts. Such approach allows you to have that nice, modular end-to-end test. Can it get better? Probably not without sacrificing some other nice features and simplicity.
In terms of the core functionalities important to us there are the actual coding, great test data management, execution and reporting. Those are the basis, which are shared across many test automation tools.
We also value that the test execution and reporting are stored in a database. We can extract the results and get the data out more easily comparing to text log files. That's different compared to other tools on the market. In Certify, we have a structured test execution and overall structured data.
Last not least is multi interface support is important for our applications landscape. With certify we can test SAP, Net, Java, Web and Android applications with one tool.
Certify's web UI testing abilities for testing of modern applications like SAP Fiori was good when we started and they developed it to be even better. We all know that web applications can change objects in DOM quite fast and it breaks tests. To counter it Certify has made object recognition more flexible and generic, so we don't have any troubles.
For standard and UI stable applications like SAP ECC 6.0 we have no trouble at all due to predefined objects and rich library of methods. That means we don't need to spend time writing a library, which could happen in some of the open source frameworks.
One caveat is that if you start running models in different parts of the end-to-end process — when you really try to hit the sky and make everything automatic, to cover multiple supply chain tiers processes in one e2e test, or similar processes that are really complicated — then tool simplicity turns into disadvantage. On other hand it stops us from unnecessary over engineering the test automation framework.
Architecturally, because Worksoft specifically built a database-oriented application, you essentially store code in the database. Git and text files orientation is more traditional approach with boundless set of tools to control versions, manipulation and analysis. But at the same time, Worksoft supplies us with their own version control inside Certify that has sufficient functionality for now.
When it comes to mobile testing, we have a small bottleneck there. You have to buy third-party separate licenses if you want to test on a mobile. Business wise we see room for improvement there, although it's that really critical for us.
None of these are showstoppers for our operations. Worksoft proved to delivery significant improvements in last 3 years and more we wait from 11.5 version. Overall, we are quite well covered with test automation related tools and nothing special is needed.
We've been using Worksoft Certify since 2017.
It's stable. Our approach is that we don't deploy the latest fresh version to production straight away. We wait for a few patches and follow with upgrade instructions. It doesn't matter if it's Certify, DB server, Linux or anything else.
Only case I could recall related to stability problem is a one mess-up in the database during over-few-versions upgrade about three years ago. It required some database cleanup with scripts provided by vendor support.
But looking at the overall stability, it doesn't go down or crash. SLA is 100 percent for us.
Because it is three times faster than manual testing, we can test more with fewer people. That gives us operational scalability already. Platform also has the Execution Manager tool if we would like to have technical scalability via a cluster of virtual machines. In addition Certify itself supports multi-agents, APIs, and integration with Jenkins and other applications including PowerShell. That means we can also develop scalable setup ourselves
As for overall scalability of Certify, licensing is the main bottleneck. You cannot scale more than you have licenses for. You cannot uplift your digital workforce just for one month and run 300 nodes strong cluster for a big upgrade tests. It is not different for majority of IT products, we don't see much of a problem for now.
Certify is our main tool for test automation for ERP, for PLM, and HR. It's covering two major applications that are at the core of any big machinery or manufacturing corporation. We continuously increase the scope, because systems are developed and extended.
We use their enterprise-level support and it pays out. We mostly use it for tricky situations. We often talk with 3rd level, with architects and other experts behind the certify. It gives us good insights. Separate gratitude goes to active key account management on vendor side.
As for our improvement requests, we speak with the community and we speak with architects and provide necessary information. We don’t know do they take it into their backlog or is it just coincidence that Worksoft have provided most of things we asked for. In any case we get what we require for our testing, we in constant contant and we feel that we are listened to.
We did not have a previous technical solution.
The initial setup was pretty straight forward, in general. There is a database, there are clients to install, and you need a licenses server.
The key thing that you need to know is what you are doing, how you do test automation, in general. That's above the specifics of any tool. You must know how to slice system-under-test into functional components, make the right model for reuse. It is important to break down end-to-end testing into smaller reusable items. That's not trivial, and you faced such challenge with any tool. You have to know what you're doing.
The deployment included one of week training and then a proof of concept for a couple of months. We ran typical end-to-end process in the proof of concept and found our ways to deal with above mentioned slicing, development guidelines and how to establish roles. There are useful guidelines provided by vendor as well. Overall it took about one quarter from the GO decision to start running regression tests.
Our approach to testing is as a "safety net," so we don't need to have end-users who run it and hunt for new defects. It's just continuously testing given scope and raises a red flag if something goes wrong. Such approach secures an immediate feedback for the development team or for the quality assurance team.
End users are not involved into testing usually although we could run hybrid testing with a mix of automatic and exploratory tests.
We didn't work with any third-party. There was initial training from the Worksoft itself for one week. That was it.
The return on investment, when we count the 160 man-hours we save monthly, will take about seven years for our scale
License model is a traditional one. As with any other manufacturing company, the software testing is not our core business, it's not in our interest to invest capital into the licenses. It would be preferable to have a pay-per-use model. At the same time, it's a fair game, for now.
Aside from their standard licensing fee, if you want to have enterprise support, there's an additional cost.
We have done a few comparisons and we're continuously looking at the market to see if there is anything better, cheaper and more suitable. Market is changing a lot, RPA solutions expanding to test automation market, Robot Framework ecosystem is thriving and some other established vendors are operating. We're keeping an eye on it for evaluation. So far, cost-wise, feature-wise, and in terms of the learning curve or skills requirements, we are good with Certify. From our comparisons now, we would pick Certify again.
As always, you have to know what you're trying to achieve. You also have to think about how do you model your system-under-test; how it is written, the quality requirements and standards. The key benefit comes from reusable test scripts.
When it comes to Certify itself as a tool, keep your eye on objects. If you map something, keep a good inventory so you understand it and you don't put all the objects in one big basket.
For me, the biggest lesson from using Certify is that you can do quite interesting and complicated things with the codeless approach. You don't really need a complicated programming language to do testing.
Overall, I would rate Worksoft Certify an 8 out of 10. Of course we would like it to be cheaper and more powerful and developed faster. But are there products that I would rate a 9 or 10? I don’t thinks so. It's at the top of the market from what we know about the market.
Partnerize Partner Management Platform (PMP) is an end-to-end, SaaS-based solution for forming, managing, analyzing, and predicting the future results of partner marketing programs using artificial intelligence. It is a real-time technology and, our platform comes up with regular updates and requires it to be tested in all major browsers. This became increasingly important as we built a wide user base across even legacy browsers.
We want our web app to work on every single device, at the peak performance, as any bug or loss of data will result in loss of time contrary to the 'Real Time' promise we give to our clients. So, we have to thoroughly test our whole environment before pushing any changes across all possible use cases.
Before LambdaTest, we were using our own infrastructure of VMs. This was not easy to manage and consuming significant resources on maintenance only. We were a small team when I started working and the DevOps team was already under constant workload, and a VM infrastructure is very costly, especially for us as we need to test our product on multiple devices.
In dire need of scale, we realized our existing practices will not going to last for long. After a lot of brainstorming and product trials, we switched partially first, and then fully to LambdaTest.
LambdaTest was handy and easy enough to save us a lot of time, effort, and money. Especially parallel testing, which was the breather we needed to scale. Without a doubt, LambdaTest is one of the big reasons behind our faster deployment and better team collaboration.
The most valuable feature is reliability. My team automates multiple tests in parallel across hundreds of browsers. The automation grid from LambdaTest is robust enough to execute them without any surprises, every time.
Integrations are very helpful and I have lost the count of integrations that LambdaTest has. The platform is cleverly coupled with all of the other platforms we need on day to day basis for our development needs. Pushing the bug on Slack is a click away, for example. LambdaTest's Slack app gives results of screenshot testing from the Slack command itself. We in fact integrated it with Trello to mark bugs in our tasks and stories. And, we integrated our Jenkins jobs with LambdaTest using its Jenkins Plugin.
Though the platform does well in almost every aspect for us, there are a couple of things they can do to give it an absolute edge.
I have been using LambdaTest for the last year and a half.
We used Virtual Machines prior to LambdaTest. That infrastructure was extremely costly to scale for our ever-increasing variety of browsers. It was also consuming a lot of our DevOps hours.
We evaluated BrowserStack, CrossBrowserTesting, and Sauce Labs.
We were testing it as a user interface on various systems which we manage.
It just added some flexibility. There was nothing that improved our coding standards, etc. because all of our UIs were functional before we tried it.
If you ask me feature-wise, I don't think I have tested all the features there—around 40 to 60 percent— nor I was interested in doing do that because the trial was so short. However, there were a few devices that we did not want to procure but we found them within BrowserStack architecture when we tried it out. More than a feature, it was the hardware support that we were able to get through the interfaces. This helped us address a very rare requirement that had come up within our end customer's stack.
Adding better integration with frameworks, particularly testing frameworks like Robot, would be of more value to customers and make their jobs easier.
I only used it for about two weeks during a 14-day trial period about four or five months ago.
This at least gave us good stability and the results that we saw were quite reasonable enough.
It was easy enough. Because I come from an IT background and have done deployments of many solutions, it was quite simple. It didn't take more than 10-15 minutes on that day.
I would rate BrowserStack a seven out of ten.
