What is our primary use case?
My main use case for Digital.ai Release is test management and automated testing solution. A specific example of how I use Digital.ai Release for test management and automated testing is that we have three brands which have their own websites and mobile apps, Android and iOS. We currently use Digital.ai's mobile manual testing capability to perform testing on our devices and for our applications across all three brands. We also conduct cross-browser, accessibility, and analytics testing using Digital.ai. This is the core of my main use case.
There are other solutions being experimented with at the moment in other departments like SaaS platforms and data center solutions, but my main use case is centered on testing and automation.
How has it helped my organization?
Since we started using the manual testing module from Digital.ai, I have seen a positive impact on my organization as our test management has improved significantly. We are now able to gather all of our test cases in a single repository and run our test cases across multiple teams through the same tool. We are also able to generate standard reports which anyone can easily access, giving us a better view of what we are testing and a better view of our code and quality coverage.
The improvements in Digital.ai Release have definitely affected my team's efficiency and productivity, with time saving being a major impact we have seen as part of this implementation of the tool. Previously, we used Confluence and Excel spreadsheets for our test case management and tried a handful of commercial tools, but they did not work for us. With the introduction of Digital.ai manual testing, we are now able to seamlessly integrate our existing automated test pipelines and our existing build pipelines, as well as our documentation tools, so that the data from Digital.ai manual testing tool seamlessly flows through other tools and we can see the coverage of our code and our test cases. The entire team is now using the tool, collaborating through the same repository, and we are creating our test cases and getting our test case coverage increased faster than previously. Standard test cases are available, and people are simply executing them and sharing links for the execution rather than downloading reports and creating PDF files to send across, resulting in significant time saved in planning, managing, and executing test cases. The major advantage for us is the visibility of our code test coverage and the reusability of our test cases.
What is most valuable?
In my experience, the best features Digital.ai Release offers are that the automated testing capability for Digital.ai is limited, but the manual testing capability is very good. What makes the manual testing capability stand out for me is that with Digital.ai's manual testing, we get access to a bunch of virtual devices hosted across various regions, allowing us to easily conduct cross-browser, cross-platform testing. They are readily available for us, and all those mobile devices more or less work like physical devices for us with capabilities of testing hybrid applications, native applications, or web view applications. They have facilities like biometrics and sensors, and we also have the capability of Apple Pay and Google Pay testing. We also have access to logs so we can look at network logs and digital logs to see what is happening on those devices. When we test it manually, everything seems to work seamlessly, but with automated testing, it is hit and miss.
Digital.ai Release definitely has good automation capabilities, but as I mentioned, it is hit and miss. It has CI/CD integrations, but the major part which is missing is that the AI capabilities are not that mature yet, which is the reason we only use it for manual testing at the moment. Although we do have capability and teams available to experiment with Digital.ai, we are leaning towards in-house open source tools which we have implemented to conduct mobile automated testing.
What needs improvement?
Digital.ai Release could be improved, particularly because the automated test capabilities are not that strong compared to some major tools like UIPath, Tosca, or SmartBear TestComplete. Finding elements, the list of assertions, or the usage of AI to automatically generate test cases based on requirements did not work entirely flawlessly for us, and we have found issues with these features.
Regarding Digital.ai Release's AI capabilities, we have not explored a lot regarding governance and security, as we have only explored mobile manual test management and automated test management. There are definitely other areas where Digital.ai Release could improve, such as workflow management, which UIPath has very good capability for. There is also support for desktop test automation, which I think can definitely be improved. Speed-wise, if you compare Digital.ai's automated testing solution with Cypress or Playwright on web applications, they are very fast and run cross-browser with parallel test execution. I think those aspects can be improved in Digital.ai test automation, along with better integration with tools. I have not seen Jira integration, but other commercial tools have extensive integration with CI/CD pipelines, documentation tools, defect management tools, and test management tools. There are also integrations with Slack, Google Chat, and emails—areas where Digital.ai can also improve.
For how long have I used the solution?
I have been using Digital.ai Release for the last six to eight months.
What do I think about the stability of the solution?
Digital.ai Release is stable, but there are definitely improvements we can make.
What do I think about the scalability of the solution?
Digital.ai Release's scalability is very good, as we can add any number of users and expand it organization-wide or to a handful of teams. I have not seen any problems with scaling out the manual testing solution, but with automated testing, we might have limitations when we run with a lot of parallel pipelines, although we have not explored that yet.
How are customer service and support?
The customer support for Digital.ai Release has been good. Last time, our finance team and our infrastructure team reached out to their team members, and they responded within a few hours. Even though we are based in Australia and there might be delays because of time zone differences, we have had good responses. For the few instances we did reach out, I think we were responded to quickly.
Which solution did I use previously and why did I switch?
Previously, we used a test management tool called Testmo, which was not a market leader but definitely a commercial tool. It did not help us due to its lack of AI capabilities, poor integration with Jira, and clumsy execution capabilities. Sharing test cases and results was also cumbersome. We were looking for a solution that could seamlessly integrate with the various tools we were using and make it easy to share reports with our stakeholders, which is why we switched to Digital.ai Release.
What was our ROI?
I have seen a return on investment with regard to the mobile testing capability we have used. For every release, our QAs used to spend around four to five hours on test management, reporting, execution, and release. Now that we have introduced this tool, we are saving at least one or two hours for each release. This means four to five hours saved for one QA on each release, and with multiple QAs doing multiple releases across our three or four different brands, we are saving days within a week.
What's my experience with pricing, setup cost, and licensing?
The pricing, setup cost, and licensing for Digital.ai Release are a little expensive when I look at it, especially the enterprise-level licenses. However, since I have not worked with licensing directly and our finance team has handled that, I cannot comment a lot on that aspect.
Which other solutions did I evaluate?
Before choosing Digital.ai Release, we evaluated Qtest for test management and also looked at Zephyr and Xray. However, those tools were purely test management tools. Digital.ai was the only available option that combined manual testing and automated testing capabilities together, but as I mentioned, the automated testing solution did not work fully for us as we expected, which is why we currently only use it for manual testing.
What other advice do I have?
The mobile manual testing capabilities are very good, while the test automation capabilities can be improved. My advice to others looking into using Digital.ai Release is to not expect a lot of new features in the tools. Digital.ai Release can be comparable with other commercial tools available in the market, but since it is a new tool, it might not have all the important and exhaustive capabilities that other tools have. Start with an open mind, assuming that it is a new tool in the market with minimum capabilities for typical manual testing and automated testing for mobile devices. However, I do think you can expect the tool to work well from the start, at least for manual testing. It is good that you are taking surveys from customers, as more surveys could lead to more capabilities and feedback, which could improve your tool. On a scale of one to ten, I would rate Digital.ai Release overall around six.
Which deployment model are you using for this solution?
On-premises