What is our primary use case?
With Perfecto, we are trying to automate our mobile test cases. These are test cases that run on mobile devices including Android and iOS phones, as well as Android tablets and iPads.
We are a financial institution and we rely on dedicated devices within a private cloud. Due to the sensitivity of our data, it is mandated that we use a private cloud.
I am the director of automation and analytics within our digital portfolio. We use Perfecto for the visualization and reporting of test results, including the relevant numbers.
How has it helped my organization?
We leverage Perfecto for reporting and analytics, and in that regard, it provides good reporting features. In terms of analytics, Perfecto still doesn't have predictive modeling or similar analytics features in the tool. In terms of the features that it does have, we use them to the fullest.
We make use of the Perfecto Reports dashboard. Within the framework, we use the Allure reports for automation execution.
We are using the Perfecto cloud for both real and virtual devices. For example, we have some cradles for real devices that we have licenses for, we have virtual devices by means of emulators and simulators, and we have browsers. These are the three types of licenses that we have.
The physical devices are sitting in the Perfecto device lab. The emulators and simulators work to simulate Android and iOS devices, and the browsers are separate again. As the main testing platform for our operations, the device lab is very important to us. We used to do everything in-house, which required on-premises infrastructure to do so. We got rid of the VMs and our physical machines, including our Macs. This is, in fact, one of the reasons that we chose Perfecto.
The fact that the device lab provides same-day access to new devices is a good thing for us. We have an SLA for procurement and if we need a new device, our SLA stipulates that Perfecto has a certain number of days to procure it. We are happy because when a new device becomes available, we get instant access.
Another example of how Perfecto has helped to improve our organization happened during COVID. Sometimes during the pandemic, when we were working away from the office but still had the on-premises infrastructure, somebody would have to travel to the office only to restart a mobile device or a Mac machine. We no longer have to do that and that's a big relief for us.
When it comes to procurement, when Apple or Samsung launches a new device, all we have to do is pass the request to Perfecto. This is an example of where we save on infrastructure but it's not the only advantage. We are able to execute all of our tests in parallel on different cloud devices. Previously, we were challenged in terms of scalability with local infrastructure. Now, scalability is not a challenge and we can add as many virtual devices, running in parallel, as we need. Previously, we were limited to running a few devices in-house and using a few different browsers.
Perfecto has helped to speed up our testing effort. This is true both in terms of time to market and in terms of execution time.
What is most valuable?
The most valuable feature is cloud-based automation and testing.
In terms of Perfecto's ability to perform cross-platform testing, I would rate it a ten out of ten. The first advantage is that we can take the entire execution to the cloud. We don't have to maintain a lab in-house. Second, we can have parallel test execution, which was tricky when we were doing everything in-house. Third, we have better browser automation and device test coverage.
When we maintain a physical inventory in-house, we cannot get all of the operating systems or different types of phones that are available on the market. Even when it comes to browsers, for example, if I have to run multiple versions of different browsers, whether it is Chrome, Edge, Firefox, or Safari, it is much easier when we can take it to the cloud. Perfecto provides that with all of the options for all of the browsers, as well as the real devices.
When we onboarded Perfecto, the test cases and the framework were already designed using open-source technologies. We used TestNG for the framework and Appium for the automation. As Perfecto supports Appium, it was easy for us to implement and we are still running these technologies.
What needs improvement?
I would like to see the inclusion of machine learning features. If we can have that, it will be a better tool.
On the analytics side, Perfecto is in need of improvement. The product has a cool dashboard, no doubt about that. But that dashboard is limited to visualization and slicing and dicing the numbers, as well as getting the reports in multiple formats. What I would like, beyond that, is if we can have some predictive intelligence capabilities. This includes machine learning and modeling capabilities. I would really like to see that from Perfecto.
As an example, I would like to be able to predict our failure rate in an upcoming release of our software.
The reliability of devices is very good, at perhaps 99.9%. However, sometimes, we find that some devices are offline and not available. I would like them to achieve even closer to 100% reliability. I know that 100% is not achievable but they can still push that number.
The timing of the software is an area that can be improved. For example, there is a new Appium release and Perfecto should be able to get it to us more quickly. They take their own time for internal testing before they roll it out to customers but we would like to use it faster, so it would be helpful if they can increase the speed of that process. It is somewhat related to improving their time to market.
For how long have I used the solution?
I have been working with Perfecto for more than five years.
What do I think about the stability of the solution?
The reliability of the access to cloud-based devices is close to ten out of ten. We totally rely on cloud devices and don't have a fallback plan. We don't have any in-house infrastructure anymore, which means that we don't test it on any devices that are on-premises. Everything is done from the cloud. As such, it has to be reliable. Performance-wise, I would rate it a ten out of ten.
Overall, I have good confidence in the stability of the solution.
What do I think about the scalability of the solution?
Using the device lab has helped us to scale our day-to-day testing operations and in turn, it has helped to scale our business overall.
Using virtual devices, such as the Android emulators and iOS simulators, is a feature that has definitely been a big help. Testing on virtual devices is a major chunk of our test cases, and it's important because we are not deepening any local infrastructure. We are totally reliant on virtual devices.
With respect to how hard it is to scale, we have a certain number of licenses and we've been told that we can scale up to whatever number we want to. Initially, there was a hiccup when we scaled to testing many devices in parallel, but I'm pretty comfortable with it now.
We generally plan to increase our usage of this product. More teams are being onboarded to the cloud and although we haven't chalked out a plan as of it, we do plan on purchasing more licenses.
How are customer service and support?
The technical support is good. Our service-level agreement states that if we open a ticket, it has to be resolved within a certain number of days. This depends on the criticality or the gravity of the defect. There is a proper process in place for that.
I don't think that the support has slipped on their deliverables, so I would rate it a nine and a half out of ten.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
We did not use another cloud provider prior to Perfecto.
Initially, we were doing our testing in-house. To do this, we were maintaining an inventory of mobile devices. Now, with the use of this tool, we have taken everything to the cloud. That means we are not maintaining the inventory anymore, but we are achieving good results. In addition, we are benefitting through cost savings.
How was the initial setup?
The product is a black box to us, and there is minimal setup. Perfecto set up the cloud and they give us access through a browser and URL. Once we get into their cloud, we can pull the devices and do what we need to do.
As the setup is done on their end, it's very easy for us.
What was our ROI?
Comparing the cost of the product and licensing to what we get for our investment, of course, there are benefits. There is a measurable ROI and we conducted an analysis. In fact, the anticipated ROI is one of the reasons that we chose Perfecto.
Getting rid of our local infrastructure has translated to cost savings. We got rid of VMs, Mac machines, and other infrastructure that was needed. Of course, we do still maintain some devices in-house, but otherwise, we save the money spent on infrastructure. This is from the perspective of purchasing the infrastructure, as well as paying to maintain it.
Overall, it has definitely translated to cost savings and the reason is multifold. We don't have to have a local infrastructure, saving on procurement, support, and maintenance costs. Also, being able to execute tests in parallel leads to additional cost savings.
The additional coverage we get is for the same cost. Specifically, I used to only test using the Chrome browser, whereas now with Perfecto, I get a lot of options. I can test using Chrome, Edge, Firefox, and Safari. This takes the same amount of time and we incur the same cost.
What's my experience with pricing, setup cost, and licensing?
Pricing is an area where Perfecto can do a little better. When we obtain additional licenses, we enter into negotiations with them.
For customers like us, who have been with Perfecto for many years, it would be helpful if there were more discounts available.
Which other solutions did I evaluate?
Before choosing the product, we did an evaluation and the good part was that, in another area of the company, in the wealth management department, they had onboarded Perfecto before we did. This gave us some insight into the product before we began our proof of concept.
After the PoC was complete, we procured the solution.
What other advice do I have?
To sum up, the advantages that we get from this product start with the infrastructure. We save on procurement, support, and maintenance of it. Next, we were limited to only a few devices that we had in-house. We were also limited to only a few browsers. We now get more coverage on the browsers and we have real devices. Finally, the parallel execution translates to cost savings.
My overall feedback on this product is good. My advice for anybody who is considering it is to evaluate and assess the product, as well as conduct an ROI analysis, before procuring it. Individual needs might be different. You might realize your return on investment after four or five years. Or, the break-even point might be longer for you.
To conclude, this tool is pretty stable and robust, so there should be nothing to worry about on the technical side.
I would rate this solution a nine out of ten.
Which deployment model are you using for this solution?
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.