Perfecto Reviews

Our use cases are execution on the cloud for web and mobile. With mobile, it could be different platforms, like iOS, and Android. We have functional execution and API test execution. That is primarily what I use Perfecto for. We have a software product that tests the product on different platforms of web and mobile. In an automated fashion, not just manually testing it but also with automating a lot of these scenarios where we execute them, Perfecto runs the execution for us against these different platforms. 

We have 100-plus members registered with Perfecto who use the product at some point. There are multiple deployments across our departments. For example, at the Bank of Montreal, our everyday banking is one application that folks use to do banking. Then, we have other products like stocks, trading, and other products. There are multiple teams who use Perfecto, even though they fall under different suborganizations within the organization. It is open right now for anybody in the company to use it. At this point, there are 100-plus people using it, at least 30-plus teams who may have used it until now, and at least 18 to 20 teams actively using Perfecto within our organization.

Perfecto has definitely helped our organization, mainly from a scaling point of view, with its reporting, and choice of frameworks.

1. It has allowed us to test across different configurations, helping us with our test configurations. 
2. It has helped us with our reporting. Perfecto's reporting allows us to have a clear view regarding what tests have been executed and the health of the application. 
3. It has allowed us to plug in our framework, i.e., the way that we designed it, into Perfecto with modifications. That fluidity and freedom are there for us. We are not tied or reliant on any one particular vendor. We are free to pick and choose with whom we want to work with. 

Perfecto has affected our software quality in a good way. It has allowed us to execute on-demand and on-choice. We also track the number of issues that we find in the product. Every single day, we tag the issues that we found. For example, if something was found by automation, that means it was found by a Perfecto execution. Over time, we realized the real value in tracking those numbers. We can see now that we have clearly been finding issues earlier. It has allowed us to catch our defects earlier, thus improving the quality of our applications.

The most valuable would be their Live Stream analysis, where I can see the live analysis of all the executions on a single device or multiple devices as well as track them. The live analysis and reporting would be the single most valuable feature.

We leverage Perfecto’s reporting and analytics a lot. From the CI Dashboard, it is mainly the status, which is the past, failure count, and time consumption, e.g., how much time did an average test or script take? Along with that, it provides the historical view compared to the previous result, e.g., am I a pass or fail? Also, the stack trace is very important. Whenever a pass occurs, we don't look beyond that. However, whenever a failure occurs, the stack trace information that it gives us is pretty critical for us when figuring out where failures lie. It gives a summary for the pass/fail count, total test count, the historical view, time consumption for each test as well as the total tests, and the stack rate of the failure.

Perfecto's analytics are very important since we use them on a daily basis. We run our executions daily. After every execution, we pull information from the Perfecto reporting system and share that with our stakeholders. Having this information accurately reported is pretty important for us, so everybody is aware of the current status of the product. That way, we can evaluate the health of the product or environment against that which has been executed. Therefore, it helps make those real-time decisions and highlights the impact to the business.

I found Perfecto to be pretty easy to use while executing against cross-platforms. The main reason is because the same script or test automation where we execute on multiple platforms has minimal changes that I need to do. Also, it is easy for me to set up an execution on one platform, then on another platform, either in parallel or one after the other. Parallel opportunities save me time. Once the execution has been completed across these different configurations, I can always check and compare, e.g., what are the differences and consistencies? 

We utilize Perfecto’s cloud-based lab to test across devices, browsers, and OSs. I use it occasionally for manual testing. Though, there are other team members who use it more frequently than I do. I use it mainly for executing my automated tests. We have the Perfecto lab, cloud devices, and machines. I can program my test to execute against any of those devices, which gives me more confidence in my product. I can compare and see how my product or application functionally behaves across these different devices and from a UI point of view, which helps me a lot.

The device lab is extremely important to our testing operations. We rely on having multiple devices up and running all the time. Whenever we kick off an execution, there are multiple reasons why executions may get triggered: 

1. CodeCommit
2. A scheduled job.
3. Might be on-demand by any stakeholder. 

We need the lab to be available, as we need devices up and running for executions to take place. Also, the devices help since they allow us to have parallel execution, and not just wait for a sequential device to become free and available. Therefore, volume is definitely key. It also gives us an opportunity to compare execution across platforms in that space.

It is extremely important to you that the lab provides same-day access to new devices since we analyze that data every single day after execution. 

Perfecto provides their own framework called Quantum Framework. That is one option. The other option is, if I want to have my own framework, I can have a Java-based Maven project, take a Selenium library, AppiumLibrary, and REST Assured library, and utilize the open-source framework. It is easy for us to connect to Perfecto, no matter what framework we use, as long as it has these core libraries in it. I can design and structure it any way that I want. The execution will happen in Perfecto no matter what since they have support for these tools or libraries. It is pretty neat that way. We are not dependent on using just one particular framework to use Perfecto. While there are still some framework limitations, there is the opportunity to use multiple, different open-source frameworks, then pass the execution to Perfecto. We can use most frameworks, then design and craft it any way that we want, then just pass the execution to Perfecto.

One improvement would be speed of execution. If it is an iOS native app, we have noticed that the speed is a bit slower. Perfecto might need to make some improvements in this area. 

I feel that some of the Perfecto reporting might require upgrades or enhancements to their APIs when pulling some numbers or stats. Perfecto reporting also requires upgrades, mainly from an API point of view.

I have been using it for five years.

I am happy with its stability. I have noticed that in the past five years or so, Perfecto has been quite stable. It is always up and available for us whenever we execute. They notify us, well in advance, if there will be any downtime or maintenance activity on their end. We have a Perfecto representative on-premises at all times, so information is funneled in well in advance about those maintenance windows. From an uptime perspective, it is pretty stable.

If you are using Perfecto, the automation solution requires some maintenance. However, just to use Perfecto, there is bare-minimum maintenance. Mainly, it is to check that our networking policies and firewall rules have not changed, and also to determine if we can still access the cloud from our company network. 

I am not sure how much of an effort it is from the Perfecto side to maintain their remote devices. We don't get involved with those. However, from an automation point of view, we need to maintain our executions and make sure that our scripts and code are healthy in order for it to work with Perfecto. So, there is minimal to moderate maintenance, not too critical or major.

Perfecto’s device lab has helped us to scale our testing operations, which has improved our organization in multiple ways. Instead of hosting our internal lab, Perfecto allows us to have a one-stop execution model where all teams plug into Perfecto and execute, and the reporting is available for everybody to see. Also, if we want to scale, we can always either acquire more licenses or cradles, then add more devices. As technology changes, or as devices change over the years, we can change them by upgrading our devices or upgrading the software. Maintaining them is not in-house. It is easy for us to contact the Perfecto help desk and ask them, "Hey, can you change a particular setting on that particular device?" or, "Can we upgrade this to a different Android version?", if we cannot do it remotely.

From a resourcing or technology point of view, it is very easy to scale. However, from a licensing point of view, scaling might be a challenge because it is expensive, but other than that, it is technically easy to scale. Licensing, while not a roadblock, is a challenge for the majority of organizations when scaling.

The customer support has improved over the past two years. They have probably remodeled their customer service, changed something, or added more resources. Now, it is easier for us to pinpoint an issue, explain it, and then get resolution for it. 

I would rate the support team as 10 out of 10.

Positive

We previously used our in-house lab, not an external vendor. We switched to Perfecto because it was hard for us to maintain our own lab. Plus, we got other benefits with Perfecto, e.g., we got to know about their Quantum Framework after we partnered with Perfecto. Mainly, we switched because of their remote lab. That was a prime motivator to move to Perfecto, exposing those licenses and configurations of devices to other team members within the organization. Also, having access to Perfecto's documentation allows us to automate tests better.

Although Perfecto is a good product for us to use, it is a bit expensive. It takes management a bit of work to find the appropriate funding for us to keep Perfecto. I imagine there could be some way to make it more accessible. 

When I moved into the organization, we were already finalizing Perfecto. I am pretty sure that there must have been comparisons made with either DeviceAnywhere or Sauce Labs, but I am not aware of what those decisions or comparisons were.

Do a proof of concept based on your organizational needs, not just treat Perfecto as a remote cloud solution. There is more to it. Check if the reporting of Perfecto suits you. Check if the time that it takes to execute on Perfecto is what you really need, e.g., are you able to plug your CI pipelines into Perfecto? 

If you go with Perfecto, use Quantum Framework, which is quite nice. It is quite easy to onboard test automation engineers to this framework. Plus, it is not too rigid. You can refine, maintain it easily, and then design or change it to your needs.

Perfecto’s virtual devices are on our to-do list, probably for this quarter's roadmap.

We don't use BlazeMeter yet.

I would rate this product as nine out of 10.

We have a website as well as a mobile app, Android and iOS. We do a lot of functional, UI-related testing. And we have automated a lot of test cases through Perfecto, and we have even started to automate mobile now.

Perfecto's device lab has helped us to scale our testing and it has made testing easier for us. We've been able to run our regression testing. We have hundreds of test cases, the majority of which are mobile-related, and we use Perfecto for them.

We've also been able to get our testing done a lot faster, especially during COVID when we weren't going to the office and didn't have our physical devices. We ended up using and being dependent on Perfecto a lot.

We have access to 15 or 20 devices. Even if some of them are being used by other testers, I can utilize another one. That was not stopping my testing when I was doing manual testing. That has helped us a lot

And for automation, we run our jobs at night when these devices are not being used by manual testers. Just the fact that we can automate our tests on Perfecto and run our jobs and have test results in the morning has given us a lot of flexibility and has helped a lot.

We've been able to automate test cases that are mobile-related. That's helping a lot with our automation. Obviously, you can't automate on a physical device, it's something that we do on a virtual machine. The fact that we can automate on mobile, Android and iOS, is working well for us.

We use the CI Dashboard. Perfecto's analytics are very important because that is where we have our test results, which we later input into Jira as well.

We use the device lab in our testing operations and mobile is one of the things we use. We have multiple devices, about 20, including Android and iOS. That's what we are running our automated test cases on. The device lab is extremely important because we are doing our automatic test cases and even our manual testing there. We don't have physical devices. We go into the Perfecto website and use all the mobile devices available on the cloud.

Also, we're working in Agile and we need results ASAP. The fact that the lab provides same-day access to new devices is extremely important to us.

We are mostly using Selenium and we are slowly putting our hands on Appium as well. That's working out well for us too.

When using devices on the cloud, it lags quite a bit at times. I know that these are real devices that are being projected on our laptop screens and monitors, but if the speed could be improved, that would be good. For example, if I open up a Google Pixel on my laptop, using Perfecto, just navigating on Google Pixel is quite slow. Improving on that would help us a lot.

I've been doing manual testing with Perfecto for a while, but I started automation a few months ago. That's when I really got to work with Perfecto properly.

My main area of concern when it comes to improvement of the solution is that it lags often. If that could be improved, then it would make our testing a lot faster and more efficient. Apart from that, I'm pretty happy with Perfecto. It's going in the right direction.

It's definitely scalable. For mobile testing, if Perfecto gets faster then we can utilize it in our bigger test cases. We do have some data-related test cases that really slow it down. If Perfecto can find a way to handle that, it can definitely grow with us.

The technical support gets back to us within a few days. I have not used it that much and I don't have complaints, nor have I heard complaints. My team lead is the one who uses the technical support more often, and he's quite happy with it.

Positive

It's a great tool for running functional, UI-related test cases. But I would give a heads-up about the speed that the mobile devices run at on our laptops or monitors, because it can slow us down a bit.

But in the bigger picture, it helps us because, obviously, even when doing manual testing, you're not going to be testing on your own personal devices. With a lot of people working from home, you cannot give cell phones to everyone to test on. Overall, it definitely helps.

We started off integrating Robot Framework, which is an automation tool, with Perfecto and mostly we've been using it for web UI automation.

Recently I've joined in with another team member to use it with a Quantum solution provided through Perfecto, instead of Robot Framework. It's like a Java-based implementation. It's already integrated with Perfecto and our use cases for that are web and API.

And, while I'm not involved with mobile, we have projects that are using it for mobile as well.

It's given us some traceability, from a QA perspective, on when certain functionalities are available in online banking and when they are down. 

We have also used it for production automation to replace the manual labor that was done overnight on deployments to production, where someone was having to test them. We offload our scripts to Perfecto. 

And we use it not only for viewing the current execution, but as a historical tool to see our past executions and see when things were working and when they were not and where in the code did something fail. Perfecto is nice because, for Java code, you can see exactly where something failed, at which steps, and it shows you exactly which file actually it failed in. That's really helpful for us in narrowing down where the failure occurred. We know whether to investigate the automation scripts or whether it was a data issue or a defect. Perfecto gives us a lot of visibility into those things.

We are catching issues a lot earlier when it comes to the environment, because if we didn't have the automated jobs running with Perfecto, we would probably let a lot of these environment issues pile up until a lot later. But we're running them daily and seeing where things fail and Perfecto is giving us that video to see exactly where it failed and in which line of code. It's giving us a bit of a bump "left" to see where things fail first. It's definitely earlier than it used to be.

It has also definitely helped to speed up our testing efforts. When the scripts are stable, Perfecto is awesome. If there's not much change, we can run thousands and thousands of test cases every day. Someone from manual testing could execute, maybe, 30 test cases a day. Perfecto can execute, say, 3,000. It's really a huge difference. Part of what I like about Perfecto is that it allows concurrent testing. You can hit multiple browsers or devices at the same time, which makes it a lot more efficient for us to execute a regression.

And in terms of software quality, in one of our online banking projects that we recently delivered to production, we found 300 defects just with automation alone, and we found regression defects in addition. Not all of them could have been caught with manual testing. We improved our application by catching defects earlier, before they went to production, thanks to Perfecto.

The number one feature, which if we didn't have out-of-the-box would be missed, is the fact that we have video execution. That gives us the ability to view errors or defects in the progression, from beginning to the end of the video. That's like a full test case. If we were to go with our automation, we would need to go either with Perfecto or some other vendor because, out-of-the-box, Selenium doesn't come with videos.

Another thing that it supports, besides Selenium and Appium, is that it plugs into Robot Framework. That is really nice because the Perfecto folks have created an integration between Robot Framework and Perfecto. Also, REST Assured is taken into consideration. The only thing is that, with Rest Assured, because there is no browser, you need to specify a connection to Perfecto. You need to call keep-alive to say, "Hey, perfecto, track this." It would be nice if Perfecto could do it out-of-the-box, even though we did it with our code. There are also other REST API libraries for Python, such as a request library in Python.

As a regular user of the solution, day-to-day, I go through the reporting. I view the failures and the summary of test sets and test results. The reporting is a very important feature. There's an API to integrate with it within the code and that's really a plus. It integrates well inside. You can also tag it with which branch you're using from Git, or you can label it with whatever you'd like. It's a very cool feature that they added soon after they had the reporting.

For reporting, I use the CI Dashboard and I use anything to do with ALM because we have our jobs running through ALM for test management. You can report your results there. We also have Robot Framework still running so the dashboards we have are OLBQA56 or OLBQA3. Other than the CI Dashboard, I check out the Report Library quite a bit. I have my own filters on the reports and I'll filter based on the last 24 hours or 12 hours, as well as based on the job name. And, of course, I definitely use the Live Stream to see that my tests are up and running.

Also, Perfecto allows us to run on the cloud. Instead of running on our local machines, we can offload it to a cloud and we can run off servers that are not even local. That helps in case we do restarts. We don't need to worry about our machine restarting. Everything is on the cloud and running robotically.

In terms of cross-platform testing, I have used it mostly for the web and it's pretty strong. It's probably an eight out of 10. The reason it's an eight is that we've had a couple of issues lately with videos not loading or browsers dying after some execution, although that happens very rarely. For mobile, from what I have seen, it's probably also an eight out of 10. 

One thing they need to have is physical devices. And I am hoping in the future that there will be more virtual devices.

Also, there is some error tagging that you can do in Perfecto, but it didn't work very well when I was using it. I wish that you could tag it with an error message per test case and it would be done automatically, instead of manually.

In addition, if Perfecto could use AI to say, "Your environment is stable," or "not stable," and give a score on that, that would be really helpful. It could notify us through email. That would help, aside from the failed and passed test cases. It would be something more intelligent. If there were certain tags with certain errors and there were a certain number of them, it could tell us if login was broken in the environment, for example. There's definitely a little bit of room for improvement in terms of AI.

I've been using Perfecto for about four and a half years.

I would rate the stability at seven or eight out of 10, because there's the occasional video issue or the browser dies.

The scalability of the solution is outstanding. I don't have any concerns with the scalability. As long as we have more licenses, we can scale. We only have 50 licenses at the bank, so sometimes things get pretty much filled up, although not very often. We may actually request more licenses, but we need to discuss that internally first.

They're quite responsive and they do reach out for more information, which is good. Lately they've been coming back with fixes, which I really enjoy. When a company can say, "Hey, it's on us. We're going to fix it," and they come back and they fix it, that really helps. We're in the midst of making sure those fixes are valid and that things are working. I appreciate that they flag those things as something important and try to fix them.

Positive

We were using Jenkins, but we had out-of-the-box browsers. It was in my first year here at BMO we adopted Perfecto.

I wasn't involved in the setup, but I do remember there was some discussion about how to connect to it. Doing so goes out to the web and we have to connect to a proxy. There was a bit of investigation into how to connect to a proxy and how to connect through SSO. It was actually quite a big change for the bank and was flagged as something that required security management to get involved with. They had to give all the permissions.

Qualitatively, we tend to find our defects a lot earlier and we're able to run 100 times the number of test cases that we could do with manual testing, on a daily basis. That saves us a lot of time by having defects caught before we go to production. If customers lose confidence in us as a bank, with online banking, we lose customers. That's definitely something we don't want to do. Perfecto is a big part of our testing and automation and quality, making it possible to have more quality software.

I don't use the Insights Dashboard or the Heatmap. I just didn't really find them useful, although maybe I just need to be instructed how to use them properly.

We use it to automate regression test cases for the mobile app of our application. We use Android and iOS for execution, in both English and French. We also use the solution to execute tests in different browsers.

It has really helped us speed up our testing process. We are able to execute tests overnight and see the results quickly.

Our devices are in the Perfecto cloud and that's extremely important because we have monthly releases and a lot of OS upgrades in our mobile application. We always have to validate our app for both forward and backward compatibility, as well as in the browser, to make sure that all the features of the application are working. Perfecto facilitates delivering the product on time, through the cloud devices, so that there is no impact to our end-users. That's extremely important for us.

The Perfecto lab also provides same-day access to new devices, which is another very important aspect. If we want to do some tests to make sure the features work in the latest Android 11 version, we don't have to contact the Perfecto team and ask them to provide us with access to that particular device. We can get that access and execute tests against the devices.

The CI dashboard tool is very good, as is the Live Stream monitoring. Whenever I want to monitor execution, I can open multiple tabs in Perfecto and it is easy for me to refer to the CI dashboard and the Live Stream. 

It's also very user-friendly. I can see and locate any issues that come in and it's easy to see which test case has failed. Under the Perfecto job name, if you click on it, it will show how many passed, how many failed and due to what issues, such as "element not found" or "object not found" or a Java error.

It also has video recording with the screenshots which is really good.

Another very good feature is the cross-platform testing. We are able to execute in multiple browsers and multiple Chrome, Safari, and Microsoft Edge versions. We have the capability in Perfecto to develop scripts. We just need to capture the capabilities of the different browsers and update them in our Quantum framework and we are able to execute. It's very easy. 

There is an option in Perfecto to capture the capabilities using the tool itself. If we want to run it in, let's say, a particular Android device, we can capture it from there and put it into the code and we can update the information and execute.

The range of open-source technologies that Perfecto supports is really good. I don't see any issue in using Perfecto for that.

In terms of its reporting and analytics, whenever I'm doing any execution for a new release, I just put in my release number and it's easy for me to look for what happened in the test case execution I ran for the previous month's release. I can easily locate it and I can get the information.

In addition, I can schedule the devices I need in Perfecto and I can block those devices so that other users on the team do not try accessing them.

The reporting can be improved due to the scrolling you need to do. We have an option for filtering and generating reports, but we are limited to three months of history. After that, we are no longer able to see the information. If they would maintain the history for a minimum of two years, that would be helpful.

Another issue is that sometimes, when we have to locate a button through physically accessing a mobile app, versus through Perfecto, we feel that Perfecto is a little slow. If they could improve on that slowness in accessing the app, when we want to click a button, that would be great because we feel the difference. An improvement in the connectivity speed is required.

I have been using Perfecto since 2019.

I would rate the scalability as an eight out of 10.

I work closely with my Perfecto vendor and he is very supportive. Every time there is a new update he will share it with us and get our feedback to help enhance the tool.  They help update the core into the Quantum framework. We have had some key, latest, knowledge-sharing sessions with their team regarding these new updates.

We also have the Perfecto support portal, where we'll raise incident tickets. If, for example, a device is not working, it's not connecting, or a tool is not supporting what we're doing, we will raise a defect ticket. We can indicate the severity and wait for updates from the Perfecto customer solution team.

In terms of objects, we had some challenges initially and we told our point of contact about them. They updated the solution as a result. Also, sometimes we were not able to access iOS devices. Every time we wanted to go to iOS devices, we had to trust the app, otherwise the lab would not be launched. And we had some challenges accessing the iOS app from the Quantum framework. We worked with the Perfecto team and we got solutions to those issues.

That process is ongoing. We ask them about whatever ideas we have for them to improve. One of the ideas I asked about is because there are a lot of things we need to export into ALM Test Lab. It would be great if Perforce could create a plugin that could be added to the Perfecto portal so that we could just provide the ALM URL domain name and password to connect it to Perfecto. Right now, we need to write code to integrate it with ALM and export the results once the execution is done. An integration with Jira would also help.

Positive

I have not used any other tool for a long time.

I wasn't involved in the initial setup but there is some ongoing script maintenance involved.

It's user-friendly, adaptable, extendable, and scalable. We are able to understand everything through the UI. They just need to enhance it further to adopt the kinds of changes we would like to see.

With Perfecto, we are trying to automate our mobile test cases. These are test cases that run on mobile devices including Android and iOS phones, as well as Android tablets and iPads.

We are a financial institution and we rely on dedicated devices within a private cloud. Due to the sensitivity of our data, it is mandated that we use a private cloud.

I am the director of automation and analytics within our digital portfolio. We use Perfecto for the visualization and reporting of test results, including the relevant numbers.

We leverage Perfecto for reporting and analytics, and in that regard, it provides good reporting features. In terms of analytics, Perfecto still doesn't have predictive modeling or similar analytics features in the tool. In terms of the features that it does have, we use them to the fullest.

We make use of the Perfecto Reports dashboard. Within the framework, we use the Allure reports for automation execution.

We are using the Perfecto cloud for both real and virtual devices. For example, we have some cradles for real devices that we have licenses for, we have virtual devices by means of emulators and simulators, and we have browsers. These are the three types of licenses that we have.

The physical devices are sitting in the Perfecto device lab. The emulators and simulators work to simulate Android and iOS devices, and the browsers are separate again. As the main testing platform for our operations, the device lab is very important to us. We used to do everything in-house, which required on-premises infrastructure to do so. We got rid of the VMs and our physical machines, including our Macs. This is, in fact, one of the reasons that we chose Perfecto.

The fact that the device lab provides same-day access to new devices is a good thing for us. We have an SLA for procurement and if we need a new device, our SLA stipulates that Perfecto has a certain number of days to procure it. We are happy because when a new device becomes available, we get instant access. 

Another example of how Perfecto has helped to improve our organization happened during COVID. Sometimes during the pandemic, when we were working away from the office but still had the on-premises infrastructure, somebody would have to travel to the office only to restart a mobile device or a Mac machine. We no longer have to do that and that's a big relief for us.

When it comes to procurement, when Apple or Samsung launches a new device, all we have to do is pass the request to Perfecto. This is an example of where we save on infrastructure but it's not the only advantage. We are able to execute all of our tests in parallel on different cloud devices. Previously, we were challenged in terms of scalability with local infrastructure. Now, scalability is not a challenge and we can add as many virtual devices, running in parallel, as we need. Previously, we were limited to running a few devices in-house and using a few different browsers.

Perfecto has helped to speed up our testing effort. This is true both in terms of time to market and in terms of execution time. 

The most valuable feature is cloud-based automation and testing. 

In terms of Perfecto's ability to perform cross-platform testing, I would rate it a ten out of ten. The first advantage is that we can take the entire execution to the cloud. We don't have to maintain a lab in-house. Second, we can have parallel test execution, which was tricky when we were doing everything in-house. Third, we have better browser automation and device test coverage.

When we maintain a physical inventory in-house, we cannot get all of the operating systems or different types of phones that are available on the market. Even when it comes to browsers, for example, if I have to run multiple versions of different browsers, whether it is Chrome, Edge, Firefox, or Safari, it is much easier when we can take it to the cloud. Perfecto provides that with all of the options for all of the browsers, as well as the real devices.

When we onboarded Perfecto, the test cases and the framework were already designed using open-source technologies. We used TestNG for the framework and Appium for the automation. As Perfecto supports Appium, it was easy for us to implement and we are still running these technologies.

I would like to see the inclusion of machine learning features. If we can have that, it will be a better tool.

On the analytics side, Perfecto is in need of improvement. The product has a cool dashboard, no doubt about that. But that dashboard is limited to visualization and slicing and dicing the numbers, as well as getting the reports in multiple formats. What I would like, beyond that, is if we can have some predictive intelligence capabilities. This includes machine learning and modeling capabilities. I would really like to see that from Perfecto.

As an example, I would like to be able to predict our failure rate in an upcoming release of our software.

The reliability of devices is very good, at perhaps 99.9%. However, sometimes, we find that some devices are offline and not available. I would like them to achieve even closer to 100% reliability. I know that 100% is not achievable but they can still push that number.

The timing of the software is an area that can be improved. For example, there is a new Appium release and Perfecto should be able to get it to us more quickly. They take their own time for internal testing before they roll it out to customers but we would like to use it faster, so it would be helpful if they can increase the speed of that process. It is somewhat related to improving their time to market.

I have been working with Perfecto for more than five years.

The reliability of the access to cloud-based devices is close to ten out of ten. We totally rely on cloud devices and don't have a fallback plan. We don't have any in-house infrastructure anymore, which means that we don't test it on any devices that are on-premises. Everything is done from the cloud. As such, it has to be reliable. Performance-wise, I would rate it a ten out of ten.

Overall, I have good confidence in the stability of the solution.

Using the device lab has helped us to scale our day-to-day testing operations and in turn, it has helped to scale our business overall. 

Using virtual devices, such as the Android emulators and iOS simulators, is a feature that has definitely been a big help. Testing on virtual devices is a major chunk of our test cases, and it's important because we are not deepening any local infrastructure. We are totally reliant on virtual devices.

With respect to how hard it is to scale, we have a certain number of licenses and we've been told that we can scale up to whatever number we want to. Initially, there was a hiccup when we scaled to testing many devices in parallel, but I'm pretty comfortable with it now.

We generally plan to increase our usage of this product. More teams are being onboarded to the cloud and although we haven't chalked out a plan as of it, we do plan on purchasing more licenses. 

The technical support is good. Our service-level agreement states that if we open a ticket, it has to be resolved within a certain number of days. This depends on the criticality or the gravity of the defect. There is a proper process in place for that.

I don't think that the support has slipped on their deliverables, so I would rate it a nine and a half out of ten.

Positive

We did not use another cloud provider prior to Perfecto.

Initially, we were doing our testing in-house. To do this, we were maintaining an inventory of mobile devices. Now, with the use of this tool, we have taken everything to the cloud. That means we are not maintaining the inventory anymore, but we are achieving good results. In addition, we are benefitting through cost savings.

The product is a black box to us, and there is minimal setup. Perfecto set up the cloud and they give us access through a browser and URL. Once we get into their cloud, we can pull the devices and do what we need to do.

As the setup is done on their end, it's very easy for us.

Comparing the cost of the product and licensing to what we get for our investment, of course, there are benefits. There is a measurable ROI and we conducted an analysis. In fact, the anticipated ROI is one of the reasons that we chose Perfecto.

Getting rid of our local infrastructure has translated to cost savings. We got rid of VMs, Mac machines, and other infrastructure that was needed. Of course, we do still maintain some devices in-house, but otherwise, we save the money spent on infrastructure. This is from the perspective of purchasing the infrastructure, as well as paying to maintain it.

Overall, it has definitely translated to cost savings and the reason is multifold. We don't have to have a local infrastructure, saving on procurement, support, and maintenance costs. Also, being able to execute tests in parallel leads to additional cost savings.

The additional coverage we get is for the same cost. Specifically, I used to only test using the Chrome browser, whereas now with Perfecto, I get a lot of options. I can test using Chrome, Edge, Firefox, and Safari. This takes the same amount of time and we incur the same cost.

Pricing is an area where Perfecto can do a little better. When we obtain additional licenses, we enter into negotiations with them. 

For customers like us, who have been with Perfecto for many years, it would be helpful if there were more discounts available. 

Before choosing the product, we did an evaluation and the good part was that, in another area of the company, in the wealth management department, they had onboarded Perfecto before we did. This gave us some insight into the product before we began our proof of concept.

After the PoC was complete, we procured the solution.

To sum up, the advantages that we get from this product start with the infrastructure. We save on procurement, support, and maintenance of it. Next, we were limited to only a few devices that we had in-house. We were also limited to only a few browsers. We now get more coverage on the browsers and we have real devices. Finally, the parallel execution translates to cost savings.

My overall feedback on this product is good. My advice for anybody who is considering it is to evaluate and assess the product, as well as conduct an ROI analysis, before procuring it. Individual needs might be different. You might realize your return on investment after four or five years. Or, the break-even point might be longer for you.

To conclude, this tool is pretty stable and robust, so there should be nothing to worry about on the technical side.

I would rate this solution a nine out of ten.

We need to test a certain bank's application on different devices, mobile and web. We're using Perfecto to set up devices on the cloud, and that way, the whole distributed team can use it and test different devices.

The Perfecto device cloud has helped us to scale our testing operations. We can do much more testing than we did previously. Business has picked up, and the latest version of OSs needs to be tested. That would not have been possible if we had physical devices and had to give them out to the whole team. And that would involve a lot of cost. Having shared devices makes it much easier for the business.

People from all over the world can use it at the same time. It's not for a particular location. That's most valuable. Their cloud-based lab to test across devices, browsers, and OSs is the main part of it. People from offshore and onshore can access the devices. We can open everything on our system so we don't have to go looking for physical devices.

It saves on the cost and effort of having to maintain our own virtual testing environment. Even our onshore team is not in the city that we work in, so that helps a lot. Even if we didn't invest a lot in getting multiple devices, having to share those devices would become a hassle.

And same-day access to new devices is quite important. When OSs are updated or something new comes onto the market, upon our request, we can have new devices installed on the cloud and have access to them. We don't have any wait time and that's great.

Overall, it has helped massively to speed up our testing efforts and not just from a device point of view. Over the last two years, the performance of the Perfecto tool has picked up as well. It's much more real-time now, compared to what it was when I started.

Also, when I started using it, we did not have the web capabilities and we used to test only on mobile. We had Android and iOS, and that was great, but around 2020, they added web functionality as well, with Windows and Mac devices on the Perfecto cloud. It's great having everything together on one platform.

We use Perfecto's reporting and analytics for automation testing. Obviously, we need to see what the results are for whatever tests we perform from an automation perspective. Reporting results to our business team plays a huge role.

One other feature is the screenshots that Perfecto has. If I'm testing something and I want to take a screenshot, it's just a click away. That makes it much easier to record all the results.

There was a discussion about having the capability to export the test results to a certain tool that we use in our project. If that were added it would be great not having to manually take screenshots, put them in a document, and share them on the different test management tools.

I've been using Perfecto since 2018.

Stability-wise, it's fine. There have been no unforeseen outages.

There is a huge possibility to scale it up. It depends on our project environment if we actually need to, but scalability-wise it is quite good.

We have a huge team of around 100 people who are using it, but it's distributed across teams. It's not like everyone uses it at once. We have a certain number of devices available and we are only able to use those devices.

Their technical support is very good. They immediately respond to our queries and try to resolve them in the best way possible. Obviously, there are times when they need more information from us, but when they have the full information, it's quite fast.

Positive

I wasn't involved in the deployment, but there is maintenance required from time to time. That's mostly on a ticket basis. If we need something done, we raise a ticket with Perfecto support and they help us do it. It's not like we get regular maintenance notifications. It's on-demand maintenance.

In terms of Perfecto's effect on our software quality, it's neutral. It does improve our software, but there's no direct impact from Perfecto on our quality.

My advice about Perfecto is "the earlier the better." I haven't looked into competitors of Perfecto, but from my experience, this is one of the best tools.

We have all these devices and we test our mobile applications on them. We also use it for web testing, as a backup, when the other web browser testing tools that we use are down. We have combinations of mobile devices on which we do manual testing and we use it for automation testing as well.

Our organization mostly focuses on the front-end. We're a "digital factory" for all the UX and UI development testing. We design and develop front-end solutions for our bank. The solution is used by testers to test any changes related to mobile. 

Having Perfecto to test all the various changes to our applications on virtual mobile devices allows us to complete our testing and regression testing within the scheduled time. It enables us to do pretty much everything that we could do in the actual devices. It's really helpful in both automation and manual testing.

The Perfecto device lab has helped us scale our testing operations. It has become embedded with the success of our business and our projects. It would cripple our productivity if we didn't have it. We're very dependent on it. Our testing has been done utilizing Perfecto a lot lately.

Because we don't have a lot of physical devices within our team—we probably have one or two—and because our team is composed of about 50 resources who do testing, we cannot send them the devices, now that we're all working virtually. Perfecto is a perfect tool for allowing everyone to work remotely.

We have 42 devices in our Perfecto cloud base and all of them are being utilized by the testers. Our testers can access one of those devices at any time. As soon as that device is available, they can use it. They don't have to go to the office to use it. They can sit in their houses and perform their activities.

It has also helped to speed up our testing efforts because testers don't have to go to an office. However, the actual testing requires the same steps that you would run, and takes the same amount of time, if you were using the actual device. The savings come from having all your documents or screenshots saved by clicking one button. That's something that helps a lot. Also, the flexibility of changing from one device to another, just by clicking a button, is very efficient.

In addition, Perfecto helps affect our product's quality because it mimics real device behavior; everything is similar. Productivity-wise, we're able to test all the features and we're able to see what we need to see. For example, we can see the behavior of our application on every device due to the screen resolution of the device. If it's an iPad, the size of the screen is different than if you're using iPhone 12 or 13 Mini. When you go to Android devices, it's also different. Having all of these devices inside our cloud within the Perfecto tools, allows us to perform testing and deliver a better quality product.

The mobile testing is the most valuable feature. Having the devices available virtually to test for all the changes we have enables our team to function. We're all working virtually, as my team is spread out globally. I have team members in India, Charlotte, and Toronto, as well as in Dublin. It would be impossible for all of them to have the actual devices to test all the changes, developments, and new applications.

In recent days I have been exploring the license keys we have with Perfecto and I found out that we have five license keys for web browser testing. That's when I started exploring that area and I'm planning to use it a lot for web browser as well.

In terms of cross-platform testing, they offer all of it, every device available in the market. It covers real scenarios that mimic production so that we don't miss out on any devices that our clients might be using to run the applications we develop. It's been great and very helpful.

The cloud-based lab, with all the devices that we need, is very important. If Perfecto were to go down, it would block our testing, as all testing of our mobile-related changes is done in Perfecto. That means it enables us to complete the tasks assigned to our team. It's 100 percent important for us.

Also, having access to new devices same-day enables the testers to test them right away. That makes everyone productive and efficient.

Because I'm a project manager, I don't use Perfecto’s reporting and analytics, but our business support manager presents me reports where I can see how many devices were used or how much testing was done. Our technical managers use the reporting and I know it has been a very useful feature of Perfecto.

And having the Perfecto tools accessible from a browser is convenient, efficient, and fast. It's easy to go to the browser and launch the application, log in with your account, and you should be good to go. You just select the device that you actually wanted to perform your testing on.

I'm hoping that Perfecto will come up with browser testing as well, because it would be easier to access it.

I've been a user of Perfecto for quite a while now. Even before my current placement, I was a tester elsewhere as well. I have used it for around five years. Now, I don't use it day-to-day, my team uses it. I'm still a user because I do admin stuff and maintain all the devices in our cloud and do acquisitions if we need more devices.

It's very stable. Even before I worked in my current company, I had never heard of it being down. It's always been there and available to use.

It covers device availability. Every time there are new devices in the market, they have it in their cloud. It depends on us if we want to purchase them. 

We're planning to increase the number of devices we have because there are always new devices on the market. We'll have to take a look at the devices that we may not need and acquire new ones, to make sure we test all the latest devices. We want to make sure that no users are left out and that our projects are up to date with the new devices.

We have 500 users signed up in Perfecto, including developers, and I'm sure they're using every different feature available in the solution. It's used by about 20 different teams. It's used for UA testing and regression testing. It's used for every level of testing in our environment.

The technical support has been very helpful. Whenever we raise a ticket from the support website portal, they reach out to us within a couple of hours. That's pretty fast for a turnaround. We don't have to revisit and ask for follow-up.

Positive

I wasn't involved in the initial setup. It was already in place when I arrived.

In terms of maintenance, once in a while we have to do failover testing to make sure we're able to continue business as usual if Perfecto were to go down. Our Perfecto business account manager is always there and she's very helpful. We have another tech guy in their support team who helps us. I and another tester perform the failover testing on the URLs that they gave us, to perform a dry run. The Perfecto team has been very helpful and approachable for any questions that we have had.

I've hardly seen Perfecto down, so far. You only lose the benefit of it when you're not able to access it. So the return, for sure, is 100 percent.

I got a quote for one device the other day. It was $1,300 for two years. That's a little expensive, but it's okay.

There are no additional costs. Support comes with it by default, other than if you need additional devices. Then you have to have that device credit, which is the $1,300 I mentioned.

It's a great tool to use if your work involves mobile development. It replaces actual devices with ease of access. It's very convenient to use. It's very user-friendly and that's something that I like about, for a user, a tester, or an admin like myself. It's very straightforward. You don't need a lot of automation knowledge to do testing. From manual testing to automation, and even for developers, it's an all-in-one package if you're looking for a mobile testing tool.

I love the idea that they have a one-stop shop for Perfecto and BlazeMeter. I saw that presentation yesterday. However, my portfolio doesn't deal with backend stuff.

Perfecto is part of our testing life. When we test, we do it in Perfecto. Without it, our team would not be able to perform testing on mobile devices. It's becoming the "backup app" for our web browser testing tools. It's the life of our testing. It's part of it. It's embedded in our testing life cycle. It's the tool that we use whenever functional testing, usability testing, and regression testing are needed. I don't hear of any other tools that we use for testing mobile changes.

On the mobile side, I'm amazed at the solution. I'm amazed by how they actually came up with the idea and all that it can do for us. It has been good. It helps us a lot in every possible way that we need it. I love Perfecto. I would recommend it, always.

We use Perfecto for our mobile testing; both real devices and virtual devices. We also use it in web online test cases, for testing the UI.

Overall, Perfecto has significantly helped us because we are not dependent on maintaining the infrastructure for various devices. Everything is maintained by Perfecto, which means that we can focus on testing.

We leverage Perfecto's reporting and analytics using the CI dashboard, reporting dashboard, and the live stream view. This functionality is quite important because it gives us the history of our past test cases and failures. From there, we can determine at which build a particular feature starts to fail.

Perfecto gives us the ability to test using virtual devices, and the testing can be done in parallel. This means that we are able to test against approximately 300 different devices.

In terms of scaling our testing operations, the option to use virtual devices is very important. If we didn't have the virtual device option then we wouldn't be able to scale as much, and our test execution time would increase.

When it comes to test creation, Perfecto supports open-source technologies such as selenium and APM. This is very good because initially, we were using most of the open-source technologies that were available. This meant that integrating with Perfecto was very smooth, as they supported what we were using.

The quality of our software has improved since we implemented this solution. Firstly, it helps that Perfecto maintains the hardware, allowing us to focus primarily on our software. Secondly, we have been able to find more defects in our applications.

The most valuable feature is the ability to record the video of every test case that we run during the automation. This means that we can go back at any time and actually see the video where the test passes or fails.

We use Perfecto's cloud-based lab for testing across multiple OS versions. The online banking team uses it to test different Chrome versions on different OS versions, which is something that really helps. It holds all of the emulators and simulators, so it's very important because we depend on those tools for our testing.

Perfecto provides same-day access when new devices are released, which is important because we sometimes make requests for these. For example, when a new iPhone is launched, it is quite important to test. This is true for all of the latest devices.

Using the device lab has definitely helped us to scale our testing operations. At this point, we run tests across 300 devices and this is possible because our test execution time has decreased significantly.

The monitoring features, in particular network traffic monitoring, could be improved.

We started using Perfecto approximately 15 months ago, in March of 2021.

With respect to stability, I would rate it a seven out of ten.

The scalability is good. We have been able to scale up with the introduction of new devices, and there are a lot of them in total.

I don't expect that our usage is going to increase.

The Perfecto technical support is good. We have had to raise multiple tickets for some of the issues that we've faced. They have a decent turnaround time when they provide solutions. Sometimes, they were not able to provide a solution, but the response time was decent.

I would rate the technical support an eight out of ten.

Positive

Prior to Perfecto, we were using in-house solutions such as local simulators and emulators. We did not have another provider.

We switched because we were looking for something that was more flexible than our in-house solution. We wanted a product that didn't require maintaining our infrastructure.

I was involved in onboarding and setting up the framework. In terms of complexity, I would rate it medium. It was not very complex but not entirely straightforward. We just had to make changes in our framework and make sure that we did test executions on the cloud instead of on our local architecture.

Our in-house team was responsible for the deployment.

The only maintenance that is required is for our real devices. In these cases, we have to make sure that the operating systems are up to date, and provide the latest ones that are in production.

The majority of the maintenance is taken care of by Perfecto. We quickly test to make sure that our scripts are not broken. This is a small team that takes care of this.

I am not directly involved in the aspects concerning costs, although if you consider the time it saves us in terms of test execution, this product has helped us.

It is correct to say that Perfecto has sped up our testing efforts. We are now about to test 300 devices in just a couple of hours with our whole regression suite.

My advice for anybody who is evaluating this solution is to make sure that for whatever framework the teams are currently using, they are prepared to support that.

When any team is doing a POC with Perfecto, they should be able to run all of their edge test cases. Do not focus on the usual test cases but rather, what is most different. Edge-level test cases, which are difficult to execute, should be able to run on Perfecto while doing the POC.

I would rate this solution an eight out of ten.