Perfecto Reviews

Our use cases are execution on the cloud for web and mobile. With mobile, it could be different platforms, like iOS, and Android. We have functional execution and API test execution. That is primarily what I use Perfecto for. We have a software product that tests the product on different platforms of web and mobile. In an automated fashion, not just manually testing it but also with automating a lot of these scenarios where we execute them, Perfecto runs the execution for us against these different platforms. 

We have 100-plus members registered with Perfecto who use the product at some point. There are multiple deployments across our departments. For example, at the Bank of Montreal, our everyday banking is one application that folks use to do banking. Then, we have other products like stocks, trading, and other products. There are multiple teams who use Perfecto, even though they fall under different suborganizations within the organization. It is open right now for anybody in the company to use it. At this point, there are 100-plus people using it, at least 30-plus teams who may have used it until now, and at least 18 to 20 teams actively using Perfecto within our organization.

Perfecto has definitely helped our organization, mainly from a scaling point of view, with its reporting, and choice of frameworks.

1. It has allowed us to test across different configurations, helping us with our test configurations. 
2. It has helped us with our reporting. Perfecto's reporting allows us to have a clear view regarding what tests have been executed and the health of the application. 
3. It has allowed us to plug in our framework, i.e., the way that we designed it, into Perfecto with modifications. That fluidity and freedom are there for us. We are not tied or reliant on any one particular vendor. We are free to pick and choose with whom we want to work with. 

Perfecto has affected our software quality in a good way. It has allowed us to execute on-demand and on-choice. We also track the number of issues that we find in the product. Every single day, we tag the issues that we found. For example, if something was found by automation, that means it was found by a Perfecto execution. Over time, we realized the real value in tracking those numbers. We can see now that we have clearly been finding issues earlier. It has allowed us to catch our defects earlier, thus improving the quality of our applications.

The most valuable would be their Live Stream analysis, where I can see the live analysis of all the executions on a single device or multiple devices as well as track them. The live analysis and reporting would be the single most valuable feature.

We leverage Perfecto’s reporting and analytics a lot. From the CI Dashboard, it is mainly the status, which is the past, failure count, and time consumption, e.g., how much time did an average test or script take? Along with that, it provides the historical view compared to the previous result, e.g., am I a pass or fail? Also, the stack trace is very important. Whenever a pass occurs, we don't look beyond that. However, whenever a failure occurs, the stack trace information that it gives us is pretty critical for us when figuring out where failures lie. It gives a summary for the pass/fail count, total test count, the historical view, time consumption for each test as well as the total tests, and the stack rate of the failure.

Perfecto's analytics are very important since we use them on a daily basis. We run our executions daily. After every execution, we pull information from the Perfecto reporting system and share that with our stakeholders. Having this information accurately reported is pretty important for us, so everybody is aware of the current status of the product. That way, we can evaluate the health of the product or environment against that which has been executed. Therefore, it helps make those real-time decisions and highlights the impact to the business.

I found Perfecto to be pretty easy to use while executing against cross-platforms. The main reason is because the same script or test automation where we execute on multiple platforms has minimal changes that I need to do. Also, it is easy for me to set up an execution on one platform, then on another platform, either in parallel or one after the other. Parallel opportunities save me time. Once the execution has been completed across these different configurations, I can always check and compare, e.g., what are the differences and consistencies? 

We utilize Perfecto’s cloud-based lab to test across devices, browsers, and OSs. I use it occasionally for manual testing. Though, there are other team members who use it more frequently than I do. I use it mainly for executing my automated tests. We have the Perfecto lab, cloud devices, and machines. I can program my test to execute against any of those devices, which gives me more confidence in my product. I can compare and see how my product or application functionally behaves across these different devices and from a UI point of view, which helps me a lot.

The device lab is extremely important to our testing operations. We rely on having multiple devices up and running all the time. Whenever we kick off an execution, there are multiple reasons why executions may get triggered: 

1. CodeCommit
2. A scheduled job.
3. Might be on-demand by any stakeholder. 

We need the lab to be available, as we need devices up and running for executions to take place. Also, the devices help since they allow us to have parallel execution, and not just wait for a sequential device to become free and available. Therefore, volume is definitely key. It also gives us an opportunity to compare execution across platforms in that space.

It is extremely important to you that the lab provides same-day access to new devices since we analyze that data every single day after execution. 

Perfecto provides their own framework called Quantum Framework. That is one option. The other option is, if I want to have my own framework, I can have a Java-based Maven project, take a Selenium library, AppiumLibrary, and REST Assured library, and utilize the open-source framework. It is easy for us to connect to Perfecto, no matter what framework we use, as long as it has these core libraries in it. I can design and structure it any way that I want. The execution will happen in Perfecto no matter what since they have support for these tools or libraries. It is pretty neat that way. We are not dependent on using just one particular framework to use Perfecto. While there are still some framework limitations, there is the opportunity to use multiple, different open-source frameworks, then pass the execution to Perfecto. We can use most frameworks, then design and craft it any way that we want, then just pass the execution to Perfecto.

One improvement would be speed of execution. If it is an iOS native app, we have noticed that the speed is a bit slower. Perfecto might need to make some improvements in this area. 

I feel that some of the Perfecto reporting might require upgrades or enhancements to their APIs when pulling some numbers or stats. Perfecto reporting also requires upgrades, mainly from an API point of view.

I have been using it for five years.

I am happy with its stability. I have noticed that in the past five years or so, Perfecto has been quite stable. It is always up and available for us whenever we execute. They notify us, well in advance, if there will be any downtime or maintenance activity on their end. We have a Perfecto representative on-premises at all times, so information is funneled in well in advance about those maintenance windows. From an uptime perspective, it is pretty stable.

If you are using Perfecto, the automation solution requires some maintenance. However, just to use Perfecto, there is bare-minimum maintenance. Mainly, it is to check that our networking policies and firewall rules have not changed, and also to determine if we can still access the cloud from our company network. 

I am not sure how much of an effort it is from the Perfecto side to maintain their remote devices. We don't get involved with those. However, from an automation point of view, we need to maintain our executions and make sure that our scripts and code are healthy in order for it to work with Perfecto. So, there is minimal to moderate maintenance, not too critical or major.

Perfecto’s device lab has helped us to scale our testing operations, which has improved our organization in multiple ways. Instead of hosting our internal lab, Perfecto allows us to have a one-stop execution model where all teams plug into Perfecto and execute, and the reporting is available for everybody to see. Also, if we want to scale, we can always either acquire more licenses or cradles, then add more devices. As technology changes, or as devices change over the years, we can change them by upgrading our devices or upgrading the software. Maintaining them is not in-house. It is easy for us to contact the Perfecto help desk and ask them, "Hey, can you change a particular setting on that particular device?" or, "Can we upgrade this to a different Android version?", if we cannot do it remotely.

From a resourcing or technology point of view, it is very easy to scale. However, from a licensing point of view, scaling might be a challenge because it is expensive, but other than that, it is technically easy to scale. Licensing, while not a roadblock, is a challenge for the majority of organizations when scaling.

The customer support has improved over the past two years. They have probably remodeled their customer service, changed something, or added more resources. Now, it is easier for us to pinpoint an issue, explain it, and then get resolution for it. 

I would rate the support team as 10 out of 10.

Positive

We previously used our in-house lab, not an external vendor. We switched to Perfecto because it was hard for us to maintain our own lab. Plus, we got other benefits with Perfecto, e.g., we got to know about their Quantum Framework after we partnered with Perfecto. Mainly, we switched because of their remote lab. That was a prime motivator to move to Perfecto, exposing those licenses and configurations of devices to other team members within the organization. Also, having access to Perfecto's documentation allows us to automate tests better.

Although Perfecto is a good product for us to use, it is a bit expensive. It takes management a bit of work to find the appropriate funding for us to keep Perfecto. I imagine there could be some way to make it more accessible. 

When I moved into the organization, we were already finalizing Perfecto. I am pretty sure that there must have been comparisons made with either DeviceAnywhere or Sauce Labs, but I am not aware of what those decisions or comparisons were.

Do a proof of concept based on your organizational needs, not just treat Perfecto as a remote cloud solution. There is more to it. Check if the reporting of Perfecto suits you. Check if the time that it takes to execute on Perfecto is what you really need, e.g., are you able to plug your CI pipelines into Perfecto? 

If you go with Perfecto, use Quantum Framework, which is quite nice. It is quite easy to onboard test automation engineers to this framework. Plus, it is not too rigid. You can refine, maintain it easily, and then design or change it to your needs.

Perfecto’s virtual devices are on our to-do list, probably for this quarter's roadmap.

We don't use BlazeMeter yet.

I would rate this product as nine out of 10.

Our main use case for Perfecto is for our testers to test their mobile or PC applications.

One of the benefits of Perfecto is that it helps us keep our devices up to date with the right OSs, because we test across different mobile OSs. The solution has been instrumental in helping us to keep our devices at the right levels for our testing.

The solution also helps us to eliminate defects. Any application defect is a bad thing. I'm the "application police." If something is wrong with it, I'm going to give you a ticket and ask you to get it fixed. As a bank, we cannot have defects in the applications and Perfecto is instrumental in that process. That's how it helps us, organization-wide. It really stinks if one of our end-users opens up a mobile app, expecting they can do something but they can't because of a problem with the application. Testing with Perfecto, we find those defects and we get the developers to get them fixed before the end-user sees them.

When we retest and they pass, we're achieving the quality results that we're looking for.

And it has saved us tremendously on the effort and the costs of maintaining our own virtual test environment. We don't have to hire so many personnel to take care of those devices. Perfecto has people employed in its lab to physically work on devices when needed. We would have had to do the same thing if we didn't use the solution. As a high-profile bank, hiring an employee includes doing extensive background checks, and those kinds of things cost the company. Using Perfecto saves time, resources, and it saves on physical intervention with the devices. Overall, it results in huge cost savings for our company. 

There are a whole bunch of things that I like about the solution, but I really love the interaction it has with mobile devices, the testing capabilities, as well as reporting capabilities that we get from the application. The reports are very detailed. They provide all of the information needed for the testers and developers to really look at their applications and know if there are any issues or not.

Perfecto is excellent when it comes to executing cross-platform testing. They offer support, and their support is excellent. They're very responsive. With a lot of those cross-platform applications, we have to work together so that we can set up the configurations between the two interfacing applications, but it's very good.

When it comes to reporting, we take a look at how many users are using Perfecto, and for which devices. That helps us to narrow down exactly which devices are being used the most for the applications our teams are testing. I find the reports and the analytics to be extremely helpful. I cross-reference the user reports with the users' lines of business because it's helpful for us to know which users and which lines of business are using Perfecto.

The most widely-used reports are the individual testers' reports. When they run their automation scripts, it gives them a lot of the details about the test run, such as which components passed or failed. And when something fails, it points that out with an error so that the development and testing teams can look at what that error is pointing to.

And the cloud-based device lab is extremely important to our testing operations. The mobile device testing is one of the main things we use Perfecto for, although we do have a couple of teams that use the PC platforms as well. On a scale of one to 10, the importance of the mobile testing is a nine or 10.

Our testers have other means of testing our PC applications. They can test some of those applications from their local machines. They may or may not need access to Perfecto's PC resources, but for mobile devices, we have users who are offshore, and they do not have access to physical mobile devices. Having Perfecto's cloud-based application, users in India can access the cloud and do their testing. It's extremely important for our organization to be able to provide mobile devices for offshore testing.

We mainly use Appium among the open-source technologies that Perfecto supports. A couple of our PC teams may be using Selenium, but for the most part, the mobile users are using Appium, and the integration between Perfecto and Appium is excellent. If there is any issue for our Appium users, a lot of the time it's not on the Perfecto side, it's on our side; usually configuration issues. 

I can't really say anything negative about Perfect, but if we talk about feature enhancement, there are a couple of things that I'm working on with our customer support manager. I'm hoping they can support on-premises instances. We have been working on a JIRA integration with Perfecto—and I'm extremely impressed that they have that—but at this time they're not supporting onsite JIRA instances, which is what we have.

I love the idea that the testers will be able to run either their manual or their automation scripts, and that the test report will automatically go into that team's JIRA instance. That will save some time. The testers won't have to extract the report, save it, and bring it over to their JIRA instances. Then, when the scrum master reviews the test results, if there are questions, they have all of the details from the report right there.

One other enhancement that I've already spoken with them about is that they don't support Android for ADA (Americans with Disabilities Act) testing. They mentioned that they're going to try to begin supporting that later this year.

I have been using it off and on for at least seven to 10 years.

I support our teams and help them with their test environments and mobile device configurations. It meshes perfectly with my job role and what I like to do. Perfecto is my favorite tool.

I would rate the stability of the solution as an 11 out of 10. There has not been one moment that Perfecto has not been accessible. They schedule their maintenance windows during the weekends, and on most weekends, nobody in our company is working. And even on most weekends the platform is up. 

I never receive an email from a user saying, "Hey, Perfecto is down," or that there's a problem. That is not to say it won't happen, but if it happens we will address it at the time. But since I've been using Perfecto, we haven't had any issues like that.

We have open cradles where we can add devices and shift things around as needed. The scalability is 10 out of 10. All we have to do is specify which devices we want, where we want them, and what OSs we need those devices on.

As a ballpark figure, on both of our clouds, there may be 1,100 to 1,200 users.

We're talking about getting a Perfecto ADA demo scheduled. If our teams like how ADA works for them with Perfecto, I anticipate a lot more usage of the application.

Perfecto's support goes above and beyond, with its automation support. We have biweekly meetings with some of Perfecto's support staff and they help us with automation issues. If there is some problem that Perfecto can help with, they are there. Their support is excellent. They're very friendly and great to work with.

From time to time, we may find a URL that needs to be whitelisted or allowed with Perfecto. The support team has been great as far as making sure that IPs are allowed, and that things are up on the Perfecto side for any integration.

Positive

When I started, we were using an in-house solution. Later, I was moved to Perfecto as the admin, and the more I started using Perfecto, the more I really liked it. Using Perfecto, there weren't as many issues as we saw on the other platform. Perfecto is better than our other tools, and I've been trying my best to move others off of other tools they're using and onto Perfecto.

The switch to Perfecto was an internal process. I was an admin on the other solution, and I had to do all the device upgrades to that platform. But there was a security issue with that process and those upgrades were moved to a secure location within our company. We still have the other solution, but we don't let the word out about it too much.

I was not involved in the initial setup, but I was extremely involved with the configuration to comply with our security requirements.

We're accessing Perfecto through private clouds that they host. We have two instances, enterprise and wholesale, and we configure our users here for activation and utilization in both of those clouds. We have an internal solution in which we add the users to Active Directory groups. If a user leaves the company, they're not able to access Perfecto through our clouds. That setup complies with security policies.

Perfecto is only used in our test environment. We cannot place any actual production data outside of our company so we use fictitious data. It's not data about real people or real accounts. Those devices are configured for our test environment only.

I go beyond what the normal Perfecto admin would do. Normally, the admin would just help facilitate the accounts and get users logged in. But I go a step further, because some of our users are unfamiliar with the environments and how to set things up. I help them with their test environments, on our side, for the mobile devices. That's not something Perfecto support would be able to do.

We had to involve a consultant from Perfecto because certain changes were needed on the Perfecto side, as well as on our side, to comply with our security regulations. That worked great. The consultant was great to work with, and we got everything matched up, configured, and working. He was extremely helpful and instrumental in our success in meeting our security requirements. Our security team is happy.

We have another platform that supports some limited ADA testing, but it has been having intermittent issues. Perfecto offers some ADA support and offers pretty much 100 percent uptime. That saves time because testing with the other platform has to wait until those intermittent issues are resolved. Because Perfecto is up, the testing can go on. The other platform can become unavailable for three or four days and that could be brought down to zero if we can use Perfecto for ADA.

There is another team within our company that went off on their own and got an application called BrowserStack. However, my manager and our team are trying our best to get that team to stop using BrowserStack and over to Perfecto. That would also result in cost savings for our company by saving the cost of that platform. From my comparison of BrowserStack and Perfecto, I like Perfecto a lot better because of the UI, reports, and everything else.

I've seen a lot of new functionality added over the years, a lot of performance and speed enhancements, and I think the product is very good. It's one of the top solutions for device simulation and for users to perform testing. I'm very happy with it and I really like the product a whole lot. It's a very good application.

Perfecto is doing everything that I need it to do, and I enjoy using it. Sure, we have a few open issues, but we're seeing timely updates.

With Perfecto, we are trying to automate our mobile test cases. These are test cases that run on mobile devices including Android and iOS phones, as well as Android tablets and iPads.

We are a financial institution and we rely on dedicated devices within a private cloud. Due to the sensitivity of our data, it is mandated that we use a private cloud.

I am the director of automation and analytics within our digital portfolio. We use Perfecto for the visualization and reporting of test results, including the relevant numbers.

We leverage Perfecto for reporting and analytics, and in that regard, it provides good reporting features. In terms of analytics, Perfecto still doesn't have predictive modeling or similar analytics features in the tool. In terms of the features that it does have, we use them to the fullest.

We make use of the Perfecto Reports dashboard. Within the framework, we use the Allure reports for automation execution.

We are using the Perfecto cloud for both real and virtual devices. For example, we have some cradles for real devices that we have licenses for, we have virtual devices by means of emulators and simulators, and we have browsers. These are the three types of licenses that we have.

The physical devices are sitting in the Perfecto device lab. The emulators and simulators work to simulate Android and iOS devices, and the browsers are separate again. As the main testing platform for our operations, the device lab is very important to us. We used to do everything in-house, which required on-premises infrastructure to do so. We got rid of the VMs and our physical machines, including our Macs. This is, in fact, one of the reasons that we chose Perfecto.

The fact that the device lab provides same-day access to new devices is a good thing for us. We have an SLA for procurement and if we need a new device, our SLA stipulates that Perfecto has a certain number of days to procure it. We are happy because when a new device becomes available, we get instant access. 

Another example of how Perfecto has helped to improve our organization happened during COVID. Sometimes during the pandemic, when we were working away from the office but still had the on-premises infrastructure, somebody would have to travel to the office only to restart a mobile device or a Mac machine. We no longer have to do that and that's a big relief for us.

When it comes to procurement, when Apple or Samsung launches a new device, all we have to do is pass the request to Perfecto. This is an example of where we save on infrastructure but it's not the only advantage. We are able to execute all of our tests in parallel on different cloud devices. Previously, we were challenged in terms of scalability with local infrastructure. Now, scalability is not a challenge and we can add as many virtual devices, running in parallel, as we need. Previously, we were limited to running a few devices in-house and using a few different browsers.

Perfecto has helped to speed up our testing effort. This is true both in terms of time to market and in terms of execution time. 

The most valuable feature is cloud-based automation and testing. 

In terms of Perfecto's ability to perform cross-platform testing, I would rate it a ten out of ten. The first advantage is that we can take the entire execution to the cloud. We don't have to maintain a lab in-house. Second, we can have parallel test execution, which was tricky when we were doing everything in-house. Third, we have better browser automation and device test coverage.

When we maintain a physical inventory in-house, we cannot get all of the operating systems or different types of phones that are available on the market. Even when it comes to browsers, for example, if I have to run multiple versions of different browsers, whether it is Chrome, Edge, Firefox, or Safari, it is much easier when we can take it to the cloud. Perfecto provides that with all of the options for all of the browsers, as well as the real devices.

When we onboarded Perfecto, the test cases and the framework were already designed using open-source technologies. We used TestNG for the framework and Appium for the automation. As Perfecto supports Appium, it was easy for us to implement and we are still running these technologies.

I would like to see the inclusion of machine learning features. If we can have that, it will be a better tool.

On the analytics side, Perfecto is in need of improvement. The product has a cool dashboard, no doubt about that. But that dashboard is limited to visualization and slicing and dicing the numbers, as well as getting the reports in multiple formats. What I would like, beyond that, is if we can have some predictive intelligence capabilities. This includes machine learning and modeling capabilities. I would really like to see that from Perfecto.

As an example, I would like to be able to predict our failure rate in an upcoming release of our software.

The reliability of devices is very good, at perhaps 99.9%. However, sometimes, we find that some devices are offline and not available. I would like them to achieve even closer to 100% reliability. I know that 100% is not achievable but they can still push that number.

The timing of the software is an area that can be improved. For example, there is a new Appium release and Perfecto should be able to get it to us more quickly. They take their own time for internal testing before they roll it out to customers but we would like to use it faster, so it would be helpful if they can increase the speed of that process. It is somewhat related to improving their time to market.

I have been working with Perfecto for more than five years.

The reliability of the access to cloud-based devices is close to ten out of ten. We totally rely on cloud devices and don't have a fallback plan. We don't have any in-house infrastructure anymore, which means that we don't test it on any devices that are on-premises. Everything is done from the cloud. As such, it has to be reliable. Performance-wise, I would rate it a ten out of ten.

Overall, I have good confidence in the stability of the solution.

Using the device lab has helped us to scale our day-to-day testing operations and in turn, it has helped to scale our business overall. 

Using virtual devices, such as the Android emulators and iOS simulators, is a feature that has definitely been a big help. Testing on virtual devices is a major chunk of our test cases, and it's important because we are not deepening any local infrastructure. We are totally reliant on virtual devices.

With respect to how hard it is to scale, we have a certain number of licenses and we've been told that we can scale up to whatever number we want to. Initially, there was a hiccup when we scaled to testing many devices in parallel, but I'm pretty comfortable with it now.

We generally plan to increase our usage of this product. More teams are being onboarded to the cloud and although we haven't chalked out a plan as of it, we do plan on purchasing more licenses. 

The technical support is good. Our service-level agreement states that if we open a ticket, it has to be resolved within a certain number of days. This depends on the criticality or the gravity of the defect. There is a proper process in place for that.

I don't think that the support has slipped on their deliverables, so I would rate it a nine and a half out of ten.

Positive

We did not use another cloud provider prior to Perfecto.

Initially, we were doing our testing in-house. To do this, we were maintaining an inventory of mobile devices. Now, with the use of this tool, we have taken everything to the cloud. That means we are not maintaining the inventory anymore, but we are achieving good results. In addition, we are benefitting through cost savings.

The product is a black box to us, and there is minimal setup. Perfecto set up the cloud and they give us access through a browser and URL. Once we get into their cloud, we can pull the devices and do what we need to do.

As the setup is done on their end, it's very easy for us.

Comparing the cost of the product and licensing to what we get for our investment, of course, there are benefits. There is a measurable ROI and we conducted an analysis. In fact, the anticipated ROI is one of the reasons that we chose Perfecto.

Getting rid of our local infrastructure has translated to cost savings. We got rid of VMs, Mac machines, and other infrastructure that was needed. Of course, we do still maintain some devices in-house, but otherwise, we save the money spent on infrastructure. This is from the perspective of purchasing the infrastructure, as well as paying to maintain it.

Overall, it has definitely translated to cost savings and the reason is multifold. We don't have to have a local infrastructure, saving on procurement, support, and maintenance costs. Also, being able to execute tests in parallel leads to additional cost savings.

The additional coverage we get is for the same cost. Specifically, I used to only test using the Chrome browser, whereas now with Perfecto, I get a lot of options. I can test using Chrome, Edge, Firefox, and Safari. This takes the same amount of time and we incur the same cost.

Pricing is an area where Perfecto can do a little better. When we obtain additional licenses, we enter into negotiations with them. 

For customers like us, who have been with Perfecto for many years, it would be helpful if there were more discounts available. 

Before choosing the product, we did an evaluation and the good part was that, in another area of the company, in the wealth management department, they had onboarded Perfecto before we did. This gave us some insight into the product before we began our proof of concept.

After the PoC was complete, we procured the solution.

To sum up, the advantages that we get from this product start with the infrastructure. We save on procurement, support, and maintenance of it. Next, we were limited to only a few devices that we had in-house. We were also limited to only a few browsers. We now get more coverage on the browsers and we have real devices. Finally, the parallel execution translates to cost savings.

My overall feedback on this product is good. My advice for anybody who is considering it is to evaluate and assess the product, as well as conduct an ROI analysis, before procuring it. Individual needs might be different. You might realize your return on investment after four or five years. Or, the break-even point might be longer for you.

To conclude, this tool is pretty stable and robust, so there should be nothing to worry about on the technical side.

I would rate this solution a nine out of ten.

We are mainly using it for automated testing for some devices that we're not able to have physically. We are also using the Perfecto service to do manual testing.

We deploy builds daily, and we have an automated pipeline that goes directly to Perfecto. We do it on a daily basis.

If anyone who doesn't have company-issued devices wants to do testing of our mobile application, we can always provide them access to Perfecto, and they can play around with the app there. That has been really nice.

We recently set up a couple of test cases for automated testing. Being able to run them automatically and being able to see the success and failure rate of those runs every day has saved us a lot of time. It has also been helpful in identifying issues, and it has also been helpful in terms of the timeframe and being able to find them a lot sooner.

It works well when it comes to executing cross-platform testing. I would give it a positive rating in that regard. We're able to test on Android and iOS devices pretty seamlessly.

We are utilizing Perfecto's cloud-based lab to test across devices, browsers, and/or OSs. The device lab is very important for our testing operations. If we have issues with the devices being offline, we wouldn't be able to run our automated tests. 

Perfecto's device lab has helped us to scale our testing operations. Currently, our company doesn't issue Android devices to us. I have an iOS device that's company-issued on which I can do manual testing. For Android testing, we've been leveraging the Android device in Perfecto. It is pretty important, and it has been helping us to determine what our baseline is with the Android builds. Because we're not able to physically see it, having it in Perfecto is super helpful.

It has saved us the cost and effort of having to maintain our own virtual test environment. It is hard to quantify in dollars, but in time, it has probably saved up to 100 hours, not more than that. As we build out our test suite more, it would save us even more time. Running those automated tests running as compared to someone manually testing the application and finding issues would definitely save time. Being able to find issues a lot quicker results in us being able to fix them a lot quicker.

It has helped to speed up our testing efforts, and it has also affected our software quality. It has improved the quality of our software due to us being able to find and fix the bugs a lot sooner.

Manual testing is definitely valuable. The insights feature is also valuable. I just started looking into that to get more data around the results of the testing. I'm looking to explore that one a bit more, but definitely, the manual testing and the insights are the top ones.

Their team is really great to work with. They're very flexible, and they always show care. They prioritize our work, our company, and our working relationship. I appreciate the ad hoc sessions that they schedule to provide help with troubleshooting, provide the information that we're looking for, or do a demo of a new feature that they have. They're always willing and very quick to get that scheduled for us. I appreciate that a lot.

If we could run an accessibility test in Perfecto against builds, it would help us a lot. Currently, that's a very manual process for us. We haven't found a tool that can do accessibility scans for iOS and doesn't depend on engineering effort. Having a feature related to that would be really awesome for us.

I've been using Perfecto for about a year and a half.

We have had some issues in terms of our environment, and it's hard to say whether it's because of Perfecto, or it is actually our app build that is introducing some of those errors that we have to maintain. So, overall, I would rate Perfecto a seven out of ten because sometimes, we also have lagging with some of the devices. We get error messages, but we're not quite sure what's causing them. However, whenever we do submit tickets for them for support, they're resolved pretty quickly. So, I would rate it a seven out of ten for stability.

We have an end-to-end (E2E) environment that we deploy for testing in Perfecto. Currently, it just supports our product team, which has about 10 people. In the Credit organization or department, we're the only mobile team. The web team, with about 50 people, is evaluating using Perfecto, but currently, our product team is the only one that I'm aware of in our organization. In the main organization, there is another app team that uses Perfecto. They use it a ton more than we do just because their product is much bigger. Their team roughly has 200 people.

There is a ton of potential for scalability because the other team that uses it is huge as compared to ours. They use it pretty heavily. They run thousands of tests a day, and I'm able to see their metrics as well. So, it definitely has a huge capability for scaling.

Their support is amazing. We have a technical consultant from Perfecto with whom we work. Whenever we need help with troubleshooting, he's very responsive over email. He is also very responsive in terms of scheduling a working session with the engineering team. It is easier to troubleshoot that way. Their customer support is great. I would rate them a ten out of ten.

Positive

We didn't have anything else in place before.

I wasn't involved in its deployment. In terms of maintenance, across the enterprise, I'm sure everyone has their own scenarios, but in our product team, we do have to have maintenance. We run into scenarios a lot where there are issues with our end-to-end environment, and we have to invest some time to make sure that everything is running as expected.

It is not a scheduled maintenance per se. Usually, if we have a line of sight to some updates that are coming from Perfecto, we can account for them, but a lot of the time, they are the changes within the app, which as such wouldn't be considered as maintenance. So, we do have to maintain it, but it is not scheduled maintenance, and we do have to put in effort just to make sure everything is running. Usually, just a pair of engineers do that, and it's more on an ad-hoc basis as opposed to on a scheduled basis.

We started seeing its benefits in the last three months. Last year, we were still doing a lot of the initial setup, and we weren't utilizing it as much for testing, but this year, we are getting a lot more usage out of it. It is just because of the way the team has been working. It is not a limitation of the tool.

I am not sure about its pricing, but from our perspective, licensing has been easy. Anytime I have new users or requests for users that want to get added, it's a very simple process. I just give the architectural owner of the product the name and email address, and they're able to easily add a new user. We don't have any issues in regards to getting licenses, but I don't have any insights into pricing.

I personally didn't evaluate other options. This is our enterprise tool. When they selected it, they did evaluate other options, but it was done way before we got the tools. So, I'm not sure what the other options were.

Perfecto has a lot of taped demos and webinars and informative videos to help you discover the features of the tool. They are very good to get more familiar with how to use it and how to find the best features that work for your team. I would also recommend working closely with their account management team. Their customer support has also been very helpful. When we get stuck and we don't really know which direction to go, we have a contact, as opposed to just trying to figure it out ourselves. It is always easier to just talk to someone from their team to get a more clear direction. So, the informational videos that they offer as well as leaning on the actual representative from the company to help you define the best use of the tool for each product are the way to go.

At this time, we are leveraging Perfecto's reporting and analytics very lightly. We're just using the default dashboards that come with the tool. We're just getting into having more useful data and using it more. I expect to be doing more with that. These analytics will be important for us because we'll be able to understand our failure versus pass rate for a lot of the automated testing. We won't be able to understand why those failures are occurring, of course, but at least, it will give us an indication of fixes that we need to address which might have got missed with manual testing.

Perfecto's cloud-based lab provides same-day access to new devices, but it is not too important for us. It depends on what the time window is. Usually, we get it within a week of our original request, and that seems to work fine for us. So far, we haven't had an immediate need to have the device available on the same day.

Perfecto supports a range of open-source technologies, but I don't have enough experience with that. Currently, we're using Appium by default, and I don't know if that's a limitation of Perfecto or an industry limitation.

I would rate Perfecto an eight out of ten.

We have all these devices and we test our mobile applications on them. We also use it for web testing, as a backup, when the other web browser testing tools that we use are down. We have combinations of mobile devices on which we do manual testing and we use it for automation testing as well.

Our organization mostly focuses on the front-end. We're a "digital factory" for all the UX and UI development testing. We design and develop front-end solutions for our bank. The solution is used by testers to test any changes related to mobile. 

Having Perfecto to test all the various changes to our applications on virtual mobile devices allows us to complete our testing and regression testing within the scheduled time. It enables us to do pretty much everything that we could do in the actual devices. It's really helpful in both automation and manual testing.

The Perfecto device lab has helped us scale our testing operations. It has become embedded with the success of our business and our projects. It would cripple our productivity if we didn't have it. We're very dependent on it. Our testing has been done utilizing Perfecto a lot lately.

Because we don't have a lot of physical devices within our team—we probably have one or two—and because our team is composed of about 50 resources who do testing, we cannot send them the devices, now that we're all working virtually. Perfecto is a perfect tool for allowing everyone to work remotely.

We have 42 devices in our Perfecto cloud base and all of them are being utilized by the testers. Our testers can access one of those devices at any time. As soon as that device is available, they can use it. They don't have to go to the office to use it. They can sit in their houses and perform their activities.

It has also helped to speed up our testing efforts because testers don't have to go to an office. However, the actual testing requires the same steps that you would run, and takes the same amount of time, if you were using the actual device. The savings come from having all your documents or screenshots saved by clicking one button. That's something that helps a lot. Also, the flexibility of changing from one device to another, just by clicking a button, is very efficient.

In addition, Perfecto helps affect our product's quality because it mimics real device behavior; everything is similar. Productivity-wise, we're able to test all the features and we're able to see what we need to see. For example, we can see the behavior of our application on every device due to the screen resolution of the device. If it's an iPad, the size of the screen is different than if you're using iPhone 12 or 13 Mini. When you go to Android devices, it's also different. Having all of these devices inside our cloud within the Perfecto tools, allows us to perform testing and deliver a better quality product.

The mobile testing is the most valuable feature. Having the devices available virtually to test for all the changes we have enables our team to function. We're all working virtually, as my team is spread out globally. I have team members in India, Charlotte, and Toronto, as well as in Dublin. It would be impossible for all of them to have the actual devices to test all the changes, developments, and new applications.

In recent days I have been exploring the license keys we have with Perfecto and I found out that we have five license keys for web browser testing. That's when I started exploring that area and I'm planning to use it a lot for web browser as well.

In terms of cross-platform testing, they offer all of it, every device available in the market. It covers real scenarios that mimic production so that we don't miss out on any devices that our clients might be using to run the applications we develop. It's been great and very helpful.

The cloud-based lab, with all the devices that we need, is very important. If Perfecto were to go down, it would block our testing, as all testing of our mobile-related changes is done in Perfecto. That means it enables us to complete the tasks assigned to our team. It's 100 percent important for us.

Also, having access to new devices same-day enables the testers to test them right away. That makes everyone productive and efficient.

Because I'm a project manager, I don't use Perfecto’s reporting and analytics, but our business support manager presents me reports where I can see how many devices were used or how much testing was done. Our technical managers use the reporting and I know it has been a very useful feature of Perfecto.

And having the Perfecto tools accessible from a browser is convenient, efficient, and fast. It's easy to go to the browser and launch the application, log in with your account, and you should be good to go. You just select the device that you actually wanted to perform your testing on.

I'm hoping that Perfecto will come up with browser testing as well, because it would be easier to access it.

I've been a user of Perfecto for quite a while now. Even before my current placement, I was a tester elsewhere as well. I have used it for around five years. Now, I don't use it day-to-day, my team uses it. I'm still a user because I do admin stuff and maintain all the devices in our cloud and do acquisitions if we need more devices.

It's very stable. Even before I worked in my current company, I had never heard of it being down. It's always been there and available to use.

It covers device availability. Every time there are new devices in the market, they have it in their cloud. It depends on us if we want to purchase them. 

We're planning to increase the number of devices we have because there are always new devices on the market. We'll have to take a look at the devices that we may not need and acquire new ones, to make sure we test all the latest devices. We want to make sure that no users are left out and that our projects are up to date with the new devices.

We have 500 users signed up in Perfecto, including developers, and I'm sure they're using every different feature available in the solution. It's used by about 20 different teams. It's used for UA testing and regression testing. It's used for every level of testing in our environment.

The technical support has been very helpful. Whenever we raise a ticket from the support website portal, they reach out to us within a couple of hours. That's pretty fast for a turnaround. We don't have to revisit and ask for follow-up.

Positive

I wasn't involved in the initial setup. It was already in place when I arrived.

In terms of maintenance, once in a while we have to do failover testing to make sure we're able to continue business as usual if Perfecto were to go down. Our Perfecto business account manager is always there and she's very helpful. We have another tech guy in their support team who helps us. I and another tester perform the failover testing on the URLs that they gave us, to perform a dry run. The Perfecto team has been very helpful and approachable for any questions that we have had.

I've hardly seen Perfecto down, so far. You only lose the benefit of it when you're not able to access it. So the return, for sure, is 100 percent.

I got a quote for one device the other day. It was $1,300 for two years. That's a little expensive, but it's okay.

There are no additional costs. Support comes with it by default, other than if you need additional devices. Then you have to have that device credit, which is the $1,300 I mentioned.

It's a great tool to use if your work involves mobile development. It replaces actual devices with ease of access. It's very convenient to use. It's very user-friendly and that's something that I like about, for a user, a tester, or an admin like myself. It's very straightforward. You don't need a lot of automation knowledge to do testing. From manual testing to automation, and even for developers, it's an all-in-one package if you're looking for a mobile testing tool.

I love the idea that they have a one-stop shop for Perfecto and BlazeMeter. I saw that presentation yesterday. However, my portfolio doesn't deal with backend stuff.

Perfecto is part of our testing life. When we test, we do it in Perfecto. Without it, our team would not be able to perform testing on mobile devices. It's becoming the "backup app" for our web browser testing tools. It's the life of our testing. It's part of it. It's embedded in our testing life cycle. It's the tool that we use whenever functional testing, usability testing, and regression testing are needed. I don't hear of any other tools that we use for testing mobile changes.

On the mobile side, I'm amazed at the solution. I'm amazed by how they actually came up with the idea and all that it can do for us. It has been good. It helps us a lot in every possible way that we need it. I love Perfecto. I would recommend it, always.

We have seven servers in this environment running 4,000 different tests, so the load is distributed across these servers. Perfecto is used in this automated testing environment and helps ensure that the client's operations are running smoothly in production every day. We simulate a real person interacting with the app, to guarantee that it will work in the production environment like they need to work for the final customer.

Perfecto helps us sell automation projects to our clients. The main concern is how to buy, maintain, and administer all the devices. Now that people are working remotely, we also need a way to share devices among testing teams.

Perfecto helps our operations in that. Since the start of the pandemic, we haven't been able to work in the office, and we can't send real devices to the testers' houses. Thanks to the Perfecto platform, we can sell automated testing services again, overcoming those problems.

We are testing on the latest, most costly iPhones per example, so Perfecto enables us to test different versions of OS, and devices and to have many tests running in parallel. These features are essential for our clients, who will buy blocks to run the automation or test manually, so they don't need to buy the devices for testing.

Perfecto helps us scale our operations when we need to test multiple kinds of devices simultaneously. Real devices are ideal, but virtual devices are always available. We prefer real ones, but we use virtual devices when real ones are unavailable.

It also saves us money because the cost of each license is 10 times lower than what we would spend to maintain infrastructure here in Brazil. We would need to pay for the devices, buy licenses for the software required to administer the connections remotely, and hire a person specializing in these technologies. Perfecto is a game-changer for us, we save at least $10,000 monthly.

Perfecto speeds up our testing by reducing the setup time. The devices are always there, and it takes just a few minutes to supply access to the analysts. We saw improvements in our operations, and Perfecto has tools to inspect the elements to help us decide the best selector for each interaction on the automation. It saves us a lot of time, I estimate it saves us 14 to 20 hours a week and implementing testing takes half the time with Perfecto. 

It improves the quality of our client's software. I work for a consulting firm specializing in quality assurance software testing. It helps us achieve the testing metrics and execute the test. The performance is always outstanding, so it helps us maintain the highest standards when we test our client's software.

The most valuable feature is the ability to automate testing on real devices. We use Perfecto to test various mobile devices, like phones and tablets. We can use Perfecto to test across multiple platforms. We use it on the website to do manual testing and testing on all of the platforms I've seen before. We also use it through the CI servers, such as Jenkins and Azure DevOps, using authentication tokens to authenticate the process we are running. It also works with mobile operating systems, such as Android and iOS over the web testing tool.

I also like the reporting functions. We are constantly downloading the reports and sharing them with our final customers. They help us understand what kind of bugs are happening through the applications. The recording feature is handy because it lets us see a video of the automation that we run through the pipeline and discover the correct point at which the automation is breaking.

There is a dashboard where we can download those reports and see the results of our latest runs. We can do many kinds of filters and use some custom ones. The dashboard is fantastic compared to other tools I have used before. The analytics are crucial, too. We use QA analytics every day here in our operations. We use this feature to access the devices and see which ones are available for testing. 

The physical devices can be shared on the cloud, but sometimes a device is unavailable because another person already uses it. A tester can log on, select one of the available devices for manual testing, and later use the tools to automate the test. 

Perfecto's support for a broad range of open-source tools is critical because that is what we primarily use. In cases where we are using open-source tools, we use a lot of frameworks that rely on Python and Java. We also use Ruby-based technologies with Selenium or Appium as an intermediary. They always support us when it comes to implementing a new automated testing suite and provide many examples of how we can do it with each framework and language. They have excellent documentation and lots of models on their GitHub page too.

Sometimes, when the automated tests sync up or we have to debug remotely, we cannot interact directly with the device. We can interact with the code in the debugging tool, but we cannot directly click on the element on the screen or send other kinds of inputs to the device. This is possible with other tools. 

Also, it would be nice if there were some kind of API to get a list of available devices. Currently, we have to look at the web interface to see the available devices, but the pipelines can't do it on their own there. We always need to do this manually, so it would be better if this feature was automated.

I've been using Perfecto for about eight or nine months. 

Perfecto is stable. We never had an issue with availability.

The only scalability issue that we have, is that we always need to check manually if the devices are available. Our contract with most clients is for 10 to 12 slots, and it's already available. It still scales enough for our purposes. 

I rate Perfecto support a ten out of ten. They've done more than one call with my team and helped the people here until the problem was solved. After the issue was addressed, they followed up with us about the process to see if the automation was stable. 

They are always available for calls and provide guidance when we implement client services. We work as consultants here, so we use it in various contexts and many different lines of business. I had an issue two months ago when I was trying to set up the Ruby automation to interact with Perfecto, and it had some bug that interrupted the automation to work. They did a small diagnosis and solved the problem in a couple of hours.

Positive

We didn't use any other solution before this one, but we did do some research and testing with other platforms, and Perfecto was selected over the others.

Setting up the Perfecto component was straightforward. We only need to point to the remote URL and supply the authentication code. We have VPS Ubuntu servers and set up automation with Python code. We run it on a pipeline using Jenkins that runs the code over the devices on Perfecto Cloud. 

We can do the deployment of new automation code in an hour or less for both platforms (Android or iOS). We receive the apps from the developers, upload the applications to Perfecto and update the scripts from GitHub on our servers. We're currently setting up the platforms manually, but we plan to automate this kind of deployment.

We handle all the daily maintenance without any help from Perfecto. We mostly upload a new version of the software twice per month and use the automation scripts daily.

An in-house team with some support from the Perfecto team helped with the first operation. Now we can do it all by ourselves.

In terms of value, I rate Perfecto a nine out of ten. It's the best deal that we found. We know that we are getting the best prices from our partnership with Perfecto.

Perfecto's price is excellent compared to other products with similar features. It was the lowest of the three we evaluated. We also established a partnership with Perfecto, so they provide discounts when we sell Perfecto projects and licenses to our customers. 

We've tried Saucelabs and BrowserStack, but the prices were higher. Every dollar counts because the dollar is much stronger than our currency. Price is a decisive factor for Brazilian operations. Perfecto also has local staff in Brazil who can speak Portuguese. They helped us teach our analysts how they use the tool, and most of them don't speak English.

I rate Perfecto a nine out of ten. They always have room for improvement, but Perfecto is our best option compared to other players in the market. If you plan to implement Perfecto, I recommend considering if you constantly need to check whether devices are available. I would reconsider using Perfecto if you need to do this multiple times per day.

If you're comfortable running a test that points to a single device and only looking for it once daily, you can go ahead with Perfecto. This is the only issue that I feel the need to raise with other people.

We started off integrating Robot Framework, which is an automation tool, with Perfecto and mostly we've been using it for web UI automation.

Recently I've joined in with another team member to use it with a Quantum solution provided through Perfecto, instead of Robot Framework. It's like a Java-based implementation. It's already integrated with Perfecto and our use cases for that are web and API.

And, while I'm not involved with mobile, we have projects that are using it for mobile as well.

It's given us some traceability, from a QA perspective, on when certain functionalities are available in online banking and when they are down. 

We have also used it for production automation to replace the manual labor that was done overnight on deployments to production, where someone was having to test them. We offload our scripts to Perfecto. 

And we use it not only for viewing the current execution, but as a historical tool to see our past executions and see when things were working and when they were not and where in the code did something fail. Perfecto is nice because, for Java code, you can see exactly where something failed, at which steps, and it shows you exactly which file actually it failed in. That's really helpful for us in narrowing down where the failure occurred. We know whether to investigate the automation scripts or whether it was a data issue or a defect. Perfecto gives us a lot of visibility into those things.

We are catching issues a lot earlier when it comes to the environment, because if we didn't have the automated jobs running with Perfecto, we would probably let a lot of these environment issues pile up until a lot later. But we're running them daily and seeing where things fail and Perfecto is giving us that video to see exactly where it failed and in which line of code. It's giving us a bit of a bump "left" to see where things fail first. It's definitely earlier than it used to be.

It has also definitely helped to speed up our testing efforts. When the scripts are stable, Perfecto is awesome. If there's not much change, we can run thousands and thousands of test cases every day. Someone from manual testing could execute, maybe, 30 test cases a day. Perfecto can execute, say, 3,000. It's really a huge difference. Part of what I like about Perfecto is that it allows concurrent testing. You can hit multiple browsers or devices at the same time, which makes it a lot more efficient for us to execute a regression.

And in terms of software quality, in one of our online banking projects that we recently delivered to production, we found 300 defects just with automation alone, and we found regression defects in addition. Not all of them could have been caught with manual testing. We improved our application by catching defects earlier, before they went to production, thanks to Perfecto.

The number one feature, which if we didn't have out-of-the-box would be missed, is the fact that we have video execution. That gives us the ability to view errors or defects in the progression, from beginning to the end of the video. That's like a full test case. If we were to go with our automation, we would need to go either with Perfecto or some other vendor because, out-of-the-box, Selenium doesn't come with videos.

Another thing that it supports, besides Selenium and Appium, is that it plugs into Robot Framework. That is really nice because the Perfecto folks have created an integration between Robot Framework and Perfecto. Also, REST Assured is taken into consideration. The only thing is that, with Rest Assured, because there is no browser, you need to specify a connection to Perfecto. You need to call keep-alive to say, "Hey, perfecto, track this." It would be nice if Perfecto could do it out-of-the-box, even though we did it with our code. There are also other REST API libraries for Python, such as a request library in Python.

As a regular user of the solution, day-to-day, I go through the reporting. I view the failures and the summary of test sets and test results. The reporting is a very important feature. There's an API to integrate with it within the code and that's really a plus. It integrates well inside. You can also tag it with which branch you're using from Git, or you can label it with whatever you'd like. It's a very cool feature that they added soon after they had the reporting.

For reporting, I use the CI Dashboard and I use anything to do with ALM because we have our jobs running through ALM for test management. You can report your results there. We also have Robot Framework still running so the dashboards we have are OLBQA56 or OLBQA3. Other than the CI Dashboard, I check out the Report Library quite a bit. I have my own filters on the reports and I'll filter based on the last 24 hours or 12 hours, as well as based on the job name. And, of course, I definitely use the Live Stream to see that my tests are up and running.

Also, Perfecto allows us to run on the cloud. Instead of running on our local machines, we can offload it to a cloud and we can run off servers that are not even local. That helps in case we do restarts. We don't need to worry about our machine restarting. Everything is on the cloud and running robotically.

In terms of cross-platform testing, I have used it mostly for the web and it's pretty strong. It's probably an eight out of 10. The reason it's an eight is that we've had a couple of issues lately with videos not loading or browsers dying after some execution, although that happens very rarely. For mobile, from what I have seen, it's probably also an eight out of 10. 

One thing they need to have is physical devices. And I am hoping in the future that there will be more virtual devices.

Also, there is some error tagging that you can do in Perfecto, but it didn't work very well when I was using it. I wish that you could tag it with an error message per test case and it would be done automatically, instead of manually.

In addition, if Perfecto could use AI to say, "Your environment is stable," or "not stable," and give a score on that, that would be really helpful. It could notify us through email. That would help, aside from the failed and passed test cases. It would be something more intelligent. If there were certain tags with certain errors and there were a certain number of them, it could tell us if login was broken in the environment, for example. There's definitely a little bit of room for improvement in terms of AI.

I've been using Perfecto for about four and a half years.

I would rate the stability at seven or eight out of 10, because there's the occasional video issue or the browser dies.

The scalability of the solution is outstanding. I don't have any concerns with the scalability. As long as we have more licenses, we can scale. We only have 50 licenses at the bank, so sometimes things get pretty much filled up, although not very often. We may actually request more licenses, but we need to discuss that internally first.

They're quite responsive and they do reach out for more information, which is good. Lately they've been coming back with fixes, which I really enjoy. When a company can say, "Hey, it's on us. We're going to fix it," and they come back and they fix it, that really helps. We're in the midst of making sure those fixes are valid and that things are working. I appreciate that they flag those things as something important and try to fix them.

Positive

We were using Jenkins, but we had out-of-the-box browsers. It was in my first year here at BMO we adopted Perfecto.

I wasn't involved in the setup, but I do remember there was some discussion about how to connect to it. Doing so goes out to the web and we have to connect to a proxy. There was a bit of investigation into how to connect to a proxy and how to connect through SSO. It was actually quite a big change for the bank and was flagged as something that required security management to get involved with. They had to give all the permissions.

Qualitatively, we tend to find our defects a lot earlier and we're able to run 100 times the number of test cases that we could do with manual testing, on a daily basis. That saves us a lot of time by having defects caught before we go to production. If customers lose confidence in us as a bank, with online banking, we lose customers. That's definitely something we don't want to do. Perfecto is a big part of our testing and automation and quality, making it possible to have more quality software.

I don't use the Insights Dashboard or the Heatmap. I just didn't really find them useful, although maybe I just need to be instructed how to use them properly.

We started using Perfecto to help us save costs. We gave out devices to a lot of content partners. At one point, we had 25 content partners, mostly in the US, and we gave them devices every year. We wanted a solution that would help us save on shipping devices to the US to our content partners.

In about six years, I've given 25 devices a year to at least 20 vendors. There are thousands of active devices and the roaming costs are mostly from mobile data, voice data, and text messages. Initially, we got into it from a cost-saving perspective, but then moved into doing our own internal testing and device qualification.

We have a hybrid deployment model. It's a public cloud, but we have credentials to log into it, which provides security. The devices are at our head office in our labs, but the web service is at Perfecto's data center. There's a VPN connectivity tunnel between our head office and their data center, and then it goes out to the web. Most people have all their devices at their data centers, but because of the sensitivity of our usage, we don't do that. Most of the devices that we have on the cloud are pre-commercial devices. They're not released to the market yet. We have a hybrid solution because we are working with sensitive NDA devices.

Everything is managed by Perfecto when it comes to the cloud itself.

Our team is using this solution and all of our preferred vendors signed NDAs. There are people in India, the US, and Europe who use this solution. Currently, in my system, I have 250 users, which includes people in marketing and customer service who log into the device before it launches and use it for training.

We also use the system for tutorials. We can take screenshots of new apps or services on the device. If someone wants to know how to turn on or turn off data roaming, we can click on settings and it will link each page as we click.

We have seen the biggest improvement in cost savings. Initially, we save $2.6 million a year just on roaming charges. Our testing is more efficient because we can automate testing. Before, we did everything manually. Three of our partners love it because they don't have to travel to Canada to test on our devices. They can stay where they are and get access to our devices and our footprint, so it has really improved our relationship with our vendors as well.

We also have devices in the US and devices in Europe so our team doesn't have to travel to do roaming testing.

Perfecto helps us save on the cost and effort of having to maintain our own virtual test environment. Saving costs was the key driver in our business case. We also saved time by automating the test cases, which saves us $2.6 million initially a year, including roaming charges and shipping devices to different vendors.

The automation piece is the most valuable feature. Every time we had a new version of either OS or an application, we found that being able to automate the testing across different devices is very valuable.

Perfecto is great at executing cross-platform testing. From a carrier perspective, it's all mobile, but we also have web applications. We used to be able to test on Symbian, Windows OS, Android, and iOS. Today, we can go on a big screen and test on Chrome, Firefox, Explorer, and other browsers.

My team leverages Perfecto's reporting and analytics. When we start stress testing an application, we can look at all the reporting that comes from that, and we can tell at what part of the day the app is performing better.

Our analytics are very important to us. Presentations are important, so it's good to have reports with graphs that show the time of day or how the app was performing for months at a time. We use those reports to screen capture, go with the data, and to show to upper management. We can do a comparison of one version performance to another version of the application or device performance.

We utilize Perfecto's cloud-based lab to test across devices, browsers, and OSs. We are a carrier, so we have our devices with different OS versions of devices. Right now, it's mostly just Apple and Androids. Back in the day, they had many different versions of the OS, but the only other thing on top of that is web-based browsers. We don't test Linux, which is an operating system of Windows.

We don't use Perfecto's cloud. We use our own devices because we test on only our organization's devices. Unless we do some kind of competitive analysis, it's not really a test; it's just a comparison to see how other solutions compare to ours.

The range of open-source technologies that Perfecto supports is great. I have a team of contractors that report to me and do all the testing for our organization, and that team turns over every couple of years. We mostly utilize the in-house automation that is available for anyone who isn't a great programmer.

They have support for all these other drivers, and they have something for non-programmers who want to automate their work, which is amazing.

We don't use Perforce's BlazeMeter with Perfecto. From my perspective, it's not really relevant.

I have used Perfecto for 13 years.

The stability is really reliable. I would rate it as nine out of ten.

The scalability is good. We have been with Perfecto for so long that we have seen different operating systems come and go. When Apple changed and upgraded the operating system and changed a lot of the APIs and drivers, they always found a way, even with Android, to get the solution up and running before launch. They have a really strong partnership with the Android community and the iOS community.

Technical support is amazing. They have 24/7 support. I was amazed that we could have support that doesn't involve calling someone, unless we need to use Webex so someone can walk us through something that broke on the server.

I would rate technical support as eight out of ten.

Positive

We previously used DeviceAnywhere. Mobile Complete was the name of the company, but they were acquired by another company. We used them for four years before we started to use Perfecto.

When we used DeviceAnywhere from Mobile Complete, the time to market was a couple of weeks because they took the device, broke it into pieces, and plugged it into a server. Perfecto's time to market is very fast in comparison. If we need a new device connected to the cloud, it's done in 24 to 48 hours. Perfecto came in at the right time when Android was mostly taking over from all the other proprietary OS devices.

I manage the system myself. It's still at the head office, and the web service was at the head office before they came up with the hybrid solution. With the POC, setup took three months.

All of the maintenance and support is done remotely by Perfecto. When new devices come out, I need to plug it into the system. I have been cross-trained on how to manage plugging in and disconnecting a device. If it freezes or if something goes wrong, I can troubleshoot it on my own.

Perfecto is about 30-40% cheaper than Device Anywhere. That was a big reason why we switched. Perfecto also solves some of the issues that we had with Device Anywhere. We have grown by 100% since we started to use Perfecto, and now we have devices roaming. When we look at the competition, we would still stick with Perfecto.

We also looked at LogMeIn and Communicate, but the solution was too new.

I would rate this solution 10 out of 10.

My advice is to compare this solution to the competitors and put all the features together. My job is to always evaluate and look at other products. I still haven't come across one that beats Perfecto when it comes to a full range of features. I wouldn't say that you should take this product right away and not look at anything else, because that would be a disservice.

Look at other solutions and do a strong RFP. If you compare Perfecto to other solutions and consider all of the features and functionalities, you will always come back to Perfecto.