Perfecto Reviews

We use Perfecto for our mobile testing; both real devices and virtual devices. We also use it in web online test cases, for testing the UI.

Overall, Perfecto has significantly helped us because we are not dependent on maintaining the infrastructure for various devices. Everything is maintained by Perfecto, which means that we can focus on testing.

We leverage Perfecto's reporting and analytics using the CI dashboard, reporting dashboard, and the live stream view. This functionality is quite important because it gives us the history of our past test cases and failures. From there, we can determine at which build a particular feature starts to fail.

Perfecto gives us the ability to test using virtual devices, and the testing can be done in parallel. This means that we are able to test against approximately 300 different devices.

In terms of scaling our testing operations, the option to use virtual devices is very important. If we didn't have the virtual device option then we wouldn't be able to scale as much, and our test execution time would increase.

When it comes to test creation, Perfecto supports open-source technologies such as selenium and APM. This is very good because initially, we were using most of the open-source technologies that were available. This meant that integrating with Perfecto was very smooth, as they supported what we were using.

The quality of our software has improved since we implemented this solution. Firstly, it helps that Perfecto maintains the hardware, allowing us to focus primarily on our software. Secondly, we have been able to find more defects in our applications.

The most valuable feature is the ability to record the video of every test case that we run during the automation. This means that we can go back at any time and actually see the video where the test passes or fails.

We use Perfecto's cloud-based lab for testing across multiple OS versions. The online banking team uses it to test different Chrome versions on different OS versions, which is something that really helps. It holds all of the emulators and simulators, so it's very important because we depend on those tools for our testing.

Perfecto provides same-day access when new devices are released, which is important because we sometimes make requests for these. For example, when a new iPhone is launched, it is quite important to test. This is true for all of the latest devices.

Using the device lab has definitely helped us to scale our testing operations. At this point, we run tests across 300 devices and this is possible because our test execution time has decreased significantly.

The monitoring features, in particular network traffic monitoring, could be improved.

We started using Perfecto approximately 15 months ago, in March of 2021.

With respect to stability, I would rate it a seven out of ten.

The scalability is good. We have been able to scale up with the introduction of new devices, and there are a lot of them in total.

I don't expect that our usage is going to increase.

The Perfecto technical support is good. We have had to raise multiple tickets for some of the issues that we've faced. They have a decent turnaround time when they provide solutions. Sometimes, they were not able to provide a solution, but the response time was decent.

I would rate the technical support an eight out of ten.

Positive

Prior to Perfecto, we were using in-house solutions such as local simulators and emulators. We did not have another provider.

We switched because we were looking for something that was more flexible than our in-house solution. We wanted a product that didn't require maintaining our infrastructure.

I was involved in onboarding and setting up the framework. In terms of complexity, I would rate it medium. It was not very complex but not entirely straightforward. We just had to make changes in our framework and make sure that we did test executions on the cloud instead of on our local architecture.

Our in-house team was responsible for the deployment.

The only maintenance that is required is for our real devices. In these cases, we have to make sure that the operating systems are up to date, and provide the latest ones that are in production.

The majority of the maintenance is taken care of by Perfecto. We quickly test to make sure that our scripts are not broken. This is a small team that takes care of this.

I am not directly involved in the aspects concerning costs, although if you consider the time it saves us in terms of test execution, this product has helped us.

It is correct to say that Perfecto has sped up our testing efforts. We are now about to test 300 devices in just a couple of hours with our whole regression suite.

My advice for anybody who is evaluating this solution is to make sure that for whatever framework the teams are currently using, they are prepared to support that.

When any team is doing a POC with Perfecto, they should be able to run all of their edge test cases. Do not focus on the usual test cases but rather, what is most different. Edge-level test cases, which are difficult to execute, should be able to run on Perfecto while doing the POC.

I would rate this solution an eight out of ten.

We have seven servers in this environment running 4,000 different tests, so the load is distributed across these servers. Perfecto is used in this automated testing environment and helps ensure that the client's operations are running smoothly in production every day. We simulate a real person interacting with the app, to guarantee that it will work in the production environment like they need to work for the final customer.

Perfecto helps us sell automation projects to our clients. The main concern is how to buy, maintain, and administer all the devices. Now that people are working remotely, we also need a way to share devices among testing teams.

Perfecto helps our operations in that. Since the start of the pandemic, we haven't been able to work in the office, and we can't send real devices to the testers' houses. Thanks to the Perfecto platform, we can sell automated testing services again, overcoming those problems.

We are testing on the latest, most costly iPhones per example, so Perfecto enables us to test different versions of OS, and devices and to have many tests running in parallel. These features are essential for our clients, who will buy blocks to run the automation or test manually, so they don't need to buy the devices for testing.

Perfecto helps us scale our operations when we need to test multiple kinds of devices simultaneously. Real devices are ideal, but virtual devices are always available. We prefer real ones, but we use virtual devices when real ones are unavailable.

It also saves us money because the cost of each license is 10 times lower than what we would spend to maintain infrastructure here in Brazil. We would need to pay for the devices, buy licenses for the software required to administer the connections remotely, and hire a person specializing in these technologies. Perfecto is a game-changer for us, we save at least $10,000 monthly.

Perfecto speeds up our testing by reducing the setup time. The devices are always there, and it takes just a few minutes to supply access to the analysts. We saw improvements in our operations, and Perfecto has tools to inspect the elements to help us decide the best selector for each interaction on the automation. It saves us a lot of time, I estimate it saves us 14 to 20 hours a week and implementing testing takes half the time with Perfecto. 

It improves the quality of our client's software. I work for a consulting firm specializing in quality assurance software testing. It helps us achieve the testing metrics and execute the test. The performance is always outstanding, so it helps us maintain the highest standards when we test our client's software.

The most valuable feature is the ability to automate testing on real devices. We use Perfecto to test various mobile devices, like phones and tablets. We can use Perfecto to test across multiple platforms. We use it on the website to do manual testing and testing on all of the platforms I've seen before. We also use it through the CI servers, such as Jenkins and Azure DevOps, using authentication tokens to authenticate the process we are running. It also works with mobile operating systems, such as Android and iOS over the web testing tool.

I also like the reporting functions. We are constantly downloading the reports and sharing them with our final customers. They help us understand what kind of bugs are happening through the applications. The recording feature is handy because it lets us see a video of the automation that we run through the pipeline and discover the correct point at which the automation is breaking.

There is a dashboard where we can download those reports and see the results of our latest runs. We can do many kinds of filters and use some custom ones. The dashboard is fantastic compared to other tools I have used before. The analytics are crucial, too. We use QA analytics every day here in our operations. We use this feature to access the devices and see which ones are available for testing. 

The physical devices can be shared on the cloud, but sometimes a device is unavailable because another person already uses it. A tester can log on, select one of the available devices for manual testing, and later use the tools to automate the test. 

Perfecto's support for a broad range of open-source tools is critical because that is what we primarily use. In cases where we are using open-source tools, we use a lot of frameworks that rely on Python and Java. We also use Ruby-based technologies with Selenium or Appium as an intermediary. They always support us when it comes to implementing a new automated testing suite and provide many examples of how we can do it with each framework and language. They have excellent documentation and lots of models on their GitHub page too.

Sometimes, when the automated tests sync up or we have to debug remotely, we cannot interact directly with the device. We can interact with the code in the debugging tool, but we cannot directly click on the element on the screen or send other kinds of inputs to the device. This is possible with other tools. 

Also, it would be nice if there were some kind of API to get a list of available devices. Currently, we have to look at the web interface to see the available devices, but the pipelines can't do it on their own there. We always need to do this manually, so it would be better if this feature was automated.

I've been using Perfecto for about eight or nine months. 

Perfecto is stable. We never had an issue with availability.

The only scalability issue that we have, is that we always need to check manually if the devices are available. Our contract with most clients is for 10 to 12 slots, and it's already available. It still scales enough for our purposes. 

I rate Perfecto support a ten out of ten. They've done more than one call with my team and helped the people here until the problem was solved. After the issue was addressed, they followed up with us about the process to see if the automation was stable. 

They are always available for calls and provide guidance when we implement client services. We work as consultants here, so we use it in various contexts and many different lines of business. I had an issue two months ago when I was trying to set up the Ruby automation to interact with Perfecto, and it had some bug that interrupted the automation to work. They did a small diagnosis and solved the problem in a couple of hours.

Positive

We didn't use any other solution before this one, but we did do some research and testing with other platforms, and Perfecto was selected over the others.

Setting up the Perfecto component was straightforward. We only need to point to the remote URL and supply the authentication code. We have VPS Ubuntu servers and set up automation with Python code. We run it on a pipeline using Jenkins that runs the code over the devices on Perfecto Cloud. 

We can do the deployment of new automation code in an hour or less for both platforms (Android or iOS). We receive the apps from the developers, upload the applications to Perfecto and update the scripts from GitHub on our servers. We're currently setting up the platforms manually, but we plan to automate this kind of deployment.

We handle all the daily maintenance without any help from Perfecto. We mostly upload a new version of the software twice per month and use the automation scripts daily.

An in-house team with some support from the Perfecto team helped with the first operation. Now we can do it all by ourselves.

In terms of value, I rate Perfecto a nine out of ten. It's the best deal that we found. We know that we are getting the best prices from our partnership with Perfecto.

Perfecto's price is excellent compared to other products with similar features. It was the lowest of the three we evaluated. We also established a partnership with Perfecto, so they provide discounts when we sell Perfecto projects and licenses to our customers. 

We've tried Saucelabs and BrowserStack, but the prices were higher. Every dollar counts because the dollar is much stronger than our currency. Price is a decisive factor for Brazilian operations. Perfecto also has local staff in Brazil who can speak Portuguese. They helped us teach our analysts how they use the tool, and most of them don't speak English.

I rate Perfecto a nine out of ten. They always have room for improvement, but Perfecto is our best option compared to other players in the market. If you plan to implement Perfecto, I recommend considering if you constantly need to check whether devices are available. I would reconsider using Perfecto if you need to do this multiple times per day.

If you're comfortable running a test that points to a single device and only looking for it once daily, you can go ahead with Perfecto. This is the only issue that I feel the need to raise with other people.

We are using Perfecto to do remote testing on physical devices. 

The device is quite significant for us because we are part of a telecom company and quite a lot of testing is sensitive. We need to have private cloud hosting as well as public cloud hosting. Perfecto gives us both options, and we are using both private as well as public cloud. When a new feature comes out, we cannot just put it on the public cloud for everyone to test. That is why we use a private cloud. We can access it over the internet anywhere in the world, which is very important for us.

We use it to work across multiple teams and departments.

Perfecto plays a very crucial part in our day-to-day test executions. We do a lot of mobile testing, so Perfecto is very useful for us for either manual testing or automation.

It is very important to us that the lab provides same-day access to new devices. We have a very fast-moving, agile team. Therefore, we need access to the lab all the time.

We have a small development team who uses their emulators for development purposes. For testing, we still rely a lot on the actual devices. However, at times, when we launch something, we do use the virtual devices as well. Virtual devices are important when testing, although they don't give the same level of experience as real devices. I can understand that it isn't possible to purchase every single device all the time, especially when you are testing later versions of Android and don't have a virtual device or any version of iOS. We have a couple of native application teams who deploy new applications on those virtual devices all the time, depending on the requirements.

Perfecto helps a lot with productivity overall. We have a distributed team across different geographics. Because people are spread out across different countries, not everyone works at the same time, but whenever they work, they want access to the devices. The only feasible solution is to look at the cloud. Perfecto has come in really handy since it is always up and active. People can access it in their own time and at their own comfort, i.e., anyplace and anywhere. So, it has definitely helped with our fast, agile deliveries over the course of time.

Any testing is good testing. The more you test, the better software quality gets. It has definitely helped.

Recently, the network testing component where we can use Perfecto on devices has been the most valuable feature. There is very little competition out there in this regard. We use it for 4G and 5G testing of launched 5G and 4G brands. So, the network testing can be accessed from anywhere in the world. That is very valuable for us.

Perfecto is very good when it comes to executing cross-platform testing. We have just gotten started on cross-browser testing. We already use Perfecto for the native-application testing on multiple devices. I think now we also use Perfecto for browser-related testing. So, all the cross-browser testing works well. We got a couple of licenses that we use.

We can mostly see usage over time using the device logs. This is what matters the most to us, e.g., what team is using a device for how long.

We use Cypress, and it supports it quite well. We use APM, and it also supports that quite well. We used to use the native quantum framework as well in the past, which was obviously well-supported, but now we have moved onto Cypress and APM. Also, we have Applitools, which is not open source, but a paid proprietary tool.

There could be some improvements done on the interface. At times, there has been a bit of a struggle when finding things on the interface. A UI revamp would be a better option in future. That UI hasn't changed much in a long time, so I think they could just make it a bit better so that people could find stuff easily and intuitively.

We have been using Perfecto for over six years.

It is mostly stable. There could be times when the system or environment is down, then we get notified on those things. They have situations where it is not stable, and we have reached out to the team, saying, "Yeah, it just goes blank at times," or "People are not able to log in." We receive ample support for that. 

Maintenance is required, but we outsource it to a third-party vendor.

The product is pretty scalable. 

Perfecto’s device lab has helped us to scale our operations. We use the public cloud as well as the private cloud. With the public cloud, we can scale up devices very often. Our public cloud is also growing at the moment. This is definitely important to us.

The support that we are getting is really valuable. The technical support is top-notch. There is a dedicated person who takes care of us. We reach out to him, then he reaches out to his team. I would rate the support as seven or eight out of 10.

Positive

Previously, we used to purchase a lot of physical devices for testing. The main constraint was that the team was distributed across the globe. Someone would purchase a solution, then someone else needs to distribute it over to those people, which was a mess. Because Perfecto is on the cloud, we can now have devices in one place and everyone can log into the interface. We can look at their testing or whatever they want to do. That has been a massive update.

I was using Xamarin Test Cloud in the past, which had the same mobile capabilities. We were also using a tool that wasn't as scalable as Perfecto, and that was one of the reasons why we switched.

The initial setup was straightforward. While there is always complexity involved, the rollback mechanism works fine.

Deployment is usually fast. It takes about one night.

Not many people are needed for the deployment. We have a small DevOps team and it takes two to three people tops.

It creates a faster production cycle and is quick to market. Things get deployed earlier because the testing happens on time. We can do a lot of panelization, so a lot of test phases can happen in a panel. People don't have to wait for a device to come to them. They can access multiple devices at the same time and do testing at the same time.

Perfecto has definitely saved us on the costs and efforts of having to maintain our own virtual test environment. We lost about 20 devices in the past to maintenance and audit. That was a massive loss for us, as a company, because we were giving devices to someone, but don't know whether we would get it back or not. Having those virtual labs, we don't need to worry about these kinds of things. We are easily saving $5,000 to $10,000 a month on device costing.

It easily saves us a couple of hours a day, every day, in terms of waiting on the devices, preparing devices, etc.

We are definitely getting benefits out of using Perfecto. That is why we have an ongoing relationship with them.

We use it on the enterprise level. The pricing is reasonable.

We haven't evaluated any other options in five or six years.

Perfecto is doing a very good job. Definitely give it a go. Do some PoCs on your environment with the cloud solution to see if it works for your team. If it does, then definitely give it a go. It is worth the money.

We haven't explored the BlazeMeter component yet. We have had a lot of demos on it and are looking really closely at BlazeMeter.

I would rate Perfecto as seven out of 10. Overall, we are happy, but there is room for improvement.

We started using Perfecto to help us save costs. We gave out devices to a lot of content partners. At one point, we had 25 content partners, mostly in the US, and we gave them devices every year. We wanted a solution that would help us save on shipping devices to the US to our content partners.

In about six years, I've given 25 devices a year to at least 20 vendors. There are thousands of active devices and the roaming costs are mostly from mobile data, voice data, and text messages. Initially, we got into it from a cost-saving perspective, but then moved into doing our own internal testing and device qualification.

We have a hybrid deployment model. It's a public cloud, but we have credentials to log into it, which provides security. The devices are at our head office in our labs, but the web service is at Perfecto's data center. There's a VPN connectivity tunnel between our head office and their data center, and then it goes out to the web. Most people have all their devices at their data centers, but because of the sensitivity of our usage, we don't do that. Most of the devices that we have on the cloud are pre-commercial devices. They're not released to the market yet. We have a hybrid solution because we are working with sensitive NDA devices.

Everything is managed by Perfecto when it comes to the cloud itself.

Our team is using this solution and all of our preferred vendors signed NDAs. There are people in India, the US, and Europe who use this solution. Currently, in my system, I have 250 users, which includes people in marketing and customer service who log into the device before it launches and use it for training.

We also use the system for tutorials. We can take screenshots of new apps or services on the device. If someone wants to know how to turn on or turn off data roaming, we can click on settings and it will link each page as we click.

We have seen the biggest improvement in cost savings. Initially, we save $2.6 million a year just on roaming charges. Our testing is more efficient because we can automate testing. Before, we did everything manually. Three of our partners love it because they don't have to travel to Canada to test on our devices. They can stay where they are and get access to our devices and our footprint, so it has really improved our relationship with our vendors as well.

We also have devices in the US and devices in Europe so our team doesn't have to travel to do roaming testing.

Perfecto helps us save on the cost and effort of having to maintain our own virtual test environment. Saving costs was the key driver in our business case. We also saved time by automating the test cases, which saves us $2.6 million initially a year, including roaming charges and shipping devices to different vendors.

The automation piece is the most valuable feature. Every time we had a new version of either OS or an application, we found that being able to automate the testing across different devices is very valuable.

Perfecto is great at executing cross-platform testing. From a carrier perspective, it's all mobile, but we also have web applications. We used to be able to test on Symbian, Windows OS, Android, and iOS. Today, we can go on a big screen and test on Chrome, Firefox, Explorer, and other browsers.

My team leverages Perfecto's reporting and analytics. When we start stress testing an application, we can look at all the reporting that comes from that, and we can tell at what part of the day the app is performing better.

Our analytics are very important to us. Presentations are important, so it's good to have reports with graphs that show the time of day or how the app was performing for months at a time. We use those reports to screen capture, go with the data, and to show to upper management. We can do a comparison of one version performance to another version of the application or device performance.

We utilize Perfecto's cloud-based lab to test across devices, browsers, and OSs. We are a carrier, so we have our devices with different OS versions of devices. Right now, it's mostly just Apple and Androids. Back in the day, they had many different versions of the OS, but the only other thing on top of that is web-based browsers. We don't test Linux, which is an operating system of Windows.

We don't use Perfecto's cloud. We use our own devices because we test on only our organization's devices. Unless we do some kind of competitive analysis, it's not really a test; it's just a comparison to see how other solutions compare to ours.

The range of open-source technologies that Perfecto supports is great. I have a team of contractors that report to me and do all the testing for our organization, and that team turns over every couple of years. We mostly utilize the in-house automation that is available for anyone who isn't a great programmer.

They have support for all these other drivers, and they have something for non-programmers who want to automate their work, which is amazing.

We don't use Perforce's BlazeMeter with Perfecto. From my perspective, it's not really relevant.

I have used Perfecto for 13 years.

The stability is really reliable. I would rate it as nine out of ten.

The scalability is good. We have been with Perfecto for so long that we have seen different operating systems come and go. When Apple changed and upgraded the operating system and changed a lot of the APIs and drivers, they always found a way, even with Android, to get the solution up and running before launch. They have a really strong partnership with the Android community and the iOS community.

Technical support is amazing. They have 24/7 support. I was amazed that we could have support that doesn't involve calling someone, unless we need to use Webex so someone can walk us through something that broke on the server.

I would rate technical support as eight out of ten.

Positive

We previously used DeviceAnywhere. Mobile Complete was the name of the company, but they were acquired by another company. We used them for four years before we started to use Perfecto.

When we used DeviceAnywhere from Mobile Complete, the time to market was a couple of weeks because they took the device, broke it into pieces, and plugged it into a server. Perfecto's time to market is very fast in comparison. If we need a new device connected to the cloud, it's done in 24 to 48 hours. Perfecto came in at the right time when Android was mostly taking over from all the other proprietary OS devices.

I manage the system myself. It's still at the head office, and the web service was at the head office before they came up with the hybrid solution. With the POC, setup took three months.

All of the maintenance and support is done remotely by Perfecto. When new devices come out, I need to plug it into the system. I have been cross-trained on how to manage plugging in and disconnecting a device. If it freezes or if something goes wrong, I can troubleshoot it on my own.

Perfecto is about 30-40% cheaper than Device Anywhere. That was a big reason why we switched. Perfecto also solves some of the issues that we had with Device Anywhere. We have grown by 100% since we started to use Perfecto, and now we have devices roaming. When we look at the competition, we would still stick with Perfecto.

We also looked at LogMeIn and Communicate, but the solution was too new.

I would rate this solution 10 out of 10.

My advice is to compare this solution to the competitors and put all the features together. My job is to always evaluate and look at other products. I still haven't come across one that beats Perfecto when it comes to a full range of features. I wouldn't say that you should take this product right away and not look at anything else, because that would be a disservice.

Look at other solutions and do a strong RFP. If you compare Perfecto to other solutions and consider all of the features and functionalities, you will always come back to Perfecto.

With Perfecto, we are trying to automate our mobile test cases. These are test cases that run on mobile devices including Android and iOS phones, as well as Android tablets and iPads.

We are a financial institution and we rely on dedicated devices within a private cloud. Due to the sensitivity of our data, it is mandated that we use a private cloud.

I am the director of automation and analytics within our digital portfolio. We use Perfecto for the visualization and reporting of test results, including the relevant numbers.

We leverage Perfecto for reporting and analytics, and in that regard, it provides good reporting features. In terms of analytics, Perfecto still doesn't have predictive modeling or similar analytics features in the tool. In terms of the features that it does have, we use them to the fullest.

We make use of the Perfecto Reports dashboard. Within the framework, we use the Allure reports for automation execution.

We are using the Perfecto cloud for both real and virtual devices. For example, we have some cradles for real devices that we have licenses for, we have virtual devices by means of emulators and simulators, and we have browsers. These are the three types of licenses that we have.

The physical devices are sitting in the Perfecto device lab. The emulators and simulators work to simulate Android and iOS devices, and the browsers are separate again. As the main testing platform for our operations, the device lab is very important to us. We used to do everything in-house, which required on-premises infrastructure to do so. We got rid of the VMs and our physical machines, including our Macs. This is, in fact, one of the reasons that we chose Perfecto.

The fact that the device lab provides same-day access to new devices is a good thing for us. We have an SLA for procurement and if we need a new device, our SLA stipulates that Perfecto has a certain number of days to procure it. We are happy because when a new device becomes available, we get instant access. 

Another example of how Perfecto has helped to improve our organization happened during COVID. Sometimes during the pandemic, when we were working away from the office but still had the on-premises infrastructure, somebody would have to travel to the office only to restart a mobile device or a Mac machine. We no longer have to do that and that's a big relief for us.

When it comes to procurement, when Apple or Samsung launches a new device, all we have to do is pass the request to Perfecto. This is an example of where we save on infrastructure but it's not the only advantage. We are able to execute all of our tests in parallel on different cloud devices. Previously, we were challenged in terms of scalability with local infrastructure. Now, scalability is not a challenge and we can add as many virtual devices, running in parallel, as we need. Previously, we were limited to running a few devices in-house and using a few different browsers.

Perfecto has helped to speed up our testing effort. This is true both in terms of time to market and in terms of execution time. 

The most valuable feature is cloud-based automation and testing. 

In terms of Perfecto's ability to perform cross-platform testing, I would rate it a ten out of ten. The first advantage is that we can take the entire execution to the cloud. We don't have to maintain a lab in-house. Second, we can have parallel test execution, which was tricky when we were doing everything in-house. Third, we have better browser automation and device test coverage.

When we maintain a physical inventory in-house, we cannot get all of the operating systems or different types of phones that are available on the market. Even when it comes to browsers, for example, if I have to run multiple versions of different browsers, whether it is Chrome, Edge, Firefox, or Safari, it is much easier when we can take it to the cloud. Perfecto provides that with all of the options for all of the browsers, as well as the real devices.

When we onboarded Perfecto, the test cases and the framework were already designed using open-source technologies. We used TestNG for the framework and Appium for the automation. As Perfecto supports Appium, it was easy for us to implement and we are still running these technologies.

I would like to see the inclusion of machine learning features. If we can have that, it will be a better tool.

On the analytics side, Perfecto is in need of improvement. The product has a cool dashboard, no doubt about that. But that dashboard is limited to visualization and slicing and dicing the numbers, as well as getting the reports in multiple formats. What I would like, beyond that, is if we can have some predictive intelligence capabilities. This includes machine learning and modeling capabilities. I would really like to see that from Perfecto.

As an example, I would like to be able to predict our failure rate in an upcoming release of our software.

The reliability of devices is very good, at perhaps 99.9%. However, sometimes, we find that some devices are offline and not available. I would like them to achieve even closer to 100% reliability. I know that 100% is not achievable but they can still push that number.

The timing of the software is an area that can be improved. For example, there is a new Appium release and Perfecto should be able to get it to us more quickly. They take their own time for internal testing before they roll it out to customers but we would like to use it faster, so it would be helpful if they can increase the speed of that process. It is somewhat related to improving their time to market.

I have been working with Perfecto for more than five years.

The reliability of the access to cloud-based devices is close to ten out of ten. We totally rely on cloud devices and don't have a fallback plan. We don't have any in-house infrastructure anymore, which means that we don't test it on any devices that are on-premises. Everything is done from the cloud. As such, it has to be reliable. Performance-wise, I would rate it a ten out of ten.

Overall, I have good confidence in the stability of the solution.

Using the device lab has helped us to scale our day-to-day testing operations and in turn, it has helped to scale our business overall. 

Using virtual devices, such as the Android emulators and iOS simulators, is a feature that has definitely been a big help. Testing on virtual devices is a major chunk of our test cases, and it's important because we are not deepening any local infrastructure. We are totally reliant on virtual devices.

With respect to how hard it is to scale, we have a certain number of licenses and we've been told that we can scale up to whatever number we want to. Initially, there was a hiccup when we scaled to testing many devices in parallel, but I'm pretty comfortable with it now.

We generally plan to increase our usage of this product. More teams are being onboarded to the cloud and although we haven't chalked out a plan as of it, we do plan on purchasing more licenses. 

The technical support is good. Our service-level agreement states that if we open a ticket, it has to be resolved within a certain number of days. This depends on the criticality or the gravity of the defect. There is a proper process in place for that.

I don't think that the support has slipped on their deliverables, so I would rate it a nine and a half out of ten.

Positive

We did not use another cloud provider prior to Perfecto.

Initially, we were doing our testing in-house. To do this, we were maintaining an inventory of mobile devices. Now, with the use of this tool, we have taken everything to the cloud. That means we are not maintaining the inventory anymore, but we are achieving good results. In addition, we are benefitting through cost savings.

The product is a black box to us, and there is minimal setup. Perfecto set up the cloud and they give us access through a browser and URL. Once we get into their cloud, we can pull the devices and do what we need to do.

As the setup is done on their end, it's very easy for us.

Comparing the cost of the product and licensing to what we get for our investment, of course, there are benefits. There is a measurable ROI and we conducted an analysis. In fact, the anticipated ROI is one of the reasons that we chose Perfecto.

Getting rid of our local infrastructure has translated to cost savings. We got rid of VMs, Mac machines, and other infrastructure that was needed. Of course, we do still maintain some devices in-house, but otherwise, we save the money spent on infrastructure. This is from the perspective of purchasing the infrastructure, as well as paying to maintain it.

Overall, it has definitely translated to cost savings and the reason is multifold. We don't have to have a local infrastructure, saving on procurement, support, and maintenance costs. Also, being able to execute tests in parallel leads to additional cost savings.

The additional coverage we get is for the same cost. Specifically, I used to only test using the Chrome browser, whereas now with Perfecto, I get a lot of options. I can test using Chrome, Edge, Firefox, and Safari. This takes the same amount of time and we incur the same cost.

Pricing is an area where Perfecto can do a little better. When we obtain additional licenses, we enter into negotiations with them. 

For customers like us, who have been with Perfecto for many years, it would be helpful if there were more discounts available. 

Before choosing the product, we did an evaluation and the good part was that, in another area of the company, in the wealth management department, they had onboarded Perfecto before we did. This gave us some insight into the product before we began our proof of concept.

After the PoC was complete, we procured the solution.

To sum up, the advantages that we get from this product start with the infrastructure. We save on procurement, support, and maintenance of it. Next, we were limited to only a few devices that we had in-house. We were also limited to only a few browsers. We now get more coverage on the browsers and we have real devices. Finally, the parallel execution translates to cost savings.

My overall feedback on this product is good. My advice for anybody who is considering it is to evaluate and assess the product, as well as conduct an ROI analysis, before procuring it. Individual needs might be different. You might realize your return on investment after four or five years. Or, the break-even point might be longer for you.

To conclude, this tool is pretty stable and robust, so there should be nothing to worry about on the technical side.

I would rate this solution a nine out of ten.

We have all these devices and we test our mobile applications on them. We also use it for web testing, as a backup, when the other web browser testing tools that we use are down. We have combinations of mobile devices on which we do manual testing and we use it for automation testing as well.

Our organization mostly focuses on the front-end. We're a "digital factory" for all the UX and UI development testing. We design and develop front-end solutions for our bank. The solution is used by testers to test any changes related to mobile. 

Having Perfecto to test all the various changes to our applications on virtual mobile devices allows us to complete our testing and regression testing within the scheduled time. It enables us to do pretty much everything that we could do in the actual devices. It's really helpful in both automation and manual testing.

The Perfecto device lab has helped us scale our testing operations. It has become embedded with the success of our business and our projects. It would cripple our productivity if we didn't have it. We're very dependent on it. Our testing has been done utilizing Perfecto a lot lately.

Because we don't have a lot of physical devices within our team—we probably have one or two—and because our team is composed of about 50 resources who do testing, we cannot send them the devices, now that we're all working virtually. Perfecto is a perfect tool for allowing everyone to work remotely.

We have 42 devices in our Perfecto cloud base and all of them are being utilized by the testers. Our testers can access one of those devices at any time. As soon as that device is available, they can use it. They don't have to go to the office to use it. They can sit in their houses and perform their activities.

It has also helped to speed up our testing efforts because testers don't have to go to an office. However, the actual testing requires the same steps that you would run, and takes the same amount of time, if you were using the actual device. The savings come from having all your documents or screenshots saved by clicking one button. That's something that helps a lot. Also, the flexibility of changing from one device to another, just by clicking a button, is very efficient.

In addition, Perfecto helps affect our product's quality because it mimics real device behavior; everything is similar. Productivity-wise, we're able to test all the features and we're able to see what we need to see. For example, we can see the behavior of our application on every device due to the screen resolution of the device. If it's an iPad, the size of the screen is different than if you're using iPhone 12 or 13 Mini. When you go to Android devices, it's also different. Having all of these devices inside our cloud within the Perfecto tools, allows us to perform testing and deliver a better quality product.

The mobile testing is the most valuable feature. Having the devices available virtually to test for all the changes we have enables our team to function. We're all working virtually, as my team is spread out globally. I have team members in India, Charlotte, and Toronto, as well as in Dublin. It would be impossible for all of them to have the actual devices to test all the changes, developments, and new applications.

In recent days I have been exploring the license keys we have with Perfecto and I found out that we have five license keys for web browser testing. That's when I started exploring that area and I'm planning to use it a lot for web browser as well.

In terms of cross-platform testing, they offer all of it, every device available in the market. It covers real scenarios that mimic production so that we don't miss out on any devices that our clients might be using to run the applications we develop. It's been great and very helpful.

The cloud-based lab, with all the devices that we need, is very important. If Perfecto were to go down, it would block our testing, as all testing of our mobile-related changes is done in Perfecto. That means it enables us to complete the tasks assigned to our team. It's 100 percent important for us.

Also, having access to new devices same-day enables the testers to test them right away. That makes everyone productive and efficient.

Because I'm a project manager, I don't use Perfecto’s reporting and analytics, but our business support manager presents me reports where I can see how many devices were used or how much testing was done. Our technical managers use the reporting and I know it has been a very useful feature of Perfecto.

And having the Perfecto tools accessible from a browser is convenient, efficient, and fast. It's easy to go to the browser and launch the application, log in with your account, and you should be good to go. You just select the device that you actually wanted to perform your testing on.

I'm hoping that Perfecto will come up with browser testing as well, because it would be easier to access it.

I've been a user of Perfecto for quite a while now. Even before my current placement, I was a tester elsewhere as well. I have used it for around five years. Now, I don't use it day-to-day, my team uses it. I'm still a user because I do admin stuff and maintain all the devices in our cloud and do acquisitions if we need more devices.

It's very stable. Even before I worked in my current company, I had never heard of it being down. It's always been there and available to use.

It covers device availability. Every time there are new devices in the market, they have it in their cloud. It depends on us if we want to purchase them. 

We're planning to increase the number of devices we have because there are always new devices on the market. We'll have to take a look at the devices that we may not need and acquire new ones, to make sure we test all the latest devices. We want to make sure that no users are left out and that our projects are up to date with the new devices.

We have 500 users signed up in Perfecto, including developers, and I'm sure they're using every different feature available in the solution. It's used by about 20 different teams. It's used for UA testing and regression testing. It's used for every level of testing in our environment.

The technical support has been very helpful. Whenever we raise a ticket from the support website portal, they reach out to us within a couple of hours. That's pretty fast for a turnaround. We don't have to revisit and ask for follow-up.

Positive

I wasn't involved in the initial setup. It was already in place when I arrived.

In terms of maintenance, once in a while we have to do failover testing to make sure we're able to continue business as usual if Perfecto were to go down. Our Perfecto business account manager is always there and she's very helpful. We have another tech guy in their support team who helps us. I and another tester perform the failover testing on the URLs that they gave us, to perform a dry run. The Perfecto team has been very helpful and approachable for any questions that we have had.

I've hardly seen Perfecto down, so far. You only lose the benefit of it when you're not able to access it. So the return, for sure, is 100 percent.

I got a quote for one device the other day. It was $1,300 for two years. That's a little expensive, but it's okay.

There are no additional costs. Support comes with it by default, other than if you need additional devices. Then you have to have that device credit, which is the $1,300 I mentioned.

It's a great tool to use if your work involves mobile development. It replaces actual devices with ease of access. It's very convenient to use. It's very user-friendly and that's something that I like about, for a user, a tester, or an admin like myself. It's very straightforward. You don't need a lot of automation knowledge to do testing. From manual testing to automation, and even for developers, it's an all-in-one package if you're looking for a mobile testing tool.

I love the idea that they have a one-stop shop for Perfecto and BlazeMeter. I saw that presentation yesterday. However, my portfolio doesn't deal with backend stuff.

Perfecto is part of our testing life. When we test, we do it in Perfecto. Without it, our team would not be able to perform testing on mobile devices. It's becoming the "backup app" for our web browser testing tools. It's the life of our testing. It's part of it. It's embedded in our testing life cycle. It's the tool that we use whenever functional testing, usability testing, and regression testing are needed. I don't hear of any other tools that we use for testing mobile changes.

On the mobile side, I'm amazed at the solution. I'm amazed by how they actually came up with the idea and all that it can do for us. It has been good. It helps us a lot in every possible way that we need it. I love Perfecto. I would recommend it, always.

We are mainly using it for automated testing for some devices that we're not able to have physically. We are also using the Perfecto service to do manual testing.

We deploy builds daily, and we have an automated pipeline that goes directly to Perfecto. We do it on a daily basis.

If anyone who doesn't have company-issued devices wants to do testing of our mobile application, we can always provide them access to Perfecto, and they can play around with the app there. That has been really nice.

We recently set up a couple of test cases for automated testing. Being able to run them automatically and being able to see the success and failure rate of those runs every day has saved us a lot of time. It has also been helpful in identifying issues, and it has also been helpful in terms of the timeframe and being able to find them a lot sooner.

It works well when it comes to executing cross-platform testing. I would give it a positive rating in that regard. We're able to test on Android and iOS devices pretty seamlessly.

We are utilizing Perfecto's cloud-based lab to test across devices, browsers, and/or OSs. The device lab is very important for our testing operations. If we have issues with the devices being offline, we wouldn't be able to run our automated tests. 

Perfecto's device lab has helped us to scale our testing operations. Currently, our company doesn't issue Android devices to us. I have an iOS device that's company-issued on which I can do manual testing. For Android testing, we've been leveraging the Android device in Perfecto. It is pretty important, and it has been helping us to determine what our baseline is with the Android builds. Because we're not able to physically see it, having it in Perfecto is super helpful.

It has saved us the cost and effort of having to maintain our own virtual test environment. It is hard to quantify in dollars, but in time, it has probably saved up to 100 hours, not more than that. As we build out our test suite more, it would save us even more time. Running those automated tests running as compared to someone manually testing the application and finding issues would definitely save time. Being able to find issues a lot quicker results in us being able to fix them a lot quicker.

It has helped to speed up our testing efforts, and it has also affected our software quality. It has improved the quality of our software due to us being able to find and fix the bugs a lot sooner.

Manual testing is definitely valuable. The insights feature is also valuable. I just started looking into that to get more data around the results of the testing. I'm looking to explore that one a bit more, but definitely, the manual testing and the insights are the top ones.

Their team is really great to work with. They're very flexible, and they always show care. They prioritize our work, our company, and our working relationship. I appreciate the ad hoc sessions that they schedule to provide help with troubleshooting, provide the information that we're looking for, or do a demo of a new feature that they have. They're always willing and very quick to get that scheduled for us. I appreciate that a lot.

If we could run an accessibility test in Perfecto against builds, it would help us a lot. Currently, that's a very manual process for us. We haven't found a tool that can do accessibility scans for iOS and doesn't depend on engineering effort. Having a feature related to that would be really awesome for us.

I've been using Perfecto for about a year and a half.

We have had some issues in terms of our environment, and it's hard to say whether it's because of Perfecto, or it is actually our app build that is introducing some of those errors that we have to maintain. So, overall, I would rate Perfecto a seven out of ten because sometimes, we also have lagging with some of the devices. We get error messages, but we're not quite sure what's causing them. However, whenever we do submit tickets for them for support, they're resolved pretty quickly. So, I would rate it a seven out of ten for stability.

We have an end-to-end (E2E) environment that we deploy for testing in Perfecto. Currently, it just supports our product team, which has about 10 people. In the Credit organization or department, we're the only mobile team. The web team, with about 50 people, is evaluating using Perfecto, but currently, our product team is the only one that I'm aware of in our organization. In the main organization, there is another app team that uses Perfecto. They use it a ton more than we do just because their product is much bigger. Their team roughly has 200 people.

There is a ton of potential for scalability because the other team that uses it is huge as compared to ours. They use it pretty heavily. They run thousands of tests a day, and I'm able to see their metrics as well. So, it definitely has a huge capability for scaling.

Their support is amazing. We have a technical consultant from Perfecto with whom we work. Whenever we need help with troubleshooting, he's very responsive over email. He is also very responsive in terms of scheduling a working session with the engineering team. It is easier to troubleshoot that way. Their customer support is great. I would rate them a ten out of ten.

Positive

We didn't have anything else in place before.

I wasn't involved in its deployment. In terms of maintenance, across the enterprise, I'm sure everyone has their own scenarios, but in our product team, we do have to have maintenance. We run into scenarios a lot where there are issues with our end-to-end environment, and we have to invest some time to make sure that everything is running as expected.

It is not a scheduled maintenance per se. Usually, if we have a line of sight to some updates that are coming from Perfecto, we can account for them, but a lot of the time, they are the changes within the app, which as such wouldn't be considered as maintenance. So, we do have to maintain it, but it is not scheduled maintenance, and we do have to put in effort just to make sure everything is running. Usually, just a pair of engineers do that, and it's more on an ad-hoc basis as opposed to on a scheduled basis.

We started seeing its benefits in the last three months. Last year, we were still doing a lot of the initial setup, and we weren't utilizing it as much for testing, but this year, we are getting a lot more usage out of it. It is just because of the way the team has been working. It is not a limitation of the tool.

I am not sure about its pricing, but from our perspective, licensing has been easy. Anytime I have new users or requests for users that want to get added, it's a very simple process. I just give the architectural owner of the product the name and email address, and they're able to easily add a new user. We don't have any issues in regards to getting licenses, but I don't have any insights into pricing.

I personally didn't evaluate other options. This is our enterprise tool. When they selected it, they did evaluate other options, but it was done way before we got the tools. So, I'm not sure what the other options were.

Perfecto has a lot of taped demos and webinars and informative videos to help you discover the features of the tool. They are very good to get more familiar with how to use it and how to find the best features that work for your team. I would also recommend working closely with their account management team. Their customer support has also been very helpful. When we get stuck and we don't really know which direction to go, we have a contact, as opposed to just trying to figure it out ourselves. It is always easier to just talk to someone from their team to get a more clear direction. So, the informational videos that they offer as well as leaning on the actual representative from the company to help you define the best use of the tool for each product are the way to go.

At this time, we are leveraging Perfecto's reporting and analytics very lightly. We're just using the default dashboards that come with the tool. We're just getting into having more useful data and using it more. I expect to be doing more with that. These analytics will be important for us because we'll be able to understand our failure versus pass rate for a lot of the automated testing. We won't be able to understand why those failures are occurring, of course, but at least, it will give us an indication of fixes that we need to address which might have got missed with manual testing.

Perfecto's cloud-based lab provides same-day access to new devices, but it is not too important for us. It depends on what the time window is. Usually, we get it within a week of our original request, and that seems to work fine for us. So far, we haven't had an immediate need to have the device available on the same day.

Perfecto supports a range of open-source technologies, but I don't have enough experience with that. Currently, we're using Appium by default, and I don't know if that's a limitation of Perfecto or an industry limitation.

I would rate Perfecto an eight out of ten.

We use it to test the mobile apps that we develop.

The Perfecto device lab has allowed us to run tests on multiple devices at the same time and across multiple platforms as well. That increases the efficiency of our testing and saves money, time, and effort. It would take us six times longer to do what we now do.

We are also able to run tests without needing to monitor them. It delivers comprehensive testing that we otherwise wouldn't be able to achieve.

In addition, it has allowed us to maintain high app quality. By testing the app thoroughly, which Perfecto allows us to do, we get that high quality.

The automated test reporting functionality is the most valuable feature. We use the CI Dashboard. It's very important as it is the main reporting tool for our automated tests.

Another important feature is being able to manually control a device.

When it comes to executing cross-platform testing, we only use two platforms, but it's still pretty good.

We also utilize Perfecto’s cloud-based lab to test across devices, browsers, and OSs, and that is really important as well. The lab provides us with same-day access to devices.

The flakiness, or the accuracy, of the test execution can be improved. Also, the responsiveness of their cloud lab could be improved as well.

I have been using Perfecto for a year and three months.

It's stable most of the time but we do come across bugs and we have to reach out to support.

It's very scalable, although it's unlikely we will increase our usage of the solution.

It is used across at least 10 teams in our organization. 

We get pretty good response times and I have someone I can contact very quickly if I have a problem.

Positive

We did not have a previous solution

It took us about three months to realize the benefits, from the time of deployment.

It's a pretty fair investment.

It's definitely on the higher end of prices for this type of service.

Look at the cost of the product and check its feasibility in the context of your projects, and what you want to use it for.

The maintenance involved with the solution can be done by one person and the main duty is to ensure that the scripts are running correctly.

We are using it to test the digital application that we are building. It is an insurance application. It is a web application that we test on multiple browsers and mobile devices. It is not a mobile native app.

We are an insurance company. As we expand the feature set or offerings that we have for our customers, we have to ensure that we are catering to the gamut of users. More and more people are becoming digital savvy, and they are using multiple devices to access applications online and on the road. So, we need to test our application on all possible devices that people may use. Even though it is not a mobile application, people do use mobiles for accessing web applications.

We have been using a competing product. We encountered some challenges in that competing solution in the cross-browser area. Perfecto has certainly brought positive change. We no longer have the defects or issues we were encountering earlier. It has been great from that aspect.

We use Perfecto’s cloud-based lab to test across devices, browsers, and OSs. It is very important for us. It saves the effort and the cost of investing in all different devices, which can become obsolete very soon. Continuous maintenance is not needed, and there is a single platform to access everything. All these benefits don't just make it cost-effective but also make things administratively light. It is very useful for us.

They provide same-day access to new devices, but we didn't require that. The majority of our tests focus on the in-market version. So, we haven't used it, but that's a good capability to have.

Perfecto’s device lab has helped us to scale our testing operations. One good thing is that you just need an email address, and you can share it with anyone. It is very prompt.

We are able to offer a quality product that has been tested fully, which improves our customer satisfaction. That is a good thing. It has also reduced our IT infrastructure cost. We don't have to spend a lot on setting up infrastructure, which becomes redundant or obsolete very soon. It helps in offsetting that cost.

It certainly has brought more capabilities to our coordinating teams. We are able to cater to different testing asks from the business and IT teams, and we are able to meet some of the key areas that our enterprise testing strategies are requiring us to do.

It is very important for scaling our testing. Investing in all different devices and then managing them, when it is not your primary job as a company, is not easy. Using virtual devices is certainly something that we can easily do at an enterprise level.

It affects our software quality. Our solutions are now tested for compatibility and accessibility. In the Ontario region where we are operating, accessibility is mandated by the government. It has also made us compliant with the accessibility requirements.

The inventory of browser options with the latest versions is valuable. There are various iOS and Android options. They are useful to us. We are using its cross-browser capability. It is certainly helpful for us. We have CrossBrowser for Mobile. We use Edge as our enterprise browser, but we know people use Chrome, Safari, and other browsers. We test our application through those browsers.

We also like some of the plugins and integrations that Perfecto has.

Going by the dashboard or analytics capabilities that Perfecto or Perforce is looking to offer in its roadmap, it will certainly help if they also cater to executing and enabling decision-making, rather than just focusing on standard testing metrics such as execution, efficiency, and defect rate. These are good metrics, but they don't necessarily enable decision-making for SLTs. Any improvements in the dashboards and reporting tools should focus on metrics or SLAs that can help with decision-making.

My team and I started using Perfecto a couple of months ago. It has been about three months, but different units in our organization have been using Perfecto and various other tools. 

It is on the cloud. So, it is available anytime. Its stability is very good.

It is positive. We haven't noted any challenges so far.

So far, so good. We haven't created an official service ticket yet, but the inquiry and the follow-up seem to be good. 

Positive

We used SmartBear. There were certain challenges that were noted by the users of the tool. The testers on my team and on the vendor's team noted a few challenges. The tool was showing some issues that we couldn't see on the actual mobile device or browser. There were some false red flags that the tool was throwing. Sometimes, the UI controls were not clickable. We had to invest a lot in defect analysis and resolution. So, we started looking at alternatives, and we started to evaluate Perfecto. 

In terms of the overall usage in the testing space, we have very limited use of Perfecto. We still rely on major enterprise tooling sets. As we transform our IT services, at least in the area where we have identified its use, it has brought positive change.

Perfecto hasn't helped to speed up our testing efforts. We were using a similar tool, and the testing effort has been the same as before.

I was not involved in its setup.

It has saved us the costs and efforts of having to maintain our own virtual test environment, but I don't have the metrics. We haven't done an ROI yet. At the end of the year, we might undertake that exercise.

When assessing the cost of our investment in the solution against the benefits we receive from it, we start at an equal playing field because we had another tool previously. However, the cost of investing in this tool to do testing to catch defects before going live would be six times less than the cost of finding defects and fixing them in production. We will use that as a benchmark. We will also consider the effort that it has taken for us to execute test cases specifically on devices and browsers. Those will be the basis for us to calculate the ROI.

Pricing-wise, it is fine. It is not as expensive as what we used to have in the past from HP, IBM, and others. It is decently priced.

It came through recommendations from other teams that were using it. So, we started with a trial. We used BitBar, and we also used CrossBrowser for Mobile.

Even though we are not using a lot of integrations, we do like some of the plugins and integrations that Perfecto has. Essentially, the tool is not integrated into our entire testing life cycle. 

With BitBar, we did not see a direct Jira plugin. So, the scope of use may be limited, or we may have not explored that integration. Perfecto seems to have that integration, and based on the documentation I had from the team, it seems that it is also well integrated with the Jira life cycle, which is valuable to me as a QE Manager.

I would advise specifically identifying the use cases that you want to try with the solution. The range of offerings that Perfecto is planning to bring to the market seems good. There is a lot of innovation happening in the space of test data management and usability and accessibility testing. If you are not looking at spending a lot of licensing money on enterprise tools, Perfecto is certainly good for starting small and then scaling up at the enterprise level. 

Perfecto supports open-source technologies, such as Selenium and Appium among others, but we are not using these capabilities. Based on my theoretical understanding and assessment of other tools, the range seems really impressive. Their roadmap is also impressive. It is good to see the integrations and collaboration it has with most of our testing tools. That's really impressive. 

They are certainly a good provider of services and solutions. We do not use Perforce BlazeMeter with Perfecto, but its collaboration with BlazeMeter excites me. I'm happy about that.

We don't leverage Perfecto's reporting and analytics capability, and we also don't use it for cross-platform testing.

Based on the size of the solution offering and the way they operate, I would rate it a nine out of ten. 

Our use cases are execution on the cloud for web and mobile. With mobile, it could be different platforms, like iOS, and Android. We have functional execution and API test execution. That is primarily what I use Perfecto for. We have a software product that tests the product on different platforms of web and mobile. In an automated fashion, not just manually testing it but also with automating a lot of these scenarios where we execute them, Perfecto runs the execution for us against these different platforms. 

We have 100-plus members registered with Perfecto who use the product at some point. There are multiple deployments across our departments. For example, at the Bank of Montreal, our everyday banking is one application that folks use to do banking. Then, we have other products like stocks, trading, and other products. There are multiple teams who use Perfecto, even though they fall under different suborganizations within the organization. It is open right now for anybody in the company to use it. At this point, there are 100-plus people using it, at least 30-plus teams who may have used it until now, and at least 18 to 20 teams actively using Perfecto within our organization.

Perfecto has definitely helped our organization, mainly from a scaling point of view, with its reporting, and choice of frameworks.

1. It has allowed us to test across different configurations, helping us with our test configurations. 
2. It has helped us with our reporting. Perfecto's reporting allows us to have a clear view regarding what tests have been executed and the health of the application. 
3. It has allowed us to plug in our framework, i.e., the way that we designed it, into Perfecto with modifications. That fluidity and freedom are there for us. We are not tied or reliant on any one particular vendor. We are free to pick and choose with whom we want to work with. 

Perfecto has affected our software quality in a good way. It has allowed us to execute on-demand and on-choice. We also track the number of issues that we find in the product. Every single day, we tag the issues that we found. For example, if something was found by automation, that means it was found by a Perfecto execution. Over time, we realized the real value in tracking those numbers. We can see now that we have clearly been finding issues earlier. It has allowed us to catch our defects earlier, thus improving the quality of our applications.

The most valuable would be their Live Stream analysis, where I can see the live analysis of all the executions on a single device or multiple devices as well as track them. The live analysis and reporting would be the single most valuable feature.

We leverage Perfecto’s reporting and analytics a lot. From the CI Dashboard, it is mainly the status, which is the past, failure count, and time consumption, e.g., how much time did an average test or script take? Along with that, it provides the historical view compared to the previous result, e.g., am I a pass or fail? Also, the stack trace is very important. Whenever a pass occurs, we don't look beyond that. However, whenever a failure occurs, the stack trace information that it gives us is pretty critical for us when figuring out where failures lie. It gives a summary for the pass/fail count, total test count, the historical view, time consumption for each test as well as the total tests, and the stack rate of the failure.

Perfecto's analytics are very important since we use them on a daily basis. We run our executions daily. After every execution, we pull information from the Perfecto reporting system and share that with our stakeholders. Having this information accurately reported is pretty important for us, so everybody is aware of the current status of the product. That way, we can evaluate the health of the product or environment against that which has been executed. Therefore, it helps make those real-time decisions and highlights the impact to the business.

I found Perfecto to be pretty easy to use while executing against cross-platforms. The main reason is because the same script or test automation where we execute on multiple platforms has minimal changes that I need to do. Also, it is easy for me to set up an execution on one platform, then on another platform, either in parallel or one after the other. Parallel opportunities save me time. Once the execution has been completed across these different configurations, I can always check and compare, e.g., what are the differences and consistencies? 

We utilize Perfecto’s cloud-based lab to test across devices, browsers, and OSs. I use it occasionally for manual testing. Though, there are other team members who use it more frequently than I do. I use it mainly for executing my automated tests. We have the Perfecto lab, cloud devices, and machines. I can program my test to execute against any of those devices, which gives me more confidence in my product. I can compare and see how my product or application functionally behaves across these different devices and from a UI point of view, which helps me a lot.

The device lab is extremely important to our testing operations. We rely on having multiple devices up and running all the time. Whenever we kick off an execution, there are multiple reasons why executions may get triggered: 

1. CodeCommit
2. A scheduled job.
3. Might be on-demand by any stakeholder. 

We need the lab to be available, as we need devices up and running for executions to take place. Also, the devices help since they allow us to have parallel execution, and not just wait for a sequential device to become free and available. Therefore, volume is definitely key. It also gives us an opportunity to compare execution across platforms in that space.

It is extremely important to you that the lab provides same-day access to new devices since we analyze that data every single day after execution. 

Perfecto provides their own framework called Quantum Framework. That is one option. The other option is, if I want to have my own framework, I can have a Java-based Maven project, take a Selenium library, AppiumLibrary, and REST Assured library, and utilize the open-source framework. It is easy for us to connect to Perfecto, no matter what framework we use, as long as it has these core libraries in it. I can design and structure it any way that I want. The execution will happen in Perfecto no matter what since they have support for these tools or libraries. It is pretty neat that way. We are not dependent on using just one particular framework to use Perfecto. While there are still some framework limitations, there is the opportunity to use multiple, different open-source frameworks, then pass the execution to Perfecto. We can use most frameworks, then design and craft it any way that we want, then just pass the execution to Perfecto.

One improvement would be speed of execution. If it is an iOS native app, we have noticed that the speed is a bit slower. Perfecto might need to make some improvements in this area. 

I feel that some of the Perfecto reporting might require upgrades or enhancements to their APIs when pulling some numbers or stats. Perfecto reporting also requires upgrades, mainly from an API point of view.

I have been using it for five years.

I am happy with its stability. I have noticed that in the past five years or so, Perfecto has been quite stable. It is always up and available for us whenever we execute. They notify us, well in advance, if there will be any downtime or maintenance activity on their end. We have a Perfecto representative on-premises at all times, so information is funneled in well in advance about those maintenance windows. From an uptime perspective, it is pretty stable.

If you are using Perfecto, the automation solution requires some maintenance. However, just to use Perfecto, there is bare-minimum maintenance. Mainly, it is to check that our networking policies and firewall rules have not changed, and also to determine if we can still access the cloud from our company network. 

I am not sure how much of an effort it is from the Perfecto side to maintain their remote devices. We don't get involved with those. However, from an automation point of view, we need to maintain our executions and make sure that our scripts and code are healthy in order for it to work with Perfecto. So, there is minimal to moderate maintenance, not too critical or major.

Perfecto’s device lab has helped us to scale our testing operations, which has improved our organization in multiple ways. Instead of hosting our internal lab, Perfecto allows us to have a one-stop execution model where all teams plug into Perfecto and execute, and the reporting is available for everybody to see. Also, if we want to scale, we can always either acquire more licenses or cradles, then add more devices. As technology changes, or as devices change over the years, we can change them by upgrading our devices or upgrading the software. Maintaining them is not in-house. It is easy for us to contact the Perfecto help desk and ask them, "Hey, can you change a particular setting on that particular device?" or, "Can we upgrade this to a different Android version?", if we cannot do it remotely.

From a resourcing or technology point of view, it is very easy to scale. However, from a licensing point of view, scaling might be a challenge because it is expensive, but other than that, it is technically easy to scale. Licensing, while not a roadblock, is a challenge for the majority of organizations when scaling.

The customer support has improved over the past two years. They have probably remodeled their customer service, changed something, or added more resources. Now, it is easier for us to pinpoint an issue, explain it, and then get resolution for it. 

I would rate the support team as 10 out of 10.

Positive

We previously used our in-house lab, not an external vendor. We switched to Perfecto because it was hard for us to maintain our own lab. Plus, we got other benefits with Perfecto, e.g., we got to know about their Quantum Framework after we partnered with Perfecto. Mainly, we switched because of their remote lab. That was a prime motivator to move to Perfecto, exposing those licenses and configurations of devices to other team members within the organization. Also, having access to Perfecto's documentation allows us to automate tests better.

Although Perfecto is a good product for us to use, it is a bit expensive. It takes management a bit of work to find the appropriate funding for us to keep Perfecto. I imagine there could be some way to make it more accessible. 

When I moved into the organization, we were already finalizing Perfecto. I am pretty sure that there must have been comparisons made with either DeviceAnywhere or Sauce Labs, but I am not aware of what those decisions or comparisons were.

Do a proof of concept based on your organizational needs, not just treat Perfecto as a remote cloud solution. There is more to it. Check if the reporting of Perfecto suits you. Check if the time that it takes to execute on Perfecto is what you really need, e.g., are you able to plug your CI pipelines into Perfecto? 

If you go with Perfecto, use Quantum Framework, which is quite nice. It is quite easy to onboard test automation engineers to this framework. Plus, it is not too rigid. You can refine, maintain it easily, and then design or change it to your needs.

Perfecto’s virtual devices are on our to-do list, probably for this quarter's roadmap.

We don't use BlazeMeter yet.

I would rate this product as nine out of 10.