What is our primary use case?
Our use cases are execution on the cloud for web and mobile. With mobile, it could be different platforms, like iOS, and Android. We have functional execution and API test execution. That is primarily what I use Perfecto for. We have a software product that tests the product on different platforms of web and mobile. In an automated fashion, not just manually testing it but also with automating a lot of these scenarios where we execute them, Perfecto runs the execution for us against these different platforms.
We have 100-plus members registered with Perfecto who use the product at some point. There are multiple deployments across our departments. For example, at the Bank of Montreal, our everyday banking is one application that folks use to do banking. Then, we have other products like stocks, trading, and other products. There are multiple teams who use Perfecto, even though they fall under different suborganizations within the organization. It is open right now for anybody in the company to use it. At this point, there are 100-plus people using it, at least 30-plus teams who may have used it until now, and at least 18 to 20 teams actively using Perfecto within our organization.
How has it helped my organization?
Perfecto has definitely helped our organization, mainly from a scaling point of view, with its reporting, and choice of frameworks.
- It has allowed us to test across different configurations, helping us with our test configurations.
- It has helped us with our reporting. Perfecto's reporting allows us to have a clear view regarding what tests have been executed and the health of the application.
- It has allowed us to plug in our framework, i.e., the way that we designed it, into Perfecto with modifications. That fluidity and freedom are there for us. We are not tied or reliant on any one particular vendor. We are free to pick and choose with whom we want to work with.
Perfecto has affected our software quality in a good way. It has allowed us to execute on-demand and on-choice. We also track the number of issues that we find in the product. Every single day, we tag the issues that we found. For example, if something was found by automation, that means it was found by a Perfecto execution. Over time, we realized the real value in tracking those numbers. We can see now that we have clearly been finding issues earlier. It has allowed us to catch our defects earlier, thus improving the quality of our applications.
What is most valuable?
The most valuable would be their Live Stream analysis, where I can see the live analysis of all the executions on a single device or multiple devices as well as track them. The live analysis and reporting would be the single most valuable feature.
We leverage Perfecto’s reporting and analytics a lot. From the CI Dashboard, it is mainly the status, which is the past, failure count, and time consumption, e.g., how much time did an average test or script take? Along with that, it provides the historical view compared to the previous result, e.g., am I a pass or fail? Also, the stack trace is very important. Whenever a pass occurs, we don't look beyond that. However, whenever a failure occurs, the stack trace information that it gives us is pretty critical for us when figuring out where failures lie. It gives a summary for the pass/fail count, total test count, the historical view, time consumption for each test as well as the total tests, and the stack rate of the failure.
Perfecto's analytics are very important since we use them on a daily basis. We run our executions daily. After every execution, we pull information from the Perfecto reporting system and share that with our stakeholders. Having this information accurately reported is pretty important for us, so everybody is aware of the current status of the product. That way, we can evaluate the health of the product or environment against that which has been executed. Therefore, it helps make those real-time decisions and highlights the impact to the business.
I found Perfecto to be pretty easy to use while executing against cross-platforms. The main reason is because the same script or test automation where we execute on multiple platforms has minimal changes that I need to do. Also, it is easy for me to set up an execution on one platform, then on another platform, either in parallel or one after the other. Parallel opportunities save me time. Once the execution has been completed across these different configurations, I can always check and compare, e.g., what are the differences and consistencies?
We utilize Perfecto’s cloud-based lab to test across devices, browsers, and OSs. I use it occasionally for manual testing. Though, there are other team members who use it more frequently than I do. I use it mainly for executing my automated tests. We have the Perfecto lab, cloud devices, and machines. I can program my test to execute against any of those devices, which gives me more confidence in my product. I can compare and see how my product or application functionally behaves across these different devices and from a UI point of view, which helps me a lot.
The device lab is extremely important to our testing operations. We rely on having multiple devices up and running all the time. Whenever we kick off an execution, there are multiple reasons why executions may get triggered:
- CodeCommit
- A scheduled job.
- Might be on-demand by any stakeholder.
We need the lab to be available, as we need devices up and running for executions to take place. Also, the devices help since they allow us to have parallel execution, and not just wait for a sequential device to become free and available. Therefore, volume is definitely key. It also gives us an opportunity to compare execution across platforms in that space.
It is extremely important to you that the lab provides same-day access to new devices since we analyze that data every single day after execution.
Perfecto provides their own framework called Quantum Framework. That is one option. The other option is, if I want to have my own framework, I can have a Java-based Maven project, take a Selenium library, AppiumLibrary, and REST Assured library, and utilize the open-source framework. It is easy for us to connect to Perfecto, no matter what framework we use, as long as it has these core libraries in it. I can design and structure it any way that I want. The execution will happen in Perfecto no matter what since they have support for these tools or libraries. It is pretty neat that way. We are not dependent on using just one particular framework to use Perfecto. While there are still some framework limitations, there is the opportunity to use multiple, different open-source frameworks, then pass the execution to Perfecto. We can use most frameworks, then design and craft it any way that we want, then just pass the execution to Perfecto.
What needs improvement?
One improvement would be speed of execution. If it is an iOS native app, we have noticed that the speed is a bit slower. Perfecto might need to make some improvements in this area.
I feel that some of the Perfecto reporting might require upgrades or enhancements to their APIs when pulling some numbers or stats. Perfecto reporting also requires upgrades, mainly from an API point of view.
For how long have I used the solution?
I have been using it for five years.
What do I think about the stability of the solution?
I am happy with its stability. I have noticed that in the past five years or so, Perfecto has been quite stable. It is always up and available for us whenever we execute. They notify us, well in advance, if there will be any downtime or maintenance activity on their end. We have a Perfecto representative on-premises at all times, so information is funneled in well in advance about those maintenance windows. From an uptime perspective, it is pretty stable.
If you are using Perfecto, the automation solution requires some maintenance. However, just to use Perfecto, there is bare-minimum maintenance. Mainly, it is to check that our networking policies and firewall rules have not changed, and also to determine if we can still access the cloud from our company network.
I am not sure how much of an effort it is from the Perfecto side to maintain their remote devices. We don't get involved with those. However, from an automation point of view, we need to maintain our executions and make sure that our scripts and code are healthy in order for it to work with Perfecto. So, there is minimal to moderate maintenance, not too critical or major.
What do I think about the scalability of the solution?
Perfecto’s device lab has helped us to scale our testing operations, which has improved our organization in multiple ways. Instead of hosting our internal lab, Perfecto allows us to have a one-stop execution model where all teams plug into Perfecto and execute, and the reporting is available for everybody to see. Also, if we want to scale, we can always either acquire more licenses or cradles, then add more devices. As technology changes, or as devices change over the years, we can change them by upgrading our devices or upgrading the software. Maintaining them is not in-house. It is easy for us to contact the Perfecto help desk and ask them, "Hey, can you change a particular setting on that particular device?" or, "Can we upgrade this to a different Android version?", if we cannot do it remotely.
From a resourcing or technology point of view, it is very easy to scale. However, from a licensing point of view, scaling might be a challenge because it is expensive, but other than that, it is technically easy to scale. Licensing, while not a roadblock, is a challenge for the majority of organizations when scaling.
How are customer service and support?
The customer support has improved over the past two years. They have probably remodeled their customer service, changed something, or added more resources. Now, it is easier for us to pinpoint an issue, explain it, and then get resolution for it.
I would rate the support team as 10 out of 10.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
We previously used our in-house lab, not an external vendor. We switched to Perfecto because it was hard for us to maintain our own lab. Plus, we got other benefits with Perfecto, e.g., we got to know about their Quantum Framework after we partnered with Perfecto. Mainly, we switched because of their remote lab. That was a prime motivator to move to Perfecto, exposing those licenses and configurations of devices to other team members within the organization. Also, having access to Perfecto's documentation allows us to automate tests better.
What's my experience with pricing, setup cost, and licensing?
Although Perfecto is a good product for us to use, it is a bit expensive. It takes management a bit of work to find the appropriate funding for us to keep Perfecto. I imagine there could be some way to make it more accessible.
Which other solutions did I evaluate?
When I moved into the organization, we were already finalizing Perfecto. I am pretty sure that there must have been comparisons made with either DeviceAnywhere or Sauce Labs, but I am not aware of what those decisions or comparisons were.
What other advice do I have?
Do a proof of concept based on your organizational needs, not just treat Perfecto as a remote cloud solution. There is more to it. Check if the reporting of Perfecto suits you. Check if the time that it takes to execute on Perfecto is what you really need, e.g., are you able to plug your CI pipelines into Perfecto?
If you go with Perfecto, use Quantum Framework, which is quite nice. It is quite easy to onboard test automation engineers to this framework. Plus, it is not too rigid. You can refine, maintain it easily, and then design or change it to your needs.
Perfecto’s virtual devices are on our to-do list, probably for this quarter's roadmap.
We don't use BlazeMeter yet.
I would rate this product as nine out of 10.
Which deployment model are you using for this solution?
Private Cloud