Share your experience using AWS Device Farm

We have used the solution for malware testing and APM.

Actually, it gives the whole log, especially the output reports. The best part is the video part. It gives us the whole video of our execution. It has some in-built functions where we could keep our external files, and then we can give all those references of external files in our code. So, basically, we are not required to keep all those folders related to those files in the root folder.

It is slow. It is super slow. Performance is an area that can be improved. Whenever we upload our Zip files to that particular platform, you know, it takes quite some time, especially if the file is heavy. So on average, I have seen that it takes around ten minutes to upload our executable files.

There are some other advantages of AWS compared to some of the other services. I should have used it, but I could not use it since there was no need for them in my current project.

I have been using AWS Device Farm for around two years. My company has a partnership with Amazon.

I rate the stability an eight out of ten.

Scalability is good, but the only thing is that they have some timing, you know, time limits they have, so if you are going to use it for three or four hours, you have to distribute your project into four parts since the time limit is an hour and a half. I rate the scalability a seven out of ten. Seven automaton test engineers were using the solution in my company.

It is meant for large and medium organizations, not for the smaller ones.

Support comes under some sort of premium service. So, if we do not have premium service, there will be no technical support. Whenever I had to connect, I connected with the support team. They were helpful and very good.

I rate the support a nine out of ten.

Positive

I have been working on Perfecto and SeeTest for the last three or four months. Before that, I was working with AWS Device Farm.

The setup phase was fine because AWS had given the whole tree structure on their website. So, basically, if we go here and there apart from that tree structure, it does not support it. So they have a very sophisticated tree structure. So that is the only tree structure that we have to follow. Else, there will be problems.

I rate the setup process a seven out of ten, with ten being the easiest setup.

Since I had to study the solution, it took me a week to properly deploy it.

I just followed the instructions given by AWS to deploy the solution. So on their website, whatever the steps they had given, I simply kept on following it.

After our evaluation, we were planning to go for SeeTest, but there were some problems with its stability. So we used that for around one month and went to Perfecto. SeeTest was breaking a lot. There were some pop-ups or something which was not coming in SeeTest. Speed-wise, it was better than AWS Device Farm. Perfecto, I feel like it is better than AWS Device and SeeTest.

I would recommend others to use the solution.

It is a good solution because, you know, it is a distributed system. So we can give access to all of our team members, and they can go there and check for the reports. I am just a little, you know, concerned about whether we can do all those things in Jenkins. So I would say that if something can be done on Jenkins, we should not opt for a cloud solution since Jenkins is free, and you can do everything for free.

I rate the overall solution a seven out of ten.

I use AWS Device Farm as an emulation environment, and it's also where you can utilize real, physical devices that can install applications, then you can use Appium to run and perform tests on your applications on the devices set up on AWS Device Farm.

AWS Device Farm is beneficial to our company, and the benefits we get from it are the same as the benefits we receive from Appium because we use Appium in conjunction with AWS Device Farm.

What I like best about AWS Device Farm is that it offers actual physical devices that let you do more accurate testing because physical devices depict the live testing scenarios much better as opposed to emulated devices.

AWS Device Farm is a pretty nice solution. Because it's an AWS service, you can use the CLI to tie in several steps that can create the pipeline, and run it efficiently. AWS Device Farm also gives you monitoring ability, observability, logging, etc., so I'm pretty satisfied with the solution.

An area for improvement in AWS Device Farm is that it lacks a lot of features that would tie it in with other AWS services. The solution doesn't have great connectivity with other services offered by AWS, for example, AWS Secrets Manager. This should be improved because a lot of times, that missing functionality hampers the quality and engineering standards in terms of deploying the full AWS suite of services.

What I'd like to see in the next version of AWS Device Farm is for it to link better, or have some type of enrollment that would tie it in with other AWS services, such as EventBridge, Lambda, Secrets Manager, and any other new service from AWS.

I've been using AWS Device Farm for about two years now.

AWS Device Farm is a stable solution.

I didn't see any scaling issues with AWS Device Farm. It's a scalable solution, but it's a bit expensive to scale.

AWS support is quite responsive, and whenever my team had any issues with the AWS Device Farm setup, the AWS support team has always been very responsive and always gave guidance.

We only use AWS Device Farm and have not used any other solution. AWS Device Farm is one of the most cost-effective solutions out there, and it also ties in with other AWS services. As we use AWS extensively in our organization, it just made more sense to use AWS Device Farm instead of others that would offer the same functionality.

Setting up AWS Device Farm wasn't difficult. It didn't take up much time.

We haven't been making money off of AWS Device Farm. There have been cost savings or additional revenue, but those aren't revenue that's directly linked to the solution.

AWS Device Farm is an expensive service overall. You pay per device, and the cost for each device isn't cheap.

My company paid for a device slot. It's a yearly subscription for a single device slot, so that's more cost-effective for my company.

On a scale of one to five, where one is very expensive and five is very cheap, I'd give AWS Device Farm a two. It's on the more expensive side.

I'm using the latest version of AWS Device Farm. The solution is specifically provided by AWS, and it doesn't have a non-cloud version. My company runs it on AWS.

Between four to five people use AWS Device Farm in my company.

A single person can deploy AWS Device Farm, but it would still depend on how much you want to scale. If you have hundreds and hundreds of test cases running at a single time, then that would require more people, but to get AWS Device Farm up and running, you don't need more than one person to do that.

My company doesn't have plans of increasing AWS Device Farm usage because it's pretty expensive, but it's being used for a lot of work streams within the organization.

My advice to anyone who wants to use AWS Device Farm is that there's an expiration, and you have to check the expiry date on the test packages and test specifications. It would be nice to automate that part of AWS Device Farm.

The rating I'd give AWS Device Farm is six out of ten.

My company has a partnership with AWS.