We had some specific features that we opened tickets on, although they were not earth-shattering. For example, the way the menus scroll could be improved because it does not have a bar, the way that people are used to, where you can move up and down. It is something that you have to flick in order to activate.

When we were in development, it was a bit of a pain because we have onshore and offshore development. One of our development shops is in India, and we were running tests over there. When some of the users tried to log in, it was slow for them or we didn't have enough licenses. That was during the core development before we even launched. 

We got around that by purchasing more seats, tinkering with some of the virtualization pieces, and scaling. Now we don't have that issue, because we scaled back the offshore team significantly, so when we run it overnight, there's really no effect. We come in the next morning and review the results. It doesn't affect the overall business or the offshore team. 

There were also some bottlenecks because of the amount of time testing takes, so we started using more tunnels and running it in parallel. That was the main issue that we faced initially, but now that it's all set up, we're good to go. We were struggling with the volume of tests, and Sauce Labs suggested we run everything in parallel.

Sauce Labs isn't lacking any features that we want, and it has several we're not using, like mobile and API testing. They've also introduced a ton more features since we launched, so I don't see anything missing on their end. 

Something that needs improvement is the speed. Relatively speaking, it's slow, but that's because we are communicating with the Sauce Labs cloud service and, obviously, there are network delays. We're hoping that in the future it can just be a bit quicker.

Another aspect that could be improved is having the virtual devices boot up a little quicker. Sometimes we wait up to a minute before a device is available. It would be nice if that was 30 seconds or so.

With the desktop browser, we can inspect any screen with the web developer option, but they should provide something for mobiles so that we can quickly inspect elements on the device. To write the Selenium scripts, we require web locators. We have to capture them from the local and execute the script on Sauce Labs. If Sauce Labs can provide a solution where we can inspect any of the mobile devices online, it will be very helpful for us. 

I also faced an issue where during the execution time, the iCloud password used to expire within a certain period, which used to create pop-ups on the screen making our tests fail. That was one thing that we faced many times. If that can be improved, it will be very helpful.

Multi-domain SSO is a big concern for us right now, especially as we've been merged into a larger company. I suddenly have teams coming from 20 different domains, and because the main master Sauce Labs account is locked down to one SSO domain, there are teams that can't run a test right now. 

I've heard they're working on a solution and they've been very communicative with us about it. A solution to that would help us a lot.

The other issue is that when you're behind a corporate proxy and inside a corporate network, it's a nightmare in general. And the problem that we've come up against over and over again is that a lot of our network staff don't want to open up giant CIDR blocks to internal traffic. Opening it up to one endpoint which is Sauce Labs, through a little API is perfectly fine. That's one IP address and it's no big deal. 

But for their virtual machines they have one jump box and that jump box contains an entire CIDR block that changes each time the tunnel starts up. In other words, if every call went into one IP and that got bounced back to a load balancer or net gateway, that would help a lot too. When I go to the network team and say, "Hey, we're using a product, a piece of software, and I need an entire CIDR block opened," the first and almost eternal response to that will be "absolutely not."

Getting things set up and running is always going to be a bit of a struggle within any corporate environment. Most of that is not Sauce Labs' fault, but the things I just mentioned are things they could do to help deal with that struggle.

Some of the trends that it shows. It only allows you to view the last month. Having it go beyond a month, e.g., yearly trends, would be good. While the yearly trends are available, they are available to Sauce Labs administrators, who are internal to Sauce Labs. I don't know if they are available for customers yet.

Every time that we run scenarios where we need to discover the geolocation of our customers, by default it shows as Palo Alto, California. That's a problem for us and we need a workaround for those cases. It would be great if Sauce Labs didn't provide any geolocation by default, and we could provide the geolocation that we want. It would be helpful if we could enter a latitude and longitude into Sauce Labs configuration and say, "When you run a virtual Chrome device or an iPhone, make this your default location. Then, provide me that device so I can run my scenarios," because we have stores in different regions across the United States.

That is the only "ask" that I have for a feature, a geolocation that we can set by configuration. That would be awesome.

From a company use standpoint - I have no complaints. The one issue I have is the 14-day trial that a new user gets for free. I understand the concept of the trial period; however, I think this could be revamped to a free 30-minute run time every few months or after a significant update once the trial period has ended. Personally speaking - I like to maintain my automation framework and would like to occasionally test that it continues to run on Sauce Labs after releases without stealing from Sauce Labs.

I can't remove team members that have left the organization. I can only set them as inactive. It would be really nice to clean up my data and delete them from the team management.

One of the challenging areas for us is the reporting and the matrix. It should be based on roles, but right now it is only available for the admin role. The admin role can really do a lot of stuff, but our infrastructure team, which holds the admin role, is not ready to hand it over to us on the testing team. If Sauce Labs had permissions associated with roles, where this role could do this activity and that role could do that activity, it would be easier for us.

We have raised this suggestion with our product manager and she has put it in the product backlog, but, of course, she cannot guarantee when that will be looked at.

Sauce Labs' logging features could be improved. For example, when we run a mobile test on iOS, we want a way to capture the logs generated. It has some drawbacks there. Sauce Labs isn't able to give us everything we test for. 

Sauce Labs' dashboard could be improved by adding more filters and allowing more customization options. There was one instance where the dashboard on the Sauce Labs UI didn't meet our requirements, so we had to use the Sauce Labs API to create some apps and dashboards on our own. The API endpoints could be a little more robust and customizable. 

The real concern is the load time of applications or real devices when we start our tests. It takes some time to load the application or web browser. Sometimes, it is frustrating too. Since they are real devices, we understand it takes some time to load. However, if it were to improve, then that would be a great asset to the solution. So, we would like better responsive times when opening applications and running tests.

A really nice feature would be a pay-as-you-use model, as opposed to a strict number of VMs that you pay for continuously.

A service that could be improved is some of their Wiki documents are either outdated or point to broken URLs; however when contacting the customer service they are able to respond in a quick manner with either an updated document or with the steps to fix the issue. 

Another feature that could still be improved on is more error clarity. Sometimes when running automated scripts the test will fail on the device side instead of the script and errors only show a 500 try again message instead of a detailed script that could of a been a timeout error from the code. 

Improve easy way to automation testing and generate script automatically while running manual test cases on Sauce. 

1. When running manual test cases, provide us a way to record and playback the same steps

2. Provide JIRA integration plugin so that we can easily log issues.

3. Provide easy way to create scripts automatically for manual steps and use sauce labs UI to execute it again

4. Sauce labs should provide their own email client and easy to send email in that user session

5. One-click setup for sauce proxy

I would like for there to be more detail in regards to the quality of our code i.e. how many failures occurred, how many passed based on industry standard metrics, etc.

Nothing comes to mind right now, but this might be because I'm an administrator and not an end-user who is familiar with the application's use case. If we have teams asking for something that isn't already there, we communicate that back to Sauce Labs. That hasn't happened so far in the time that I've been in this role. 

I may not know what should be improved on the platform, but I think it could offer a greater variety of testing solutions. I know there might be competing solutions that cover more aspects that I'm unaware of, and it could expand a bit more. We've done a lot of automated testing scenarios, and that's all the rage these days, so I think Sauce Labs could look at potential ways to improve and cover other scenarios.

The process can be a little slow to configure and get started. It's a cloud platform and you're trying to run automated tests. It first has to be configured, then the test downloaded, then you have to configure the environment that you're going to run on, then kick off the test based on what your needs are. All of this takes time.

We have faced challenges with the availability of mobile devices. There was once or twice where there were no mobile devices available. These instances may have been due to a service outage.

The market is changing and we're seeing a lot of shift towards testing on OTT devices, like Apple TV, Fire TV Sticks, Chromecast and Roku. This is really an emerging market and has the potential to grow exponentially. We've been getting a lot of clients asking us to test on OTT devices, either manually or via automation. We have been able to automate testing on OTT devices, but it is done with local devices hosted on our premises, so we can't scale that fast or cover all of the huge demand we are seeing. We're looking for the ability to do this in the cloud, so if Sauce Labs offered such a feature, that would be really great. Working on OTT devices usually includes testing on desktop browsers or phones/tablets and having all of those supported in one platform would be ideal.

While Sauce Labs pretty much covers everything we need, another exception is HarmonyOS. I know it's a new thing and it's not supported by a lot of providers, but it is something that clients have started asking for. That is something we'd be really interested in seeing, for both manual and automated testing.

Also, latency, due to Sauce Labs being a cloud-based solution, has been a concern. We work in different continents and countries, but last time I checked, Sauce Labs was only offering two data centers, one in the EU and another in the US. If you're not in either of those two places, you would have latency and issues running your test scripts. However, I'm working with clients in different countries and we've seen some latency issues, depending on the country, although it's not huge. In comparison to other providers who have data centers in different places, there is some noticeable latency.

As a web product QA team, we sometimes need to spot check some new child site on multiple browsers and OS(es). It was a little time consuming for us since we need to click on each of the browser/OS combinations and start a new session to test. Every sprint, with new features and child pages being added, we mostly need to do the same steps over and over again. We would like to see if Sauce Labs can be able to open multiple different browsers on the same website at the same time, just by 1 click. It will reduce a lot of time for manually click and change the devices/OS combination.

The testing process is difficult. I need to prove the complete competency of the tool, and I am finding that challenging. 

Unable to segregate reports for tests that are currently being developed, and might not be returning useful results.

Overall, I think Sauce Labs provides us with a valuable tool and resource. As far as what could be improved, I would say the overall test execution time. Some of the calls take a bit longer than I expect, for example in web browser tests; while the execution time isn't obnoxious, it could be improved so that overall tests/test suites finish faster.

Within the insights section of the Sauce Labs tool, I think it would be nice if you could go back farther than 30 days worth of data. I like all the custom filters available, but being able to go back say 90 days (or more) would show more statistics on our data/test results.

In the next release, I'd like to see continual new additions of OS's. This area is usually pretty up-to-date, please keep at it!

We have used Sauce Labs extensively for the last 5 years across automated and manual testing and have found some problems during this time. Firstly as we use Sauce Labs to connect the tunnel to connect to apps in our internal network we have found that during automated testing this can be very slow. This causes inconsistencies with the tests. It's very difficult to rely on a service when you can't be sure if a test will pass or fail the next time it runs. This means building in a lot of sync time into the tests which in turn slows them down.  If this speed could be improved then the service would be much better.

An image comparison would be a nice feature to include in the Sauce Labs product. 

Rarely, some tests fail, seemingly without cause, and will succeed again the next time we run the tests. This isn't a behavior we saw when testing locally.

No

My company wanted test coverage for some devices that were not available on Sauce Labs so I still had to manage a small "test lab" with those devices. Some of the devices were Windows Phone 8 and Blackberry 7 and 10. On our site the only reported instances of those devices accessing our site was the test devices after 6 months so I am still unsure why we wanted to support those.

They could improve the platform's customer support services. Many users, including myself, have experienced significant delays in response times and unsatisfactory resolutions when reaching out for assistance. The support team often provides generic recommendations that do not address the specific issues raised, leading to prolonged downtime. Additionally, there needs to be more proactive communication and follow-up on reported problems.

Another area for improvement is keeping pace with updates to the latest libraries and technologies. For example, Sauce Labs' proprietary driver and Saucectl need to be updated more frequently to support the latest versions of popular testing frameworks like Playwright. This lag in compatibility means that users may miss out on critical features and improvements available in newer versions, necessitating workarounds or compromises in their testing workflows.

If I had to speak of an area that could be improved it would probably have to be the speed of interaction with the devices. There is at times a considerable amount of lag while using some of the virtual and at times even physical device farms. While this is absolutely better than not having it situation Sauce Labs certainly would benefit as well as we the user would certainly benefit from gaining a software interaction speed boost. Having it more consistent would be greatly appreciated if the long run. This is the only thing affecting the testing time at the moment and can even be seen as a disadvantage in comparison to having an actual physical device in hand.

The Jenkins Sauce OnDemand plugin could have more options available to tap into more of the custom capabilities Sauce Labs actually supports. Currently, in order to tap into these capabilities, a programmatic solution is required from the test code rather than being able to configure them via the Jenkins Plugin UI, e.g., desktop resolution, browser automation binary versions, remote Selenium version, and browser console logs. These are all things it would be nice to be able to customize straight from the Jenkins Plugin UI. 

In our experience, we have also had some issues with the Sauce Connect Proxy on our Jenkins servers failing to start, which makes the optimal CI/CD pipeline come to a halt. 

User account management needs an overhauls, allowing for user groups rather than just a hierarchy structure. Test job queuing is a first-come-first-serve, meaning there is no inbuilt way to queue batches of jobs from different Jenkins servers/accounts.

Speeds up the time it takes to run end-to-end user interface (UI) tests inside their virtual machines (VMs).

Documentation is pretty extensive, but at an advanced level could be a bit better. But they are always helpful. Besides that everything is great, they help out right away, so nothing is a big deal.

Finding ways to debug our flakier tests is an ongoing pain for us (we've fought with our own tests' flakiness across all different providers, often e.g. due to some test steps using implicit rather than explicit waits). We could always use more insights in to the causes of different failures, common advice when a failure type is detected, or more detailed guidance on best practices for test runner setup.

Running tests in the SauceCloud can take longer than running in a local environment.

I don't feel there are any real areas for improvement. As long as you follow best practices for programming automated tests and running them via Sauce Labs, then things tend to go smoothly.

It would be nice if they could ask for the URL and what browser/OS I want to test in, but not just one combination, I want the ability to start up 10 concurrently at a time. I would also like to put in a username and password before the launch of the test so it will auto put those in all on 10 instead of going into 10 tabs and having to put the same information over and over again.

The main interface, although clean, is sometimes very slow to load on slower connections and selecting a machine and waiting for it to boot up the environment takes a lot longer than other products I have used. This sluggishness also increases when the user tries to navigate the environment and interact with it. Selecting Microsoft Edge on Windows 10 gave me a really slow response time and took a while to boot up. All virtual environments are expected to be slow, however the interaction between the user and keyboard presses / mouse clicks wasn't very synchronized, and often the lag that is caused distracts away from the full experience. 

Manual testing also opens up in a separate browser tab, which I'm not sure why, but should ideally load over the main dashboard. 

1. They still have very slow support and ask many excessive questions before starting to fix the issue.

When we submit tickets in their reporting system, we expect them to start tracking down the problem as soon as possible. Instead, it looks like they ask standard questions just to delay work to a few days. E.g. 
1st day: "Could you confirm you use the latest version" - "Yes, we do"

2nd day: "Ok. Can you then try to run that with extra flag Y and grab the log for us" - "Yes, please, take the log"

<2 days of silence>

5th day: ... - "Guys, is there any progress on that?"

and so on...

Sometimes questions are really unrelated and take time like "Have you read the Getting Started guide?"

2. They still have platform-specific issues, especially on Mobile platforms. They wouldn't affect trivial tests though.

E.g. they have issues related to SauceConnect traffic routing specific to iOS 9 and Android 4.

This affect tests that want to sniff the traffic only though. Please, let me know if you want more technical details.

3. Sometimes they change configuration options (e.g. rename...

E.g. recently they changed desired capability for iPhone Simulator in a way like "iPhone 6 simulator" > "iPhone 6"

That affected multiple tests suddenly. They haven't made any notification about that though.

I would like to see it work directly with Azure Web Apps so that we do not have to have a separate step to run tests, or setup complex flows in Jenkins.  

Because our application front-end runs one Azure Websites which is a PaaS solution, we do not have server access and this makes it challenging for use to work with some external tools. I would like to see sauce labs have a direct integration with Azure Web App slots. This way we would not have to make the test runs a manual step. Currently we initiate tests manually and if they pass we do a manual promotion from an Azure Web Slot. We could integrate Jenkins or other release automation tool, but this would be additional overhead on top of what Azure provides out of the box. Another way to do this, instead of a direct integration with Azure. Is if their was a way to run tests in Sauce upon ever commit of a specific branch in GitHub. This way we could have a branch just for automation, and would not need to involve Azure Web App slots at all.

There have been various times throughout the last month or so where the service has gone down during business hours. This has impacted a release or two, so it is definitely less than ideal. Increasing uptime would be a good focus.

Getting accurate pass/fail results when running TestNG scripts in parallel by class is a challenge but after some work we figured out a solution.  I have also heard some complaints about lag when spinning up VMs but it is so much faster than the alternative that I don't mind.

A little clarity on documentation when integrating with third party APIs will help the users. Take for example, when a high-level language like Gherkin comes into the picture- how should users handle it?

Sometimes pasting text while using "text object" does not seem to work, and it slows down testing times quite a bit.

Playback needs improvement.

We still see simulators for a few mobiles in their VM's, we want to test on real devices. 

I don't really know of any, I wish everything life was a bit cheaper, so maybe that? I honestly have no complaints. They did all of the hard work so our lives are easier and the users get a better experience every time.

None as of now. Soon I will be purchasing ( if my Director approves :P) then i can honestly tell you. The 7-days scheme should be removed.

Links are not present very easily. One has to search for them.

Ability to run on physical devices is not present.

More physical devices.

It would be nice for it to ask for the URL along with browser/operating system I want to try, but not for it to simply ask for a combination. I want to be able to start ten at the same time. I would also like to put a username and password before the test launch instead of entering ten tabs and having to put the same information over and over again.

1. Ability to configure the memory and CPU for the test machines 
2. Ability to install profiles on iOS real mobile devices

Progress towards reducing application testing time can be made. There should be better and programmatic controls on request/response recordings and sharing with developers similar to new testing frameworks on the market.

Integration with Github, as well as several other similar tools, could be improved. However, Sauce Labs appears to be on that track. With all of the innovations that they have made over the past year and a half, I am looking forward to the changes they have planned for the future.

It should provide more examples of script code.

Sometimes the tests can pass or fail inconsistently. The tunnels and jobs timeout before the app loads, or it has to try 3 or 4 times to get a good connection.

The only caveat to mention, is the availability of the next release of iOS on their systems. Their adoption seems a bit slow. I understand Sauce is making an effort to enhance their current infrastructure to make updating to the latest iOS release easier in the future.

I think their logging could be improved a bit, possibly to include the output that I see on my command line (print statements, etc.) which help pin-point the error (if one happens) more easily. 

Speed is the key. We’re a business in Australia, so any tests that we perform are having to travel to America and back. This is time that we have to wait on for us to get responses back. In an active business every second counts.

Connectivity needs to increase.

Start execution time as each time a set of tests start, it will launch a new VM so it takes a bit of time.

The way the owner/user/sub-user hierarchy is set up can be confusing. It would also be nice to have more control over who can see what tests.

I think there could be improvements in the way Sauce Labs provides support to its users and could maybe provide tutorial videos on its services.

Just continue to support new versions as they come out. The speed can be a little slow, but other than that it's great.

I think that I found maybe only a few gaps which I don't think are very important. These are more like improvements which need to be added. 

It would be very good to see a short example of the code that needs to pass into Driver capabilities, something like auto-generation of this block. 

Also, it's not obvious where to look for the API key. It would be very nice to have a button on the top navigation menu.

Honestly, I can't really think of anything. It's been a joy to use and so incredibly simple.

Every product has room for improvement but for me it provided an all together complete solution that catered for all my needs.

The only drawback is the speed, it will be good if we have a server in Asia too. It will be great if we can improve speed while initialization and execution.

The connectivity needs improvement.

I would like to see improved network connectivity and it should allow playback for native apps.

CONS: None so far in my first 8 days of usage. Probably an additional 15 days of trial [just kidding], I will ping/communicate/ if I stumble across a problem for sure.

Video playback.

Documentation and speed.

The session duration is too short; it's only about 3 hours long.

I only tried the free trial so far, so not a lot to say about the cons.

Agility

Faster results

Some functionality needs improvement:

• Playback is not supported for apps.
• Speed needs some attention.

The slower speed needs to increase and the connectivity stability needs to improve.

Not really sure.  Maybe a .NET test runner option?  This may or may not already be in place, but I know it wasn't the last time I checked on it.

Sauce Labs has a test runner available for Ruby or some other language other than .NET which allows users to execute their tests directly from it.  This would be nice to have for .NET, as well.  Currently, tests are started locally and then they are run on Sauce Labs.  It would be a nice option to have it all started and executed on Sauce Labs.

It's a little bit slow sometimes.

I think it should be a longer trial period with more parallel tests available.

Some of the new features could be improved, for example, the playback on simulators.

Speed and connectivity need improvement.

The speed needs to increase. Something with real device also need to improve.

A little bit more speed needed to improve it.

Speed.

Having video recording would be great.

Nothing, all is good.

Stability needs improvement.

Better and programmatic controls on request/response recordings and sharing with developers.

Sauce Labs needs to be improved in the different platforms for farm testing, like iOS and Android farm testing and farm testing web browsers.

Sauce Labs test execution speed should improve.

More structured reporting.

Implementing user requests at a faster rate.

Internet connectivity to real devices has been a challenge.

• Provide an automation framework suite for basic websites

It would be helpful if they included more user-friendly reports. In addition, I would like to see them continue to introduce more browsers and mobile versions.