Sauce Labs OverviewUNIXBusinessApplication

Sauce Labs is the #3 ranked solution in top Test Automation Tools and #4 ranked solution in top Functional Testing Tools. PeerSpot users give Sauce Labs an average rating of 8.8 out of 10. Sauce Labs is most commonly compared to BrowserStack: Sauce Labs vs BrowserStack. Sauce Labs is popular among the large enterprise segment, accounting for 68% of users researching this solution on PeerSpot. The top industry researching this solution are professionals from a computer software company, accounting for 21% of all views.
Sauce Labs Buyer's Guide

Download the Sauce Labs Buyer's Guide including reviews and more. Updated: November 2022

What is Sauce Labs?

Sauce Labs is a functional testing tool that ensures your apps and websites work flawlessly on every browser, OS, and device. The solution allows you to automate functional testing on multiple operating systems and browsers, emulating the way that a user would use the website. With Sauce Labs, you can also run tests on various operating system and browser combinations in parallel, reducing the amount of time to get results. The Sauce Labs solution provides enterprise-grade security, scalability, and reliability.

Sauce Labs Features

Sauce Labs has many valuable key features. Some of the most useful ones include:

  • Automated cross-browser testing
  • Mobile testing
  • Secure tunneling protocol
  • Usage reporting
  • Enterprise security
  • Provisioning
  • Centralized user account management
  • Access controls
  • Support for Selenium, Appium, and JUnit Testing

Sauce Labs Benefits

There are many benefits to implementing Sauce Labs. Some of the biggest advantages the solution offers include:

  • Speed up development time: The solution makes it possible for you to test web and mobile apps in parallel and accelerate your testing time by up to 10x or more. Using Sauce Labs, you can increase developer productivity because your teams will be able to spend less time with test maintenance and debug faster.
  • No worrying about infrastructure: With Sauce Labs, you don’t have to buy or maintain more servers or mobile devices. You also don’t need to worry about updating browsers or VM licenses. Sauce Labs allows you to easily scale your tests with parallelization to speed up test cycles and boost capacity on their reliable cloud.
  • Run tests securely: The solution enables you to securely run your scripts and access your data and files from behind your firewall. Sauce Labs generates single-use VMs for every test and destroys them immediately afterwards so your data always remains secure.
  • Easy integration for your CI systems: Sauce Labs automated testing platform integrates with Jenkins, Bamboo, Microsoft VSTS, and others so you can test as fast as you develop. In addition, you can share results instantly by using the solution’s simple plugins.

Reviews from Real Users

Sauce Labs is a solution that stands out when compared to many of its competitors. It has valuable features that include error logging, helpful dashboards, and its browser.

Joel A., Product Manager - Data & APIs at a marketing services firm, says, "The error logging is very robust. If we run a test through Sauce Labs and there's some sort of issue, a log will appear on the screen. Log messages are usually heinous and horrible... Sauce Labs is incredibly good at saying things like, 'Hey, here is the exact issue. Fix this and you can run the test.' That helps in getting things up and running and executing the way they should."

Savio D., Sr. IT Architect at a healthcare company, mentions, "Sauce Labs' dashboards contain multiple useful metrics in one place. Everything is represented to us visually on the dashboard, which helps us understand where to focus our attention, what the issues are, and what we need to resolve."

PeerSpot reviewer, Rob L., Director of Quality Assurance - Shared Service at a financial services firm, writes, “The most valuable feature for us is the browser. The most critical thing is that this software aligns with our Agile and DevOps way of doing things. It integrates with kickoff scripts through DevOps.”

Sauce Labs Customers

Salesforce.com, Mozilla, Zendesk, Puppet Labs, Twitter, Bank of America, Eventbrite, Bleacher Report, Okta, Intuit, Travelocity, Sharecare, CapitalOne.

Sauce Labs Video

Sauce Labs Pricing Advice

What users are saying about Sauce Labs pricing:
  • "Their pricing is incredibly competitive."
  • "With respect to pricing, they did a bundled discount because we went with Sauce Labs for both mobile and browser. They were very competitive on pricing and provided a bundle discount for us as a larger customer."
  • "They could improve on the pricing because it seems pretty expensive. I'm sure it's justified, but it's expensive."
  • "The pricing is definitely on the higher end, and there are other options that are more cost-effective."
  • "The number of concurrent VMs that Sauce Labs provides depends on your purchase license level."
  • Sauce Labs Reviews

    Filter by:
    Filter Reviews
    Industry
    Loading...
    Filter Unavailable
    Company Size
    Loading...
    Filter Unavailable
    Job Level
    Loading...
    Filter Unavailable
    Rating
    Loading...
    Filter Unavailable
    Considered
    Loading...
    Filter Unavailable
    Order by:
    Loading...
    • Date
    • Highest Rating
    • Lowest Rating
    • Review Length
    Search:
    Showingreviews based on the current filters. Reset all filters
    Joel Alonzo - PeerSpot reviewer
    Product Manager - Data & APIs at a marketing services firm with 201-500 employees
    Real User
    Metrics about testing across the company are easily accessible, and it's easy to get teams up and running
    Pros and Cons
    • "The error logging is also very robust. If we run a test through Sauce Labs and there's some sort of issue, a log will appear on the screen. Log messages are usually heinous and horrible... Sauce Labs is incredibly good at saying things like, 'Hey, here is the exact issue. Fix this and you can run the test.' That helps in getting things up and running and executing the way they should."
    • "Multi-domain SSO is a big concern for us right now, especially as we've been merged into a larger company. I suddenly have teams coming from 20 different domains, and because the main master Sauce Labs account is locked down to one SSO domain, there are teams that can't run a test right now. I've heard they're working on a solution and they've been very communicative with us about it. A solution to that would help us a lot."

    What is our primary use case?

    When I started with my current company we had a fairly lean tech department and Sauce Labs was originally implemented for a piece of software we had just written that helped implement web testing.

    As things got bigger, and we got sucked into and were merged with another company, the number of teams and how they use Sauce Labs fundamentally changed. It went from everything running through just this one piece of software, and it managed all connections, to the point where we let everybody run tests now against the tunnel, if they're inside the network. Every use case for every team inside the company is going to be somewhat different. The majority of our approximately 85 teams that run things through Sauce Labs are web-based, and use it for tests that are run almost 24/7. We have engineers on those teams who will push changes to some type of front-end. There's a grouping of tests, depending on the team, and those tests are set via Jenkins and CI/CD to execute constantly.

    How has it helped my organization?

    The cool thing about Sauce Labs is their love of metrics. I don't like keeping track of metrics. My management is constantly screaming about metrics, but when you finally give them metrics they don't check them. We have built out individual solutions to track the metrics for various vendors' solutions, including who's using what, what time they are using it, and how long they are using it. We might spend two months building something like that and giving it to management, and it then gets ignored. Sauce Labs Insights provides an easy platform where I can see that there are 85 teams running tests; 23 teams running tests right now; most of their tests are passing; and they're across these browsers.

    The ease of access to that data is incredibly important because it means I don't have to build more metrics solutions that just don't get used. The fact Sauce Labs offers that kind of feature helps me in my job when it comes to responding to upper management and the requests that they sometimes make.

    Also, the solution definitely helps to save on manpower. I'm a part of a team that has been managing a few Selenium bridges, smaller scale, on-premises "Sauce Labs," to help with smoke testing, and managing that is a living, breathing nightmare. Overall, before I joined the company, everything was done by manual testing. One team would have one web app, a development team, and anywhere from 50 to 100 manual testers who just sat and clicked on a website all day long. When automation came in, my wonderful job was essentially getting a whole bunch of people fired or having them moved over to automation. Based on my experience with managing a few different Selenium bridges, the amount of man-hours I've spent just debugging that, I could easily have a team of five to 10 people and still probably be running a bit behind in terms of managing that grid. The fact that I don't have to do that at all saves us nine people right away. It's a solution that has really helped, but I balk at giving a more exact number because it's hard to calculate.

    The cost savings with Sauce Labs, on an annual basis are anywhere from $200,000 to $400,000, just for base level salaries for a small group of interns to provide the manpower it would take for us to manage something like Sauce Labs at a company level.

    There are also savings on testing time. A lot of that comes from the ability to get teams on board fairly quickly. With the Sauce Labs documentation, everything is made so that it's very easy for a team to just jump in and start running tests. That saves us a ton of time.

    We have different content spread across different regions and Sauce Labs also helps in enabling the testing that is required by that situation. The things that Sauce Labs helps with are things I don't want to do. I don't want to manage a giant stable of virtual machines. I also don't want to—and God bless them—but they've been managing Internet Explorer browsers. They're nightmares. The problems I have just trying to get an IE container or browser up and running, so a test can be automated against it, is insane. The fact that Sauce Labs has it down and working fairly well saves us a lot of work and headaches.

    What is most valuable?

    The ease of the application helps. It's fairly easy to get teams set up and running in Sauce Labs. It's easy to train new engineers on it if they are new to running web tests. That's a really nice benefit of Sauce Labs. 

    The error logging is also very robust. If we run a test through Sauce Labs and there's some sort of issue, a log will appear on the screen. Log messages are usually heinous and horrible. They mean nothing and they say nothing. Sauce Labs is incredibly good at saying things like, "Hey, here is the exact issue. Fix this and you can run the test." That helps in getting things up and running and executing the way they should.

    Another feature that gets used is the live testing. It's where you're not actually running an automated test but you're testing by clicking around inside of the Sauce Labs UI. That is incredibly helpful, especially for our teams with people who come from a manual testing background, because that's what they're used to. It helps in situations where people can't get automated tests up and running fast enough, but they want to test it across, say, 15 different browsers.

    What needs improvement?

    Multi-domain SSO is a big concern for us right now, especially as we've been merged into a larger company. I suddenly have teams coming from 20 different domains, and because the main master Sauce Labs account is locked down to one SSO domain, there are teams that can't run a test right now. 

    I've heard they're working on a solution and they've been very communicative with us about it. A solution to that would help us a lot.

    The other issue is that when you're behind a corporate proxy and inside a corporate network, it's a nightmare in general. And the problem that we've come up against over and over again is that a lot of our network staff don't want to open up giant CIDR blocks to internal traffic. Opening it up to one endpoint which is Sauce Labs, through a little API is perfectly fine. That's one IP address and it's no big deal. 

    But for their virtual machines they have one jump box and that jump box contains an entire CIDR block that changes each time the tunnel starts up. In other words, if every call went into one IP and that got bounced back to a load balancer or net gateway, that would help a lot too. When I go to the network team and say, "Hey, we're using a product, a piece of software, and I need an entire CIDR block opened," the first and almost eternal response to that will be "absolutely not."

    Getting things set up and running is always going to be a bit of a struggle within any corporate environment. Most of that is not Sauce Labs' fault, but the things I just mentioned are things they could do to help deal with that struggle.

    Buyer's Guide
    Sauce Labs
    November 2022
    Learn what your peers think about Sauce Labs. Get advice and tips from experienced pros sharing their opinions. Updated: November 2022.
    656,862 professionals have used our research since 2012.

    For how long have I used the solution?

    I've been using Sauce Labs as long as I've been at my current job, which makes it a little under five and a half years. At that point the company had just installed it. It was one of a few different pieces of software I was hired to manage.

    What do I think about the stability of the solution?

    Back in 2018 or 2019, I would have said that Sauce Labs' stability is horrible. We even shopped around looking for different solutions around that time, after I had been on the team for a little bit. 

    Whatever they've done since then, there has not been an issue. The emails I get that notify me of any sort of downtime usually come before my first client inside the company complains. That means I already know right away that it isn't on us, it's on Sauce Labs. 

    Except for an outage last week that lasted a bit of time—but by that time everybody was gone for the weekend, so we didn't get hit too hard—in the past four years it has been perfect. I was asked the other day if we plan on getting a higher concurrency cap because people just want more of it, and that's mainly due to how available it is.

    How are customer service and support?

    I work with a lot of different vendors, but the number-one thing about Sauce Labs is their ability to respond to questions. It is hands-down the best I've dealt with compared to any vendor. The quickness with which people respond, and the group effort they seem to make when we have questions, help out quite a lot.

    They communicate with us quickly, respond quickly, and they tend to be incredibly transparent. If something goes down, I know immediately, and I will already have 40 people in my inbox complaining about it. I can immediately determine whether it's a Sauce issue, an issue on our end such as an issue at the test level. 

    I have vendors that don't reply or it's two months before I get a reply, but with Sauce Labs I get one in under an hour, usually. They deserve a 10 out of 10 for support, compared to all the 15 or 20 different vendors I've worked with over the last five years. Hands-down, the customer service part is incredibly amazing. They've offered to send me to conferences. They've met the whole team down in Texas a couple of times. The amount that that company has pulled me in and kept me informed has been more than any other vendor has done. And not only that, but if I contact someone, and it's a tech issue and they don't necessarily know the solution, they immediately refer me to an engineer on their end.

    That whole process, whatever they're doing—and I think they've just invested heavily in customer support staff—is 100 percent working. I would give them a nine, though, because a 10 is a little too much. Still, there is very little room for making their current customer service better.

    How would you rate customer service and support?

    Positive

    What other advice do I have?

    If I were at a different company and someone said, "Why would we need to test front-end performance of the application at the beginning when most companies don't test it at all?" I'd wonder why that person hadn't been fired yet. I can't see an engineer saying something like that. That sounds like something that someone at a management level, or an admin who doesn't really understand things at a larger level, would say. What immediately comes to mind is security. If you're not testing the front-end for performance or functionality, at the very least there should be bare-bones security testing for the front-end software.

    The fact that Sauce Labs' Test Overview keeps data for longer than 50 days for the Virtual Device Cloud is a giant issue. I have arguments for both sides. My very honest response is that I don't care. I've had teams within my company come to me and say, "We need a video from a test we ran two years ago," and I just start laughing at them. Even if I was storing that data, I wouldn't store something that long. But there are also teams that come to me and say, "We need something we ran four hours ago." So there is a use to that data retention that Sauce Labs provides. As long as it doesn't literally go below a week or above 48 years—anything within that range—is enough. Whether it's 30 or 50 days wouldn't matter to me.

    We do use longer-term storage of test overview data to compare test results across quarters or teams, but we don't rely on Sauce Labs to store that data though. I think that's the best way. It's not their job to store that stuff. We have had administration come in and ask wacky questions. If the answers need to be stored, they need to throw it on their own servers. I don't want a vendor taking care of that for us in general.

    I have used the solution’s Failure Analysis feature. Is it verbose? Yes. Has it helped? Yes. Is it something I use often? No. But with the number of users we have, with 50 people per team, one of those users will likely be using the Failure Analysis at a given time.

    When I last checked a couple weeks ago, the number-one reason our tests fail—and this accounts for about 90 percent of the failures—is that people don't exit out of tests correctly. The entire test will run and it will pass. But because they don't close the session or close the browser, it is registered as a failure, which is as it should be. I'm not saying Sauce Labs is doing something wrong here. But that last 10 percent of failures have explanations that relate directly to the team and are useful every day. The 10 percent of tests aren't failing because of a user's input or a problem with Sauce Labs, they're failing because they're indicating a real issue. Those failures do get looked at with a lot of scrutiny on a team basis, which is what I'm looking for. I want the failures to be meaningful.

    Sauce Labs are incredibly good for dealing with the problem they originally tried to address. For that, the product is perfect. Nobody wants to have on-prem browsers or to manage all that stuff. That solution from Sauce Labs is perfect. Their uptime is incredible and their response is incredible. 

    I see, especially in the last two years, that they're starting to try to creep into different areas, which isn't bad. I'm not saying don't do that. Some of the solutions they purchased recently have been great. I have teams inside the company that are interested in some of those solutions. For example, I believe they have a low-code solution and that is something that helps with manual testing.

    I use a lot of software. There are tools, IDEs like IntelliJ, that were built for an original purpose and that, for a long time, were really good at that original purpose. But then they started adding so many features, bells and whistles, that all of a sudden what I used to be able to access by just clicking a button required me to go into a sub-menu. Has any of that happened yet with Sauce Labs? No. And a lot of the solutions they've purchased recently make sense to me.

    But my worry is that I don't want a hyper-complex Sauce Labs. The solution they originally built and that they have been fine-tuning over the past however many years, is absolutely wonderful. Their pricing is incredibly competitive. Everything they do with that solution is done incredibly well, especially in terms of how reinforced they've made everything in the past four years. Now, when something goes down, I get to go yell at my network team. I don't have to go yell at the Sauce Labs team, and that means a lot to me. Yelling at my network team is easier because they all know me.

    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
    PeerSpot user
    Savio De Souza - PeerSpot reviewer
    Sr. IT Architect at a healthcare company with 10,001+ employees
    Real User
    Top 10
    Enables us to have a fully functional CI/CD process while saving time and cutting down on the training required for individual testers
    Pros and Cons
    • "Sauce Labs' dashboards contain multiple useful metrics in one place. Everything is represented to us visually on the dashboard, which helps us understand where to focus our attention, what the issues are, and what we need to resolve."
    • "When we were in development, it was a bit of a pain because we have onshore and offshore development. One of our development shops is in India, and we were running tests over there. When some of the users tried to log in, it was slow for them or we didn't have enough licenses. That was during the core development before we even launched."

    What is our primary use case?

    We primarily use Sauce Labs to test browser compatibility. It's mostly functional rather than performance testing. We use a combination of tools, but Sauce Labs is mainly for compatibility testing. Selenium is our backend, and it also has compatibility testing, but we're not making use of that feature. Selenium is for capturing. 

    Our custom framework for testers combines Selenium and third-party vendors to do some of those performance metrics. At the same time, we use Sauce Labs to test cross-browser compatibility for the top five browsers that the government requires us to support. 

    We automate tests of our on-premise solution with Sauce Labs via tunnels. Sauce Labs allows individual testers to log in and test whatever they need, but we don't do it that way. Instead, we use the automation features through tunnels, and our CI/CD process will run tests for us through Sauce Labs. It returns metrics on compatibility and usage for us to review.

    Our two major platforms are Windows and Mac. We don't run tests on Linux. Even though we build everything on Linux, we don't support that for our end-users. We test our applications on the two main operating systems and variations of Safari. OS X testing is the main reason we started using Sauce Labs because we needed to test our applications on Safari, which isn't available on Windows. Initially, we purchased some Macs to do compatibility testing, but that didn't prove helpful at all. 

    We also need to test on all Chrome variations because there are multiple versions we need to support. When we launched, Microsoft had just released Edge, so very few of our users had it, but just about everyone has migrated from IE to Edge by now. Testing on variations of Firefox, Chrome, IE, Edge, and Safari is our essential requirement. 

    It's easy to set all that testing up on Sauce Labs. We could use a virtual machine to run applications on all the browser variations, but you need to get people in there to connect to it, and a homebrew solution is way too complex. With Sauce Labs, it's all already there. We just spin it up, specify the version we need, and we're done.

    Sauce Labs doesn't give us immediate feedback on every code commit. That's not how we have it set up. We've got a multistage process, so it goes through a code review for quality when we do the commit. We have unit tests that happen along the way, but when we do a full-blown merge and are ready for a release, that's when we actually launch our tests, and the tests run overnight. There are thousands of tests, which is why we don't do it on every code commit, but we do it every night. When a nightly job runs, we run a full regression test on that to get the results the following day.

    How has it helped my organization?

    From an automation standpoint, Sauce Labs enables us to have a fully functional CI/CD process while saving time and cutting down on the training required for individual testers. We can easily automate tests without training a whole bunch of people on Sauce Labs. There's a whole slew of quality and security testing tools that a tester typically needs to know. Automating and integrating with Sauce Labs reduced the number of things a tester needs to be trained on, and they don't need to use Sauce Labs daily. They just see the results.

    It's one less aspect the testers need to worry about. They come in from time to time to review the results, but it's automated, so they don't need to connect to Sauce Labs, run the test, and get the results back. They don't even have to know that it's Sauce Labs behind the scenes. We take care of that for them, so it saves us time.

    Automation allowed us to reduce the size of our team. When we initially got Sauce Labs, we had a full core development team with a lot of testers. There were 50 to 70 testers working 100 hours a week. That's around two hours each day per tester on the low end, so 50 multiplied by two is 100 hours per week. Multiply that by six to eight months of testing.

    What is most valuable?

    Sauce Labs' dashboards contain multiple useful metrics in one place. Everything is represented to us visually on the dashboard, which helps us understand where to focus our attention, what the issues are, and what we need to resolve.

    They've recently added a new tool for evaluating disability compliance, screen reader functionality, and so forth. Sauce Labs has integrated that, and they're developing it, but we haven't made the jump over to that. We are still using a third-party tool. 

    About six months to eight months ago, our technical rep was telling me about these new features, but our development and testing teams weren't ready to make that transition over. Now we're ready to go, and Sauce Labs is coming out with all these new features. We're probably not moving as fast as we should.

    We don't use two of Sauce Labs' most powerful features yet. One is mobile app testing, but that feature is one of the reasons we chose them. We plan to create a mobile app, and we'll be using Sauce Labs to test that, but not this year. That's probably something that we'll get to in 2023. The other feature is API testing. We use a lot of APIs and microservices.

    What needs improvement?

    When we were in development, it was a bit of a pain because we have onshore and offshore development. One of our development shops is in India, and we were running tests over there. When some of the users tried to log in, it was slow for them or we didn't have enough licenses. That was during the core development before we even launched. 

    We got around that by purchasing more seats, tinkering with some of the virtualization pieces, and scaling. Now we don't have that issue, because we scaled back the offshore team significantly, so when we run it overnight, there's really no effect. We come in the next morning and review the results. It doesn't affect the overall business or the offshore team. 

    There were also some bottlenecks because of the amount of time testing takes, so we started using more tunnels and running it in parallel. That was the main issue that we faced initially, but now that it's all set up, we're good to go. We were struggling with the volume of tests, and Sauce Labs suggested we run everything in parallel.

    Sauce Labs isn't lacking any features that we want, and it has several we're not using, like mobile and API testing. They've also introduced a ton more features since we launched, so I don't see anything missing on their end. 

    For how long have I used the solution?

    We've been using Sauce Labs for roughly three years. Our product has been in production for about two-and-a-half years now, so we were using it before that.

    What do I think about the stability of the solution?

    Latency hasn't been an issue for us. It's easy for us to go in and switch data centers if we need to. There was an issue two years ago, so we tried a different data center, but I don't think we've had to do that since we went live. We just let it run. Sauce Labs is one of the smoothest products we've integrated with and currently use. We haven't had any issues since we launched a little over two years ago.

    What do I think about the scalability of the solution?

    Sauce Labs is fully automated, so it scales well. When we initially talked about how many tests we would need, we underestimated how many seats and licenses to buy. We went back to Sauce Labs, and they provided us with additional metrics on how we need to grow. We purchased additional licenses so that we can scale accordingly.

    How are customer service and support?

    I rate Sauce Labs' support nine out of 10. I could go up to a perfect 10 if the response and solution times were a bit faster. Overall, Sauce Labs is a vendor I enjoy working with because we don't have any issues with them at all. With another vendor, I have a ticket that has been open for close to two-and-a-half years, and we're still trying to get it fixed. We had to put workarounds in place. Sauce Labs isn't one of those. They're always willing to help us out. I meet with them monthly, and they're constantly introducing tons of new features. It's been very smooth working with them.

    Initially, we had some hiccups, but this was way back. We always have challenges, but there haven't been any issues going forward. At the same time, I am not working with technical support much right now, so I can only speak to my experience with that particular team a few years back.

    When we started working with them, we had to reach out about some connection issues and other things that weren't working smoothly. Since we launched, there haven't been any issues. It runs in the background. It's stable. I meet with the rep every month to touch base about any new features and see if there's anything we want to introduce, but it's actually smooth.

    How was the initial setup?

    Setting up Sauce Labs is straightforward. We initially set up a few sets of test cases, and it took some time to get familiar with the product and understand how we're going to integrate. Once it was up and running, we did our own thing. We've increased the number of test cases since then. We bundle all of that up then take the CI/CD piece and just run it. There's no post-deployment maintenance on our end.

    What about the implementation team?

    We had help from Sauce Labs during the deployment, but they weren't on site. We did everything over the phone or on WebEx and Zoom. We had two QA architects building the framework doing the integration. We had about four or five technicians from our company, including two or three on the application side, and definitely two from testing.

    What's my experience with pricing, setup cost, and licensing?

    I believe the price of Sauce Labs is fair. I don't think it's over-priced or under-priced. It's a fair market value for what we're getting. We don't even use all the features, but as new features come out, my role is to educate the teams on ways they can put those features to work. 

    I've scheduled demos in the past, and they're well aware of what Sauce Labs can do. They also understand we're not fully utilizing it, but I've never heard any complaints about pricing. We negotiated with Sauce Labs, but I don't know what kind of deal we got. My role is more along the lines of evaluating the product from a technical application standpoint. License tiers and haggling over price aren't in my wheelhouse.

    Which other solutions did I evaluate?

    I am a solution architect working on multiple lines of business. When our teams present a problem to me, I have to look at it from an architectural, and a technical standpoint then figure out which tool to use. We looked at a wide variety of solutions, and I presented Sauce Labs as an option and a few others. In the end, we decided to go with Sauce Labs for two out of three of our lines of business simply because of ease of use and some of the feature sets they had.

    We considered Selenium, but it was too difficult to work with. Using Selenium for compatibility testing would require a great deal of effort to streamline Selenium for the other testers that we planned to bring on board. We are still using Selenium but in a different capacity.

    We opted for Sauce Labs for two of our lines of business, but I can't recall the solution we chose for the third one. I'm not on that team. They strictly work on mobile app testing. They decided not to go with Sauce Labs because they're dealing with iOS and Android support, and they didn't feel it was strong enough for them. 

    The fact that Sauce Labs was co-founded by the creator of Selenium and was an early mover in cross-browser testing was a big part of their marketing and sales pitch. However, it wasn't a critical factor in our decision. When we had some meetings and demos, they talked a lot about TestNG and some of the others, but we're not making use of that. 

    That really didn't come into it. We were more concerned with aspects like ease of use and stability. It was also crucial that Sauce Labs is a market leader currently deployed at other companies with a much higher volume than us. All that came into play.

    What other advice do I have?

    I rate Sauce Labs a 10 out of 10. I see no reason to move off the platform. In the future, we need to take advantage of everything Sauce Labs has to offer. It's a stable platform, and we're getting the support we need. 

    My advice to future Sauce Labs users is to use everything this feature-rich solution has. For example, there's the mobile app, automation, and API testing. Try to get as much as you can out of each of those areas and make that part of your game plan when you're developing your overall testing strategy. If you're looking at it from a cross-browser standpoint, think about what it can do for your mobile app and microservices testing. I believe most development is within those areas. Those are the key areas, so definitely use those features and have an individual strategy for integrating Sauce Labs into each area.

    Which deployment model are you using for this solution?

    Public Cloud
    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
    PeerSpot user
    Buyer's Guide
    Sauce Labs
    November 2022
    Learn what your peers think about Sauce Labs. Get advice and tips from experienced pros sharing their opinions. Updated: November 2022.
    656,862 professionals have used our research since 2012.
    Director of Quality Assurance - Shared Service at a financial services firm with 10,001+ employees
    Real User
    Top 20
    Robust documentation, helpful support representative, good licensing model
    Pros and Cons
    • "The most critical thing is that this software aligns with our Agile and DevOps way of doing things. It integrates with kickoff scripts through DevOps."
    • "We had some specific features that we opened tickets on, although they were not earth-shattering. For example, the way the menus scroll could be improved because it does not have a bar, the way that people are used to, where you can move up and down."

    What is our primary use case?

    We create banking software and we use this product for testing. We have different business units and we set up an enterprise license, so everybody feeds into it from each of our business units.

    We have about 750 websites and approximately 50 mobile apps, and we will test the different types of browsers against our automation. A good section of the work we do is running automation against different combinations, and that will expand into mobile devices once we kick off the new year.

    For the most part, it's heavy automation, but there is also testing that is manual, where they can log in and pick their devices or browsers.

    Our environment includes VMs on the cloud, as well as public and private devices. we have the CrossBrowser and we have the private and public cloud.

    How has it helped my organization?

    Sauce Labs allows us to expand our coverage without having to run them separately against different browsers on your local machine. You can just spin up an instance and run against whatever browser and operating systems combinations you need.

    Especially on the mobile side, things are improving. We had an on-premises lab, where we were managing the physical devices on a device cart. That's going away, as we are now able to use the public cloud offering, which has 2,000 plus devices.

    For locking down devices that we prefer to use without registering with other clients, we use the private Cloud. We split up the distribution 50/50 between private and public.

    Sauce Labs is what enables our different testing combinations. We run our different suites that include smoke tests, regression tests, and functionality testing. All of these types of testing need to be verified against different browsers and mobiles, and this is our solution to that problem.

    This product helps us to increase test coverage by testing both functional and visual aspects of the UI. This is very important to us because we have different types of customers. Some are end consumers, whereas others are banking customers. We won't release software without validating all of our use cases.

    Sauce Labs helps us to deliver new features and products to the market. It's part of the use cases that we have to validate, so we need tools to complement our test cases. We're trying to eliminate manual testing or how we kick off scripts. We want to be able to automate testing of our suite of use cases against our devices and operating systems.

    What is most valuable?

    The most valuable feature for us is the browser. That is all we're using right now, although come to the start of the year, we will be using the mobile feature.

    The most critical thing is that this software aligns with our Agile and DevOps way of doing things. It integrates with kickoff scripts through DevOps. Time to market is key, which means that we can't have a lot of manual interventions, and this fits into our automation program.

    What needs improvement?

    We had some specific features that we opened tickets on, although they were not earth-shattering. For example, the way the menus scroll could be improved because it does not have a bar, the way that people are used to, where you can move up and down. It is something that you have to flick in order to activate.

    For how long have I used the solution?

    We have been using Sauce Labs for between three and four years.

    What do I think about the stability of the solution?

    This solution is very stable. It's generally up 24/7.

    What do I think about the scalability of the solution?

    We have about 12 business units that use the service.

    Scalability is good on the public Cloud. You've got all of the different devices and most of the operating systems, including beta. It's all that we can ask for in that respect.

    The browsers are pretty straightforward. They usually deploy any new versions within 48 hours of the operating systems for the browser side.

    We have a thousand registered users and their roles are a combination of traditional QA testers and developers. To support them, we have a small team that I call admin support. There are only a couple of us on my team and if there are any issues, we centralize that before deciding if we need to open a ticket up to the Sauce Labs support center.

    Our small admin team works very well. Two people are sufficient to handle the training, education, and any configuration changes. In our company, one of them is the lead over all of the different tools and we license from different vendors. This person is a little bit more technical when it comes to the Sauce Labs configuration setup. The second is an admin for the end-users, teaching them how to use the tool appropriately. This person is also responsible for guiding end-users on how to test applications properly.

    How are customer service and support?

    The sales group was easy to work with. We had a service account representative, which is a nice setup. They handle the day-to-day activities and support needs, which helps to make us successful by answering questions and giving us solutions or enhancements.

    We meet with our service rep on a weekly basis and if we have any issues, we can discuss those solutions and opportunities. Basically, we centralize the communication flow between a small admin team under my team for the organization, and it seems to work very well.

    Overall, I would rate the customer service an eight and a half out of ten.

    Which solution did I use previously and why did I switch?

    We merged as a company two and a half years ago, and we had licensing agreements with BrowserStack, Kobiton, Sauce Labs, and SeeTest, which is part of Digital.ai. Some of the solutions were on-premises for the devices and we wanted to move to the cloud to simplify the infrastructure.

    We moved forward with Sauce Labs because they offered both public and private cloud options. The other ones did not necessarily have that combination. It's a win, having private and public both, as it is important to companies like us.

    How was the initial setup?

    The initial setup was very straightforward.

    We get a service rep that guides us on how to set up the tunnels to access our intra-sites, and then the device procurement is pretty straightforward. You give a list of what devices you want and they are put into each of the data centers, San Jose or Germany.

    They do offer training as part of the agreement, as well, and we're in the midst of doing that because we're rolling out the mobile component in January.

    Everything that I've needed, they've been there.

    In terms of how much effort and how long it took to deploy, because it's on the cloud, the majority of the effort is on the Sauce Labs side. We've had some basic internal procedures to follow, but that doesn't really align with Sauce Labs. This included things like security approvals and stuff like that but overall, it was very little effort on our side.

    Our implementation strategy was pretty straightforward. They make it available to us and then we do the training for the end-users. We have a registration list of about a thousand people, to which we have provided demos and documentation. Sauce Labs has a very robust document site. It's a combination of self-service education for the testers and developers in training. So, that's essentially our rollout plan.

    We had to rightsize how many licenses we needed, based on user consumption, which was part of the planning phase. We did a full pilot where Sauce Labs gave us an area to access the mobile environment, and we ran our use cases before selecting the software.

    What about the implementation team?

    We deployed in-house and we are happy with the support provided by the vendor.

    What was our ROI?

    The licensing agreement that we have has reduced our cost through solution consolidation by 30% to 40%. I don't have a precise measure of how much it saves the end-user because I don't have a good metric for it. That said, it expands our coverage and saves us from having to kick off the scripts against each of the different combinations. I would estimate that we save maybe 20% in this regard.

    In total, considering the bundled discount, we're going to save several hundred thousand dollars per year in licensing fees. The vendor has a tiered pricing structure and by going to one vendor and having enough licenses, we were able to get a more competitive price.

    What's my experience with pricing, setup cost, and licensing?

    With respect to pricing, they did a bundled discount because we went with Sauce Labs for both mobile and browser. They were very competitive on pricing and provided a bundle discount for us as a larger customer.

    I like the licensing model because it is a system of shared licenses. This is different from BrowserStack, for example, which I didn't like because they charge on a per-user basis. This matters to us because we have a situation where there are heavy users and light users. In this case, we prefer to have shared licensing.

    Shared licensing is like having a seat at the table and when we have a thousand registered users, it is easy to understand that some are heavy users and some are very light users. With the shared licensing, we don't have to manage the registration of whether they're heavy users or not. We don't worry about that. Rather, we just worry about the consumption of the licenses, which are shared amongst all.

    There are no costs in addition to the standard licensing fees.

    Which other solutions did I evaluate?

    We recently went through consolidation to get rid of BrowserStack, Kobiton, and SeeTest. We chose Sauce Labs over the other vendors and consolidated to one.

    With Sauce Labs, we did a full pilot before selecting it. The selling point and the reason that we ultimately chose this product is because we had our own service account representative that helped us to be successful.

    The features met our needs, which was the other evaluation point against the other software providers. They had both browser and mobile solutions, and it's easier to work with one vendor versus four. 

    What other advice do I have?

    We have usage reports, so we're taking the logs and evaluating that over time. Because we're consolidating from four tools to one, we considered the usage reports of all four of the vendors and came up with what our go-forward is. We expect our usage of the product to increase a little bit over time; maybe a 10% increment as we move forward and get more teams engaged.

    Although a 10% increase is in order, we won't purchase licenses without having the data to support that. We will have detailed usage reports for each business unit, which is what will help to determine how much we expand our usage of Sauce Labs. 

    My advice for anybody who is considering this product is that they need to understand their solutions and match it up against what is provided. Sauce Labs is leading in the area of both mobile capabilities and CrossBrowser and I would highly recommend looking into Sauce Labs. My decision was to choose Sauce Labs over those other leading contenders.

    The biggest lesson that I learned from implementing this solution had to do with what our options were. We had to do a comparative analysis that included breaking down all of the different features and comparing them against those other tools, to make sure they satisfied the features and functionalities we needed. Not having that knowledge across those vendors completely, we were able to do that through a pilot. Essentially, we came away with knowing the nitty-gritty by running the pilot and doing the feature analysis. As such, we were able to make sure that Sauce Labs met our needs.

    Overall, we are happy with this solution. We validated most of the functionality when we ran our pilot. Nothing critical stands out in terms of improvements that it needs.

    I would rate this solution a nine out of ten.

    Which deployment model are you using for this solution?

    Public Cloud
    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
    PeerSpot user
    Engineer at a pharma/biotech company with 10,001+ employees
    Real User
    Our test execution speed, time to results, and time to deployment have dropped significantly
    Pros and Cons
    • "Our machines are mostly Windows. Being able to test with Safari, on a Mac, and other types of browser pieces without having to manage all the infrastructure is the biggest feature that our team enjoys."
    • "On a rare occasion, I will come into a ticket where a customer will have reached out to me after reaching out to Sauce Labs, saying, "Sauce Labs doesn't understand what I am going through. They are not being very helpful." So, I try to do clean up there. Outside of those extremely rare occasions, I have only had one or two of those support issues."

    What is our primary use case?

    My company is quite large. My team supports the various different IT teams around the business who then work with their business partners. For example, if you think about our research and development division, our clinical trials division, and our manufacturing division, each one of those business groups has their own IT teams who build tools that the business team needs to do its jobs. My team provides the tools that the IT teams would need to build the tools for their teams. So, we are two or three layers removed from the broad side of the business.

    We are the team who provides this solution to other groups. However, in terms of our usage, because the company has done things by paper signatures, official paper, and hand documentation for a long time, there has not been a whole lot of progress yet on the automated testing sector. Therefore, our usage is very small compared to other businesses of our size. 

    We have 25 concurrent VM licenses for Sauce Labs today. That provision amount has not changed in my time here because it has never exceeded that capacity. So, it is a slow, upward trend, but it is very slow right now.

    How has it helped my organization?

    In January of 2021, there were 515 automated tests that ran through Sauce Labs in the company. By October, we were just shy of 3,000 tests. That is a 600% increase.

    In general, I know that there are different pockets of the business that build apps for different cases. For example, we have an iOS dev team in one part of the business. We have a Windows team in a different part of the business, and they are able to use the same platform without us having to try to support multiple different things. In that sense, the ability to test on additional browser, OS, and mobile device combinations has been a great help. 

    All our automated tests run in parallel. This helps us to demonstrate value to other parts of the business who have not yet adopted the solution. For example, when there are groups who use Sauce Labs and we try to share learnings, that team presents, and says, "I have saved this many specific hours. Or, I can run 225 tests (or however many tests) in parallel. It saves us this much time and this many dollars to have our application go out that much faster."

    What is most valuable?

    Because there is not super high usage for our team, the cross-browser functionality is a big feature for us. Our machines are mostly Windows. Being able to test with Safari, on a Mac, and other types of browser pieces without having to manage all the infrastructure is the biggest feature that our team enjoys.

    Sauce Labs is excellent in terms of the number of browser/OS combinations, mobile emulators and simulators, and real mobile devices that it offers. I have not heard any complaints from my customer teams about any lack of selection or particular combination that they are going after that Sauce Labs doesn't already have.

    Sauce Labs is optimized for automation and integration with the major CI/CD platforms and developer tools. Their API for inserting test results is fairly robust. This was an important factor for us because our adoption is still kind of low. There are not a lot of teams who know how to use it right out the gate. Now that our adoption is slowly growing and teams are figuring out how to use it, this has proven to be a big win.

    Sauce Labs provides access to automated functional testing. We have been very pleased with that. We make use of the automated piece and the manual piece. There are some teams who like to just poke around with a different browser to see how their app behaves by doing one-off tests.

    For how long have I used the solution?

    I have been with the company for three years. I have been supporting it all the time that I have been here. The company had the solution for roughly a year or so before I joined the team.

    What do I think about the stability of the solution?

    The stability is great. When I first took over, I was trying to find any general maintenance Windows, etc. so our team could be aware. I remember posting the first few outage notices internally to our customers, saying, "Sauce Lab is going to be down for such and such." Probably because of our lack of adoption, any kind of instability concerns have been really insulated for us.

    Latency has not been a concern. Once the pipeline gets set up, latency doesn't make too much of a difference because our development teams are attending their meetings or performing other work while the pipeline runs. Therefore, they are not overly concerned about the speed of what Sauce Labs can provide in terms of latency. At the same time, because the company is so archaic by modern standards with our automated testing footprint, any time savings over somebody going through and clicking the buttons by hand in an application is going to be greatly realized.

    What do I think about the scalability of the solution?

    So far, scalability has been great for what we need. It is hard to tell its true scalability since we are only running 25 VMs concurrently.

    There is definitely room for growth. Across all our separate divisions, people are realizing that automated testing is a major opportunity for improvement. As these individual groups continue to learn from each other and ramp up their own use of automated testing in solutions like Sauce Labs, that will really drive our growth.

    How are customer service and support?

    We have made use of Sauce Labs’ technical expertise to help us integrate automated testing into our CI/CD pipeline and DevOps toolchain. We have done that to great success with their support teams. I have also had the Sauce Labs team out onsite to the company at one point before the COVID pandemic hit. So, we were able to bring in several people from the company who were interested in seeing how to shape some of it for themselves. Sauce Labs was able to present and provide direct guidance to those teams, and it was a really successful event.

    The technical help wasn't overly helpful for me because I am just a really technical guy. I try to figure all that stuff out for myself. However, it was extremely helpful for our team at the company. A lot of the time, most of them don't like to try to figure things out for themselves. They want a dedicated instruction set, rule book, or whatever to say, "Here is how you go and do this." For Sauce Labs to come in and be able to actually show how it was done, I think that provided a lot of value for them.

    The technical support team has been great for any kind of issue that my team can't resolve on our own. We were always comfortable opening tickets with the support team. They have always been super responsive and educational when helping us to understand the cause behind the problem, not just how to fix it. I would rate them as nine out of 10 because there have been one or two occasions where it hasn't been absolutely perfect, but it has been really stellar overall.

    Sometimes, my customers find a way to go to Sauce Lab support directly, and either they are unable to communicate to Sauce Labs support what their issue is or Sauce Labs is not able to understand them. It is one of the two. Usually, my team plays that middleman and we can facilitate pretty well, providing context to any potential problems or issues within the company as well. For example, if there are specific company systems or things interacting with Sauce Labs that our customer teams may not know about.

    On a rare occasion, I will come into a ticket where a customer will have reached out to me after reaching out to Sauce Labs, saying, "Sauce Labs doesn't understand what I am going through. They are not being very helpful." So, I try to do clean up there. Outside of those extremely rare occasions, I have only had one or two of those support issues.

    Sauce Labs doesn't know what your test results are. This is a super application specific thing, but it is something that the company struggles with understanding. The app is just the endpoint for the execution of your commands. It can't understand whether the test that you are trying to execute is passing or failing. So, you need to be sure to take that extra step and let Sauce Labs know the results of the test if you want to allow them to assist you in debugging what may have been wrong.

    How would you rate customer service and support?

    Positive

    Which solution did I use previously and why did I switch?

    It provides access to automated functional testing, visual regression testing, and front-end performance, as well as browser/OS combinations, mobile emulators and simulators, and real mobile devices, making it an all-in-one testing suite, which is extremely important to us. This was the primary driver for why we went with Sauce Labs before I joined the company. Having this around is definitely huge.

    How was the initial setup?

    The solution was adopted before I joined the company.

    What was our ROI?

    In general, our test execution speed, time to results, and time to deployment have dropped significantly (by approximately 10%), which is huge.

    What other advice do I have?

    For the teams who have it set up, Sauce Labs runs with every code commit and provides our developers with immediate feedback. We don't have that as an organization-wide thing right now because we are trying to increase our adoption and execution across the userbase. We are trying to champion this and educate individual teams on how to use it, but they have to develop their own process first. Also, we have regulations in place that prevent us from moving as fast as other companies would like us to.

    Going forward, as we continue to adopt not only the function of automated testing, but also the methodologies, best practices, etc., having all their features will be really powerful.

    I would rate the solution as nine out of 10. This is mostly due to our company's lack of engagement and being able to fully understand the product and usability for us.

    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
    PeerSpot user
    Software Developer Engineer in Test at a retailer with 5,001-10,000 employees
    Real User
    We don't have to maintain device farms or servers, and that means no security patching or compliance issues
    Pros and Cons
    • "Sauce Labs is optimized for automation and integration with the major CI/CD platforms and developer tools. We have an integration with App Center that we're working on. They have a storage API that lets us retrieve APK and IPA, iOS and Android builds off the phone, so that we can continue testing with CI/CD. They integrate with Jenkins, and Jenkins is the main CI/CD."

      What is our primary use case?

      We use it for automation testing of our e-commerce product. We also have some apps that use React Native and they deploy to mobile devices. We also do responsive mobile testing. That means we test anything that hits a website with a browser, or on a phone through React Native, through Sauce Labs.

      We also use their VMs and their video recordings.

      We use the automation testing and the ability to run it against many device configurations. It's very convenient.

      How has it helped my organization?

      Infrastructure provisioning is a big thing. The whole point of having this expensive license for Sauce Labs is so that we don't have to maintain multiple versions of Chrome, Safari, Firefox, or Edge. We don't have to maintain our own device farms or our own servers, and that means no security patching, compliance, or auditing. A whole bunch of infrastructure headaches are offloaded to Sauce Labs.

      Using it every day, and having all the manual QAs get some experience working with it, has saved us multiple person-hours. Just having an automated testing solution, Sauce notwithstanding, means an army doesn't have to sit there and click, click, click, multiple times, every time we do a release, to test the same old things and make sure the same old features still work. Having Sauce Labs on our side, we can actually do all of that at scale with the automation.

      The number of testing environments is definitely mission-critical because it plays a part in a release. We run these automations so that we are able to catch issues and so that a customer does not experience issues. And not having to do manual QA frees people up to do exploratory testing. It frees them up to use their intuition and domain knowledge to find bugs that have come in from new features and that might affect old features. It's absolutely essential that we have Sauce Labs. There's no way we could accomplish releases at our current rapid cadence without it.

      We also run tests in parallel. It would take way too long to do it one by one. It saves us tens, hundreds, even thousands of hours. And Sauce Labs has reports on that, telling you if you're maximizing your concurrency and whether your licensing is affected by concurrency units, which is great. Knowing that we can run tests in parallel means we can focus on the tests themselves and the quality of the tests. We don't want to create duplicate tests because that would increase test maintenance. Running them in parallel means that we're getting the most for our CPU buck.

      What is most valuable?

      We send over a configuration object in JSON and it's very convenient to be able to do it that way.

      Also, Sauce Labs is optimized for automation and integration with the major CI/CD platforms and developer tools. We have an integration with App Center that we're working on. They have a storage API that lets us retrieve APK and IPA, iOS and Android builds, install them on the phone, so that we can continue testing. They integrate well with Jenkins.

      It's super-important that the solution is optimized for integrating with these major CI/CD platforms and tools because at the manager level, they want integrations out-of-the-box. They want to reduce internal tooling or internal custom stuff.

      We use the browser/OS combinations, mobile emulators, and real mobile devices. It's huge having multiple types of testing available in a single platform. It's definitely a competitive differentiator. For example, Microsoft has its own test automation through App Center and there's also BrowserStack and other competitors. It's very important to be able to tell the decision-maker, "Hey, Sauce Labs already has it, so don't worry about it."

      They also have a huge number of browser OS combinations, mobile emulators, and real mobile devices. The solution covers a ton of combinations, probably almost any combination you would encounter when a custom reports a bug. That is great for QA to be able to reproduce that issue on that exact same device. 

      Sauce Labs maintains physical devices in their data center. They go out and buy the device and provision it for you when you have a real-device contract and licensing, and that's also huge. You're on a physical device.

      And for the mobile emulation, which is great as well, they not only have Apple devices, but different iOS versions, which is a huge feature, including different Safari versions on different macOS versions and different Windows versions. More often, you only have a subset of what Sauce Labs offers because people will be mostly using cutting-edge stuff or people might be using mostly legacy. But Sauce Labs runs the gamut and they have all kinds of devices. You'll run out of combinations that are relevant to you before you run every single combination that Sauce Labs has.

      I'm pretty happy with the areas of the product that I've been using. The Appium part, even though Appium feels pretty new, is still supported. They support Selenium 4 as well as several other test frameworks, such as Cypress, XCUITest, Puppeteer, and Espresso. Sauce Labs also has artificial intelligence, the AutonomIQ test framework. With AutonomIQ you can have manual QA where you submit an Excel file and then it just automatically creates a test. That's a killer feature.

      They offer so many things that we haven't even tried yet, like performance testing and courtesy Docker containers. They are continually updating the documentation. They have performance testing and visual testing. They even acquired Backtrace, which is some sort of error monitoring solution.

      For how long have I used the solution?

      I've been using the Sauce Labs solution for about a year and a half. Our company has been using it since 2016.

      What do I think about the stability of the solution?

      We haven't had any issues. Sauce Labs has been more reliable than we have.

      What do I think about the scalability of the solution?

      Scalability is connected to the pricing. The solution is scalable if you have the money to scale. It's based on what they call concurrency units, and they can get expensive.

      We have about a dozen users of the solution. They are mostly involved in test automation, SDET.

      How are customer service and support?

      Support is great, including the support ticketing. Every time I've had a support ticket, they have replied. If they need to, they escalate it. They'll answer technical questions about things like IP whitelisting, and they'll take a look at the screenshots we provide or links to tests that are failing. Their support is empowered to really probe and ask questions.

      We haven't used their expertise to help integrate automated testing into our CI/CD pipeline. We have generally solved every issue that we've encountered so far, but they do offer software architecture assistance. It's good to have someone at the software level, and not just sales or product support. If I say I'm having a development issue, it's good to be able to talk at that level, using the jargon.

      How would you rate customer service and support?

      Positive

      How was the initial setup?

      When I came in, the solution was already set up. Tweaking it has been easy.

      One of the great things about it is that there's no maintenance. We just throw a JSON content object over and then they take over from there.

      What was our ROI?

      I can't speak about metrics, but we're able to run automation tests in parallel and that helps with releases. It's definitely a critical part of the whole process. And even moving forward to cloud, it's definitely a big part.

      What's my experience with pricing, setup cost, and licensing?

      They could improve on the pricing because it seems pretty expensive. I'm sure it's justified, but it's expensive.

      For some of the features we aren't using yet, I believe we do need to add new licenses, but for others, we just need to try them out. We just need to have the bandwidth and time.

      Which other solutions did I evaluate?

      We use other products for front-end testing but there's no significant reason we couldn't do it with Sauce Labs. It's not lacking in that solution. We use other tools mostly due to dev team mindset. They prefer something more local to their use and something they're familiar with. If we were to push the QA side to do performance testing through Sauce Labs, they'd be open to it. 

      It's not only Sauce Labs, as a vendor, that offers automation, but there's BrowserStack and others that also offer it. But using Sauce Labs has been great.

      What other advice do I have?

      Definitely try it out. They are very friendly about giving you trials and then following up with monthly syncs. They'll connect you with a sales rep, an engineering-type salesperson, and you can have monthly chats with them. They'll keep you updated about their product updates. It's free to try it. Once you try it, I think you'll see the benefits.

      Latency due to Sauce Labs being a cloud-based solution hasn't been a concern at all. It runs automatically and sometimes it runs during off-hours, so any latency is not a big deal for us. For flaky tests we use Ruby, which has a rescue retry pattern that we use a lot and that's really helped. Test flakiness is just a reality of test automation and we have good workarounds for it. So cloud latency in Sauce Labs hasn't been an issue.

      We've been pretty happy with Sauce Labs. I'd probably have to think pretty hard about what it is lacking. It's been working for us and whatever we throw at it, including Appium, mobile device simulation testing, and being able to support multiple apps. The automation testing has been great. The SC (Sauce Connect) Proxy is pretty friendly. There are the VMs and the video recording. Overall, we've been pretty happy with it. I'd be hard-pressed to find a glaring issue that hasn't been addressed.

      Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
      PeerSpot user
      Hassan Radi - PeerSpot reviewer
      Head of Automation R&D at Applause
      Real User
      Top 20
      Enables us to support multiple versions of browsers, devices and OS combinations, but OTT support would make it more complete
      Pros and Cons
      • "They update for the latest browsers and mobile phones and support a lot of combinations. They have 1,000-plus desktop combinations and browser versions, which is really great. We need that at Applause. The all-in-one testing suite aspect of it is really important because most of our clients prefer to go to one place."
      • "Latency, due to Sauce Labs being a cloud-based solution, has been a concern. We work in different continents and countries, but last time I checked, Sauce Labs was only offering two data centers, one in the EU and another in the US. If you're not in either of those two places, you would have latency and issues running your test cases."

      What is our primary use case?

      At our company, Applause, we offer software testing as a service and we always get a lot of interesting, uncommon or challenging use cases from our clients. We sometimes get ones that require specific devices or browsers to work. For example, we have clients who want to mix testing on desktop browsers and apps or test on multiple apps to achieve some kind of scenario; perhaps you are at a restaurant, and you are ordering something on your personal phone/tablet, which shows up on the restaurant's tablet or desktop browser. 

      Our clients are not only looking for executing the test cases manually, but their target is to automate all of them and be able to integrate that into their CI/CD pipeline and get faster feedback about the stability of the changes that the development team produces on a daily basis.

      Sauce Labs covers all of our automation needs and also allow us to do manual testing in case we are verifying bugs or testing something else.

      How has it helped my organization?

      My main focus is on automation testing and Sauce Labs is an integral part in our success as a company as they offer 1000+ different combinations of desktop browsers, real devices & OS versions. This allows us to pretty much cover all the automation test scenarios we need. We've also integrated them in our internal SDKs and are using their platform on a daily-basis to test the quality of our clients' apps and websites.

      The number of the real devices they offer is also very important to our business, because most clients want to support multiple devices & OS versions. We have clients who want us to run automation tests on Android 7, 8, 9, 10, 11 and sometimes even on Beta versions. Sauce Labs offers all of that, enabling us to support all of these versions, which is really nice. The same goes for their desktop browsers, although most of our clients usually use either the latest version or the one before that. I haven't seen a client who wants to use and old browser version like Firefox 50, but it's a great feature that Sauce Labs offers, in case it is needed for any reason.

      In terms of delivering software faster by getting feedback with every commit, it depends on the client and the system under test (SUT); some of our clients are testing manually and that would take them days, others have a CI/CD pipeline and run some smoke tests after each commit. Part of our job at Applause is to enhance the overall testing process of our clients & automate it as much as possible. Instead of taking days to do a full regression, it might take a couple of hours to get the same results. 

      Because we're working with Sauce Labs, and we're doing automation for our clients, they get a pretty fast cycle of feedback that allows them to make the right decisions at the right time.

      Without Sauce Labs, it would be really hard to achieve the software quality standards or the automation coverage that our clients need.

      What is most valuable?

      Towards the beginning of this year, we had to constantly switch between Sauce Labs and their legacy solution for real device testing, TestObject. This was causing a lot of issues or confusion for our engineers & clients. Now, with the new unified platform, we get everything we need in one single place and it is so intuitive and straightforward. You can easily do manual testing, view running automation scripts or switch between data centers from one single place.

      Overall, we're very happy with the combinations of browsers and devices they offer, especially because they always have multiple instances of the same device. If you want an iPhone 12 for your tests, you can get five of them for example and run your test scripts in parallel, which reduces the total execution time.

      Sauce Labs also allows specifying the browser version as "latest", which means that they automatically pick the latest version they support and run the tests against it. Previously, we had to manually change the browser version in our scripts whenever a new one becomes available and it was wasting a lot of time (as it needed to be done for a lot of projects).

      Their platform also offers some cool features if you're working with apps. You can just pass them a URL to an app build that is stored somewhere on your servers or on AWS, and they will install that on the phone/tablet and run tests against it on the spot. This has allowed for an easier integration with our SDK, as opposed to competitors who don't offer such feature.

      What needs improvement?

      The market is changing and we're seeing a lot of shift towards testing on OTT devices, like Apple TV, Fire TV Sticks, Chromecast and Roku. This is really an emerging market and has the potential to grow exponentially. We've been getting a lot of clients asking us to test on OTT devices, either manually or via automation. We have been able to automate testing on OTT devices, but it is done with local devices hosted on our premises, so we can't scale that fast or cover all of the huge demand we are seeing. We're looking for the ability to do this in the cloud, so if Sauce Labs offered such a feature, that would be really great. Working on OTT devices usually includes testing on desktop browsers or phones/tablets and having all of those supported in one platform would be ideal.

      While Sauce Labs pretty much covers everything we need, another exception is HarmonyOS. I know it's a new thing and it's not supported by a lot of providers, but it is something that clients have started asking for. That is something we'd be really interested in seeing, for both manual and automated testing.

      Also, latency, due to Sauce Labs being a cloud-based solution, has been a concern. We work in different continents and countries, but last time I checked, Sauce Labs was only offering two data centers, one in the EU and another in the US. If you're not in either of those two places, you would have latency and issues running your test scripts. However, I'm working with clients in different countries and we've seen some latency issues, depending on the country, although it's not huge. In comparison to other providers who have data centers in different places, there is some noticeable latency.

      For how long have I used the solution?

      I've been using Sauce Labs for over five years.

      What do I think about the stability of the solution?

      Sauce Labs is pretty stable in comparison to other providers. We still get some issues every now and then or random failures, especially when there is a new OS or browser version, but it's not a big deal and we can easily contact Sauce Labs' support to get things fixed.

      What do I think about the scalability of the solution?

      One of Sauce Labs' competitors does not force you to select the data center you want to run against. They just figure out which data center is closest to where you are executing your test scripts from. They offer a single, unified endpoint or URLs when it comes to automation and handle everything else internally. When we're writing automation scripts, we don't need to specify, "I want to run on the European data center," or "I want to run on the American data center." We can just say, "I want to run a test case," and depending on the location you come from, the platform is smart enough to direct it to the nearest data center to reduce latency.

      This may not be directly related to scalability, but this kind of capability would make it easier for us to build our SDKs faster and focus on other features, which in return would allow us to scale faster as a company. The learning curve for newcomers would also be easier, because they wouldn't need to worry about figuring out which data center to run against.

      While I love the fact that we can specify the data center ourselves, because it gives us more freedom, I would love to see more data centers in different places around the world, to reduce latency, and the selection of the data center implicitly done by the platform, so we don't need to worry about it.

      How are customer service and support?

      We open support tickets, and they're pretty responsive, they get back to us on time. They're really good about fixing things and making their platform more stable.

      I've run into a couple of situations where tickets have been left for weeks without feedback, but that has not happened often. When it does happen, I can go to our customer success manager and say, "We need some attention to these tickets." She can take care of it and that makes the support pretty good.

      How would you rate customer service and support?

      Positive

      What other advice do I have?

      Pay attention to selecting the correct data center, otherwise you would run into some latency-related issues.

      Although we are not using visual regression testing, that is something that would be really interesting, because a lot of our media clients want to test visual validations on their apps. If Sauce Labs is offering that out-of-the-box, it would be really interesting for us.

      Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor. The reviewer's company has a business relationship with this vendor other than being a customer: Partner
      PeerSpot user
      Sr Staff Software Engineer, QA Enablement at a tech vendor with 501-1,000 employees
      Real User
      Top 20
      Easy to set up, saves us time, good parallelization capabilities
      Pros and Cons
      • "The most valuable feature is cross-browser, cross-OS, cross-mobile device testing."
      • "We have faced challenges with the availability of mobile devices. There was once or twice where there were no mobile devices available."

      What is our primary use case?

      We have several healthcare products across the healthcare continuum, and we use the Sauce Labs platform to test our applications across different browser and OS combinations. We also use it to do mobile testing across different mobile devices that we may not have. It's not easy to set up these different configurations, so the cross-browser and cross-OS platform in the cloud gives us the ability to test across these different configurations without having to set them up or maintain them.

      At this point, we are trying to focus on API testing.

      My role is with a central team that helps other teams. If another team is struggling then we reach out to them and offer assistance. Because of this, I am familiar with how some teams are using the product. If on the other hand, a team is doing well and doesn't reach out, then I don't have any insight into how they are using the solution or how well things are going. 

      How has it helped my organization?

      We've had an event where I worked with the vendor to coordinate an automation day at my company. They worked with our QA engineers through a training process where we had a test repo that we were able to run across different browser and OS combinations using the platform.

      Technical support is very important because it helps us get through issues faster. When we are trying to integrate with a new tool, as is the case now because we're trying to move to GCP, we have a dedicated solution engineer to work with us. The engineer is helping us as we ramp up with GCP, getting our automation to work.

      We went through this a few years ago with Azure. Learning a new tool is challenging enough, and then trying to make it work with our existing processes is even more difficult. For example, we need to enable our test automation to run on the new platform and have everything work together. It can be challenging, not to mention that mobile testing is still evolving. This is why support is important. They have helped us with our questions and to get through any blockers.

      For the most part, the product is optimized for automation and integration with the major CI/CD platforms and developer tools. That said, our constant complaint is that it is kind of slow. We are trying to figure out ways that we can work around the slowness, and what things we can do better.

      We have a lot of products here in the company, and not everyone is leveraging the cross-browser testing platform that is available at Sauce Labs, but we have more and more teams adopting it in the last three years. We are trying to collect more insights into how our users are accessing our applications and using that insight to cover all those different testing configurations using the Sauce Labs platform, so we are trying to leverage the platform more in our internal testing.

      For now, Sauce labs covers our needs with respect to the number of browser/OS combinations and mobile devices. However, having more availability and looking into different strategies for making things faster are important. Our goal is always to get faster feedback on a build, which means if it takes a long time to run an automation suite then it's less likely that we'll use it. We're always seeking ways to add capabilities and expedite things.

      Having the ability to test our application on additional browser/OS/mobile device combinations opens up additional revenue streams for us because we have a larger user base. It gives us the confidence that the application we are putting out will generate a positive user experience because we've covered all of those testing combinations.

      We've gone through a lot of OKRs and have had a general goal to reduce the time it takes for us to test applications. This product enables us to run tests in parallel, which helps to make things faster. I don't have any concrete or documented examples of how long it would take to complete our testing before using the tool because we've evolved the way we write automation.

      There were times where it would take half a day to run an entire suite of tests, and through some of the improvements we've made, with and without the tool, our goal has been to be able to run a smoke test in 30 minutes. Some teams are able to achieve that, whereas others cannot.

      There is really no one answer that fits because we have teams that are across the spectrum. Some are working with mature products and some are just getting started. I know of one of our teams that optimized their automation and with Sauce Labs, they have been able to take their automated test time down from three hours to approximately 30 minutes. This is only one team out of many but it demonstrates a significant change.

      What is most valuable?

      The most valuable feature is cross-browser, cross-OS, cross-mobile device testing.

      The parallelization of automated tests is a best practice that I recommend. Some of the teams are able to do that, although some cannot because of the way the tests are formatted. If their configuration allows for it then it is definitely something that I recommend they take advantage of.

      Running testing in parallel has worked well for some teams. In one instance, a team was able to split their tests so that they ran across two platforms. Running these two configurations, they initially cut the three-hour testing time in half. Within each of those configurations, we were able to parallelize even further within our test framework, and ultimately, we were able to get the test time down to 33 minutes.

      What needs improvement?

      The process can be a little slow to configure and get started. It's a cloud platform and you're trying to run automated tests. It first has to be configured, then the test downloaded, then you have to configure the environment that you're going to run on, then kick off the test based on what your needs are. All of this takes time.

      We have faced challenges with the availability of mobile devices. There was once or twice where there were no mobile devices available. These instances may have been due to a service outage.

      For how long have I used the solution?

      I have been using Sauce Labs for approximately five years.

      What do I think about the stability of the solution?

      We run our testing on a daily basis and I don't think we've run into any stability issues. Nothing that I am aware of has been a concern.

      How are customer service and support?

      As we run into issues with anything that we are trying to do or integrating with to run our test automation, we reach out to their support team. Anytime that we need something and we need a quick answer, my first recommendation to anyone that reaches out to us for help is to call support and get it logged. This way, we can get someone to start responding.

      Some of the support team members are better than others. Overall, I would rate the support between six and seven out of ten.

      Which solution did I use previously and why did I switch?

      We did not use another similar solution prior to Sauce Labs.

      Also, personally, I have not used any other cross-browser testing platforms.

      How was the initial setup?

      I was not involved in the initial setup or getting it integrated with our automation test framework, but I think it was fairly easy.

      What about the implementation team?

      The initial setup was done in-house.

      As a SaaS product, it is always up to date.

      What's my experience with pricing, setup cost, and licensing?

      The pricing is definitely on the higher end, and there are other options that are more cost-effective. One of our teams that was using Sauce Labs separately decided to go with a different solution that was less costly.

      Which other solutions did I evaluate?

      Prior to choosing Sauce Labs, the company evaluated BrowserStack. I was not part of the decision process but the differentiating factor may have been the support.

      What other advice do I have?

      We have our internal framework and we basically use Sauce Labs for cross-browser capabilities and mobile device capabilities. Since we have our own framework, we have not explored any functional testing capabilities that the platform has.

      The vendor has acquired API Fortress, and it is available within our account for free to try out. That is something we're going to have a demo next week to look into because, at this point, we want to start focusing on API testing. It's helpful that we are able to run our UI tests across different platforms but we would like to focus more on the other layers of the application. We would like to test under the hood, the business layer, more.

      I would rate this solution an eight out of ten.

      Which deployment model are you using for this solution?

      Public Cloud
      Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
      PeerSpot user
      Senior Manager - Software QA at a hospitality company with 10,001+ employees
      Real User
      We wouldn't be able to test as quickly or simultaneously on various browsers without it
      Pros and Cons
      • "From an infrastructure support perspective, the number of VMs, browsers installations and versions that we would be maintaining without Sauce Labs would be a lot. This includes not only the infrastructure costs, but also the maintenance costs and people's time. The labor cost associated with maintaining all of that would be considerably high. In terms of efficiency, having concurrent VMs with various browser combinations available has allowed us to run multiple executions by all our teams."
      • "Sauce Labs has room for improvement with its price point. Using a real mobile device, and having that dedicated to your team, costs more than actually purchasing a mobile device. We haven't tried the real devices yet. This is because of their price point."

      What is our primary use case?

      We have been using Sauce Labs to test various browsers and OS combinations as well as test our applications. Our existing automation scripts are written in various technologies, which could be Java, JavaScript, Selenium, Cypress, etc. Jenkins is the tool that we use to typically run our jobs. Through Jenkins, they get scheduled and run in Sauce Labs. This is where we choose to run them, through various browsers and OS combinations.

      We use Sauce Labs core and whatever services that we choose to go with, like browsers and mobiles. From there, we can choose the browser and OS versions, etc.

      How has it helped my organization?

      Without Sauce Labs, we wouldn't be able to test as quickly or simultaneously on various browsers. That is the capability Sauce Labs has brought to us.

      The automated functional testing, visual regression testing are pretty important to us, as those are the use cases for which we use Sauce Labs. Specifically because it is hosted in the cloud, we don't have to use as much capacity from our own servers. We don't have to maintain and install various versions of browsers nor do we have to maintain them. It is a good solution and works well for us.

      Most of our teams who have solutions to test on mobiles are using the solution's mobile emulators and simulators. Having that availability is very important for a complete testing experience.

      We use the solution's emulators, simulators, and different browser version combinations for all our testing. For every release, we validate across various browsers. If it is a mobile application, then we need to validate across various Android and iOS devices as well as the previous few versions of each of the operating systems.

      The customer base that we typically serve is vast, diverse, and varied. This makes it very common to have our applications used across various applications, systems, and screen sizes. To test across all that, we needed a system in place. That is why we are using Sauce Labs. Without it, we would either develop something in-house or rely on possibly testing in only one particular system. For example, with Android, our entire customer base who is not Android wouldn't be able to validate and reproduce the user experience.

      When we are doing releases, the releases happen across all teams and various services. This means all of them need to test for a particular release deployment in whatever environment simultaneously. Based on the number of concurrent VMs that Sauce Labs provides, we can add or reduce VMs as needed. The amount of concurrency that we have purchased is specifically to support our various teams to do deployments simultaneously. Doing these simultaneous tests has reduced our test execution time.

      What is most valuable?

      There are a lot of analytics that you can do and look at when you run your jobs. It also gives you how much throttling has been seen across various teams over the past month. It tells you which particular line of code has been failing for however many runs that you have done. 

      The concurrent usage and VM availability are its strengths. We have found concurrent throttling very useful. So, we know exactly which job or team is using more than their capacity and might need additional capacity.

      We have also found the code analysis that gets run on Sauce Labs very useful. It tells us the line of code that has been failing or not received a new command.

      Sauce Labs is optimized for automation and integration with the major CI/CD platforms and developer tools. Most of our teams are creating CI/CD pipelines. Some of our teams do many deployments in a day. Without CI/CD, that pace is not possible. Having that capability, for whichever automation tool that we go with, is critically important for us.

      What needs improvement?

      Some of the trends that it shows. It only allows you to view the last month. Having it go beyond a month, e.g., yearly trends, would be good. While the yearly trends are available, they are available to Sauce Labs administrators, who are internal to Sauce Labs. I don't know if they are available for customers yet.

      For how long have I used the solution?

      My teams use it. I have about eight or nine teams and most of them have been using it at least for a year. Since I joined the company a year ago, I have been watching it being used.

      What do I think about the stability of the solution?

      The stability is good. I haven't seen any issues with it. We haven't experienced any latency or downtime issues.

      What do I think about the scalability of the solution?

      It has been pretty scalable. We are looking at increasing some of its concurrency capacity.

      We are currently using about 40 to 50 current VMS.

      One part-time resource is primarily needed to design the user groups where jobs will be run. Initially, we had all of our jobs run under a single user name, but then we decided to segregate that and divide it up amongst the teams. So, one part-time resource is needed just to analyze and manage how your jobs get run, then analyze the trends after that.

      How are customer service and support?

      The technical support has been pretty good. We haven't seen a lot of technical issues in which to engage their technical resources. Occasionally, we used them when we couldn't see something, didn't have an admin account, or for account creation. All those things got resolved very quickly, maybe in a day or so. I would say the technical support is good and rate it as 10 out of 10.

      How would you rate customer service and support?

      Positive

      Which solution did I use previously and why did I switch?

      It was implemented before I joined the company.

      How was the initial setup?

      It was already set up when I joined the company.

      What was our ROI?

      From an infrastructure support perspective, the number of VMs, browsers installations and versions that we would be maintaining without Sauce Labs would be a lot. This includes not only the infrastructure costs, but also the maintenance costs and people's time. The labor cost associated with maintaining all of that would be considerably high. In terms of efficiency, having concurrent VMs with various browser combinations available has allowed us to run multiple executions by all our teams.

      Typically, in a night, eight or nine teams have 40 to 50 jobs running simultaneously, which is amazing. Otherwise, if that was not concurrent, then it would have to be maintained in our own infrastructure. That would be a lot of money.

      What's my experience with pricing, setup cost, and licensing?

      Sauce Labs has room for improvement with its price point. Using a real mobile device, and having that dedicated to your team, costs more than actually purchasing a mobile device. We haven't tried the real devices yet. This is because of their price point.

      The number of concurrent VMs that Sauce Labs provides depends on your purchase license level.

      Latency has not been a concern due to Sauce Labs being a cloud-based solution. This comes back to the number of VMs and licenses that you have purchased. For example, if I have a capacity of 70 VMs, but I am running 100 jobs, then 30 of them will be throttled and we will see the latency. However, if I were to up my licensing, then I wouldn't see that latency. That is why it just depends on the license tier that you have ordered.

      Which other solutions did I evaluate?

      I have not compared it with other solutions.

      What other advice do I have?

      Plan for how you want to use it and how many teams will be using it as well as the types of accounts that it makes sense to have, different access levels, and who should have it on their team. If you plan ahead, then you don't have to fix it afterwards.

      We haven't tried the front-end performance testing.

      I would rate it as eight out of 10.

      Which deployment model are you using for this solution?

      Public Cloud
      Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
      PeerSpot user
      Buyer's Guide
      Download our free Sauce Labs Report and get advice and tips from experienced pros sharing their opinions.
      Updated: November 2022
      Buyer's Guide
      Download our free Sauce Labs Report and get advice and tips from experienced pros sharing their opinions.