We changed our name from IT Central Station: Here's why

Sauce Labs OverviewUNIXBusinessApplication

Sauce Labs is #6 ranked solution in top Functional Testing Tools and top Test Automation Tools. PeerSpot users give Sauce Labs an average rating of 8 out of 10. Sauce Labs is most commonly compared to BrowserStack: Sauce Labs vs BrowserStack. Sauce Labs is popular among the midsize enterprise segment, accounting for 59% of users researching this solution on PeerSpot. The top industry researching this solution are professionals from a computer software company, accounting for 34% of all views.
What is Sauce Labs?

Sauce Labs provides the world's largest secure and highly scalable automation cloud for testing desktop, mobile web, native, and hybrid applications. Sauce Labs helps companies accelerate software development cycles, improve application quality, and deploy with confidence across 450+ browser/OS/device/platform combinations.

Sauce Labs Buyer's Guide

Download the Sauce Labs Buyer's Guide including reviews and more. Updated: January 2022

Sauce Labs Customers

Salesforce.com, Mozilla, Zendesk, Puppet Labs, Twitter, Bank of America, Eventbrite, Bleacher Report, Okta, Intuit, Travelocity, Sharecare, CapitalOne.

Sauce Labs Video

Sauce Labs Pricing Advice

What users are saying about Sauce Labs pricing:
  • "With respect to pricing, they did a bundled discount because we went with Sauce Labs for both mobile and browser. They were very competitive on pricing and provided a bundle discount for us as a larger customer."
  • "The pricing is definitely on the higher end, and there are other options that are more cost-effective."
  • "The number of concurrent VMs that Sauce Labs provides depends on your purchase license level."
  • "We did initially go with Sauce Labs because of the pricing and integration."
  • Sauce Labs Reviews

    Filter by:
    Filter Reviews
    Industry
    Loading...
    Filter Unavailable
    Company Size
    Loading...
    Filter Unavailable
    Job Level
    Loading...
    Filter Unavailable
    Rating
    Loading...
    Filter Unavailable
    Considered
    Loading...
    Filter Unavailable
    Order by:
    Loading...
    • Date
    • Highest Rating
    • Lowest Rating
    • Review Length
    Search:
    Showingreviews based on the current filters. Reset all filters
    Director of Quality Assurance - Shared Service at a financial services firm with 10,001+ employees
    Real User
    Robust documentation, helpful support representative, good licensing model
    Pros and Cons
    • "The most critical thing is that this software aligns with our Agile and DevOps way of doing things. It integrates with kickoff scripts through DevOps."
    • "We had some specific features that we opened tickets on, although they were not earth-shattering. For example, the way the menus scroll could be improved because it does not have a bar, the way that people are used to, where you can move up and down."

    What is our primary use case?

    We create banking software and we use this product for testing. We have different business units and we set up an enterprise license, so everybody feeds into it from each of our business units.

    We have about 750 websites and approximately 50 mobile apps, and we will test the different types of browsers against our automation. A good section of the work we do is running automation against different combinations, and that will expand into mobile devices once we kick off the new year.

    For the most part, it's heavy automation, but there is also testing that is manual, where they can log in and pick their devices or browsers.

    Our environment includes VMs on the cloud, as well as public and private devices. we have the CrossBrowser and we have the private and public cloud.

    How has it helped my organization?

    Sauce Labs allows us to expand our coverage without having to run them separately against different browsers on your local machine. You can just spin up an instance and run against whatever browser and operating systems combinations you need.

    Especially on the mobile side, things are improving. We had an on-premises lab, where we were managing the physical devices on a device cart. That's going away, as we are now able to use the public cloud offering, which has 2,000 plus devices.

    For locking down devices that we prefer to use without registering with other clients, we use the private Cloud. We split up the distribution 50/50 between private and public.

    Sauce Labs is what enables our different testing combinations. We run our different suites that include smoke tests, regression tests, and functionality testing. All of these types of testing need to be verified against different browsers and mobiles, and this is our solution to that problem.

    This product helps us to increase test coverage by testing both functional and visual aspects of the UI. This is very important to us because we have different types of customers. Some are end consumers, whereas others are banking customers. We won't release software without validating all of our use cases.

    Sauce Labs helps us to deliver new features and products to the market. It's part of the use cases that we have to validate, so we need tools to complement our test cases. We're trying to eliminate manual testing or how we kick off scripts. We want to be able to automate testing of our suite of use cases against our devices and operating systems.

    What is most valuable?

    The most valuable feature for us is the browser. That is all we're using right now, although come to the start of the year, we will be using the mobile feature.

    The most critical thing is that this software aligns with our Agile and DevOps way of doing things. It integrates with kickoff scripts through DevOps. Time to market is key, which means that we can't have a lot of manual interventions, and this fits into our automation program.

    What needs improvement?

    We had some specific features that we opened tickets on, although they were not earth-shattering. For example, the way the menus scroll could be improved because it does not have a bar, the way that people are used to, where you can move up and down. It is something that you have to flick in order to activate.

    For how long have I used the solution?

    We have been using Sauce Labs for between three and four years.

    What do I think about the stability of the solution?

    This solution is very stable. It's generally up 24/7.

    What do I think about the scalability of the solution?

    We have about 12 business units that use the service.

    Scalability is good on the public Cloud. You've got all of the different devices and most of the operating systems, including beta. It's all that we can ask for in that respect.

    The browsers are pretty straightforward. They usually deploy any new versions within 48 hours of the operating systems for the browser side.

    We have a thousand registered users and their roles are a combination of traditional QA testers and developers. To support them, we have a small team that I call admin support. There are only a couple of us on my team and if there are any issues, we centralize that before deciding if we need to open a ticket up to the Sauce Labs support center.

    Our small admin team works very well. Two people are sufficient to handle the training, education, and any configuration changes. In our company, one of them is the lead over all of the different tools and we license from different vendors. This person is a little bit more technical when it comes to the Sauce Labs configuration setup. The second is an admin for the end-users, teaching them how to use the tool appropriately. This person is also responsible for guiding end-users on how to test applications properly.

    How are customer service and support?

    The sales group was easy to work with. We had a service account representative, which is a nice setup. They handle the day-to-day activities and support needs, which helps to make us successful by answering questions and giving us solutions or enhancements.

    We meet with our service rep on a weekly basis and if we have any issues, we can discuss those solutions and opportunities. Basically, we centralize the communication flow between a small admin team under my team for the organization, and it seems to work very well.

    Overall, I would rate the customer service an eight and a half out of ten.

    Which solution did I use previously and why did I switch?

    We merged as a company two and a half years ago, and we had licensing agreements with BrowserStack, Kobiton, Sauce Labs, and SeeTest, which is part of Digital.ai. Some of the solutions were on-premises for the devices and we wanted to move to the cloud to simplify the infrastructure.

    We moved forward with Sauce Labs because they offered both public and private cloud options. The other ones did not necessarily have that combination. It's a win, having private and public both, as it is important to companies like us.

    How was the initial setup?

    The initial setup was very straightforward.

    We get a service rep that guides us on how to set up the tunnels to access our intra-sites, and then the device procurement is pretty straightforward. You give a list of what devices you want and they are put into each of the data centers, San Jose or Germany.

    They do offer training as part of the agreement, as well, and we're in the midst of doing that because we're rolling out the mobile component in January.

    Everything that I've needed, they've been there.

    In terms of how much effort and how long it took to deploy, because it's on the cloud, the majority of the effort is on the Sauce Labs side. We've had some basic internal procedures to follow, but that doesn't really align with Sauce Labs. This included things like security approvals and stuff like that but overall, it was very little effort on our side.

    Our implementation strategy was pretty straightforward. They make it available to us and then we do the training for the end-users. We have a registration list of about a thousand people, to which we have provided demos and documentation. Sauce Labs has a very robust document site. It's a combination of self-service education for the testers and developers in training. So, that's essentially our rollout plan.

    We had to rightsize how many licenses we needed, based on user consumption, which was part of the planning phase. We did a full pilot where Sauce Labs gave us an area to access the mobile environment, and we ran our use cases before selecting the software.

    What about the implementation team?

    We deployed in-house and we are happy with the support provided by the vendor.

    What was our ROI?

    The licensing agreement that we have has reduced our cost through solution consolidation by 30% to 40%. I don't have a precise measure of how much it saves the end-user because I don't have a good metric for it. That said, it expands our coverage and saves us from having to kick off the scripts against each of the different combinations. I would estimate that we save maybe 20% in this regard.

    In total, considering the bundled discount, we're going to save several hundred thousand dollars per year in licensing fees. The vendor has a tiered pricing structure and by going to one vendor and having enough licenses, we were able to get a more competitive price.

    What's my experience with pricing, setup cost, and licensing?

    With respect to pricing, they did a bundled discount because we went with Sauce Labs for both mobile and browser. They were very competitive on pricing and provided a bundle discount for us as a larger customer.

    I like the licensing model because it is a system of shared licenses. This is different from BrowserStack, for example, which I didn't like because they charge on a per-user basis. This matters to us because we have a situation where there are heavy users and light users. In this case, we prefer to have shared licensing.

    Shared licensing is like having a seat at the table and when we have a thousand registered users, it is easy to understand that some are heavy users and some are very light users. With the shared licensing, we don't have to manage the registration of whether they're heavy users or not. We don't worry about that. Rather, we just worry about the consumption of the licenses, which are shared amongst all.

    There are no costs in addition to the standard licensing fees.

    Which other solutions did I evaluate?

    We recently went through consolidation to get rid of BrowserStack, Kobiton, and SeeTest. We chose Sauce Labs over the other vendors and consolidated to one.

    With Sauce Labs, we did a full pilot before selecting it. The selling point and the reason that we ultimately chose this product is because we had our own service account representative that helped us to be successful.

    The features met our needs, which was the other evaluation point against the other software providers. They had both browser and mobile solutions, and it's easier to work with one vendor versus four. 

    What other advice do I have?

    We have usage reports, so we're taking the logs and evaluating that over time. Because we're consolidating from four tools to one, we considered the usage reports of all four of the vendors and came up with what our go-forward is. We expect our usage of the product to increase a little bit over time; maybe a 10% increment as we move forward and get more teams engaged.

    Although a 10% increase is in order, we won't purchase licenses without having the data to support that. We will have detailed usage reports for each business unit, which is what will help to determine how much we expand our usage of Sauce Labs. 

    My advice for anybody who is considering this product is that they need to understand their solutions and match it up against what is provided. Sauce Labs is leading in the area of both mobile capabilities and CrossBrowser and I would highly recommend looking into Sauce Labs. My decision was to choose Sauce Labs over those other leading contenders.

    The biggest lesson that I learned from implementing this solution had to do with what our options were. We had to do a comparative analysis that included breaking down all of the different features and comparing them against those other tools, to make sure they satisfied the features and functionalities we needed. Not having that knowledge across those vendors completely, we were able to do that through a pilot. Essentially, we came away with knowing the nitty-gritty by running the pilot and doing the feature analysis. As such, we were able to make sure that Sauce Labs met our needs.

    Overall, we are happy with this solution. We validated most of the functionality when we ran our pilot. Nothing critical stands out in terms of improvements that it needs.

    I would rate this solution a nine out of ten.

    Which deployment model are you using for this solution?

    Public Cloud
    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
    Flag as inappropriate
    Hassan Radi
    Head of Automation R&D at Applause
    Real User
    Top 20
    Enables us to support multiple versions of browsers, devices and OS combinations, but OTT support would make it more complete
    Pros and Cons
    • "They update for the latest browsers and mobile phones and support a lot of combinations. They have 1,000-plus desktop combinations and browser versions, which is really great. We need that at Applause. The all-in-one testing suite aspect of it is really important because most of our clients prefer to go to one place."
    • "Latency, due to Sauce Labs being a cloud-based solution, has been a concern. We work in different continents and countries, but last time I checked, Sauce Labs was only offering two data centers, one in the EU and another in the US. If you're not in either of those two places, you would have latency and issues running your test cases."

    What is our primary use case?

    At our company, Applause, we offer software testing as a service and we always get a lot of interesting, uncommon or challenging use cases from our clients. We sometimes get ones that require specific devices or browsers to work. For example, we have clients who want to mix testing on desktop browsers and apps or test on multiple apps to achieve some kind of scenario; perhaps you are at a restaurant, and you are ordering something on your personal phone/tablet, which shows up on the restaurant's tablet or desktop browser. 

    Our clients are not only looking for executing the test cases manually, but their target is to automate all of them and be able to integrate that into their CI/CD pipeline and get faster feedback about the stability of the changes that the development team produces on a daily basis.

    Sauce Labs covers all of our automation needs and also allow us to do manual testing in case we are verifying bugs or testing something else.

    How has it helped my organization?

    My main focus is on automation testing and Sauce Labs is an integral part in our success as a company as they offer 1000+ different combinations of desktop browsers, real devices & OS versions. This allows us to pretty much cover all the automation test scenarios we need. We've also integrated them in our internal SDKs and are using their platform on a daily-basis to test the quality of our clients' apps and websites.

    The number of the real devices they offer is also very important to our business, because most clients want to support multiple devices & OS versions. We have clients who want us to run automation tests on Android 7, 8, 9, 10, 11 and sometimes even on Beta versions. Sauce Labs offers all of that, enabling us to support all of these versions, which is really nice. The same goes for their desktop browsers, although most of our clients usually use either the latest version or the one before that. I haven't seen a client who wants to use and old browser version like Firefox 50, but it's a great feature that Sauce Labs offers, in case it is needed for any reason.

    In terms of delivering software faster by getting feedback with every commit, it depends on the client and the system under test (SUT); some of our clients are testing manually and that would take them days, others have a CI/CD pipeline and run some smoke tests after each commit. Part of our job at Applause is to enhance the overall testing process of our clients & automate it as much as possible. Instead of taking days to do a full regression, it might take a couple of hours to get the same results. 

    Because we're working with Sauce Labs, and we're doing automation for our clients, they get a pretty fast cycle of feedback that allows them to make the right decisions at the right time.

    Without Sauce Labs, it would be really hard to achieve the software quality standards or the automation coverage that our clients need.

    What is most valuable?

    Towards the beginning of this year, we had to constantly switch between Sauce Labs and their legacy solution for real device testing, TestObject. This was causing a lot of issues or confusion for our engineers & clients. Now, with the new unified platform, we get everything we need in one single place and it is so intuitive and straightforward. You can easily do manual testing, view running automation scripts or switch between data centers from one single place.

    Overall, we're very happy with the combinations of browsers and devices they offer, especially because they always have multiple instances of the same device. If you want an iPhone 12 for your tests, you can get five of them for example and run your test scripts in parallel, which reduces the total execution time.

    Sauce Labs also allows specifying the browser version as "latest", which means that they automatically pick the latest version they support and run the tests against it. Previously, we had to manually change the browser version in our scripts whenever a new one becomes available and it was wasting a lot of time (as it needed to be done for a lot of projects).

    Their platform also offers some cool features if you're working with apps. You can just pass them a URL to an app build that is stored somewhere on your servers or on AWS, and they will install that on the phone/tablet and run tests against it on the spot. This has allowed for an easier integration with our SDK, as opposed to competitors who don't offer such feature.

    What needs improvement?

    The market is changing and we're seeing a lot of shift towards testing on OTT devices, like Apple TV, Fire TV Sticks, Chromecast and Roku. This is really an emerging market and has the potential to grow exponentially. We've been getting a lot of clients asking us to test on OTT devices, either manually or via automation. We have been able to automate testing on OTT devices, but it is done with local devices hosted on our premises, so we can't scale that fast or cover all of the huge demand we are seeing. We're looking for the ability to do this in the cloud, so if Sauce Labs offered such a feature, that would be really great. Working on OTT devices usually includes testing on desktop browsers or phones/tablets and having all of those supported in one platform would be ideal.

    While Sauce Labs pretty much covers everything we need, another exception is HarmonyOS. I know it's a new thing and it's not supported by a lot of providers, but it is something that clients have started asking for. That is something we'd be really interested in seeing, for both manual and automated testing.

    Also, latency, due to Sauce Labs being a cloud-based solution, has been a concern. We work in different continents and countries, but last time I checked, Sauce Labs was only offering two data centers, one in the EU and another in the US. If you're not in either of those two places, you would have latency and issues running your test scripts. However, I'm working with clients in different countries and we've seen some latency issues, depending on the country, although it's not huge. In comparison to other providers who have data centers in different places, there is some noticeable latency.

    For how long have I used the solution?

    I've been using Sauce Labs for over five years.

    What do I think about the stability of the solution?

    Sauce Labs is pretty stable in comparison to other providers. We still get some issues every now and then or random failures, especially when there is a new OS or browser version, but it's not a big deal and we can easily contact Sauce Labs' support to get things fixed.

    What do I think about the scalability of the solution?

    One of Sauce Labs' competitors does not force you to select the data center you want to run against. They just figure out which data center is closest to where you are executing your test scripts from. They offer a single, unified endpoint or URLs when it comes to automation and handle everything else internally. When we're writing automation scripts, we don't need to specify, "I want to run on the European data center," or "I want to run on the American data center." We can just say, "I want to run a test case," and depending on the location you come from, the platform is smart enough to direct it to the nearest data center to reduce latency.

    This may not be directly related to scalability, but this kind of capability would make it easier for us to build our SDKs faster and focus on other features, which in return would allow us to scale faster as a company. The learning curve for newcomers would also be easier, because they wouldn't need to worry about figuring out which data center to run against.

    While I love the fact that we can specify the data center ourselves, because it gives us more freedom, I would love to see more data centers in different places around the world, to reduce latency, and the selection of the data center implicitly done by the platform, so we don't need to worry about it.

    How are customer service and support?

    We open support tickets, and they're pretty responsive, they get back to us on time. They're really good about fixing things and making their platform more stable.

    I've run into a couple of situations where tickets have been left for weeks without feedback, but that has not happened often. When it does happen, I can go to our customer success manager and say, "We need some attention to these tickets." She can take care of it and that makes the support pretty good.

    How would you rate customer service and support?

    Positive

    What other advice do I have?

    Pay attention to selecting the correct data center, otherwise you would run into some latency-related issues.

    Although we are not using visual regression testing, that is something that would be really interesting, because a lot of our media clients want to test visual validations on their apps. If Sauce Labs is offering that out-of-the-box, it would be really interesting for us.

    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor. The reviewer's company has a business relationship with this vendor other than being a customer: Partner
    Flag as inappropriate
    Learn what your peers think about Sauce Labs. Get advice and tips from experienced pros sharing their opinions. Updated: January 2022.
    564,729 professionals have used our research since 2012.
    Sr Staff Software Engineer, QA Enablement at a tech vendor with 501-1,000 employees
    Real User
    Easy to set up, saves us time, good parallelization capabilities
    Pros and Cons
    • "The most valuable feature is cross-browser, cross-OS, cross-mobile device testing."
    • "We have faced challenges with the availability of mobile devices. There was once or twice where there were no mobile devices available."

    What is our primary use case?

    We have several healthcare products across the healthcare continuum, and we use the Sauce Labs platform to test our applications across different browser and OS combinations. We also use it to do mobile testing across different mobile devices that we may not have. It's not easy to set up these different configurations, so the cross-browser and cross-OS platform in the cloud gives us the ability to test across these different configurations without having to set them up or maintain them.

    At this point, we are trying to focus on API testing.

    My role is with a central team that helps other teams. If another team is struggling then we reach out to them and offer assistance. Because of this, I am familiar with how some teams are using the product. If on the other hand, a team is doing well and doesn't reach out, then I don't have any insight into how they are using the solution or how well things are going. 

    How has it helped my organization?

    We've had an event where I worked with the vendor to coordinate an automation day at my company. They worked with our QA engineers through a training process where we had a test repo that we were able to run across different browser and OS combinations using the platform.

    Technical support is very important because it helps us get through issues faster. When we are trying to integrate with a new tool, as is the case now because we're trying to move to GCP, we have a dedicated solution engineer to work with us. The engineer is helping us as we ramp up with GCP, getting our automation to work.

    We went through this a few years ago with Azure. Learning a new tool is challenging enough, and then trying to make it work with our existing processes is even more difficult. For example, we need to enable our test automation to run on the new platform and have everything work together. It can be challenging, not to mention that mobile testing is still evolving. This is why support is important. They have helped us with our questions and to get through any blockers.

    For the most part, the product is optimized for automation and integration with the major CI/CD platforms and developer tools. That said, our constant complaint is that it is kind of slow. We are trying to figure out ways that we can work around the slowness, and what things we can do better.

    We have a lot of products here in the company, and not everyone is leveraging the cross-browser testing platform that is available at Sauce Labs, but we have more and more teams adopting it in the last three years. We are trying to collect more insights into how our users are accessing our applications and using that insight to cover all those different testing configurations using the Sauce Labs platform, so we are trying to leverage the platform more in our internal testing.

    For now, Sauce labs covers our needs with respect to the number of browser/OS combinations and mobile devices. However, having more availability and looking into different strategies for making things faster are important. Our goal is always to get faster feedback on a build, which means if it takes a long time to run an automation suite then it's less likely that we'll use it. We're always seeking ways to add capabilities and expedite things.

    Having the ability to test our application on additional browser/OS/mobile device combinations opens up additional revenue streams for us because we have a larger user base. It gives us the confidence that the application we are putting out will generate a positive user experience because we've covered all of those testing combinations.

    We've gone through a lot of OKRs and have had a general goal to reduce the time it takes for us to test applications. This product enables us to run tests in parallel, which helps to make things faster. I don't have any concrete or documented examples of how long it would take to complete our testing before using the tool because we've evolved the way we write automation.

    There were times where it would take half a day to run an entire suite of tests, and through some of the improvements we've made, with and without the tool, our goal has been to be able to run a smoke test in 30 minutes. Some teams are able to achieve that, whereas others cannot.

    There is really no one answer that fits because we have teams that are across the spectrum. Some are working with mature products and some are just getting started. I know of one of our teams that optimized their automation and with Sauce Labs, they have been able to take their automated test time down from three hours to approximately 30 minutes. This is only one team out of many but it demonstrates a significant change.

    What is most valuable?

    The most valuable feature is cross-browser, cross-OS, cross-mobile device testing.

    The parallelization of automated tests is a best practice that I recommend. Some of the teams are able to do that, although some cannot because of the way the tests are formatted. If their configuration allows for it then it is definitely something that I recommend they take advantage of.

    Running testing in parallel has worked well for some teams. In one instance, a team was able to split their tests so that they ran across two platforms. Running these two configurations, they initially cut the three-hour testing time in half. Within each of those configurations, we were able to parallelize even further within our test framework, and ultimately, we were able to get the test time down to 33 minutes.

    What needs improvement?

    The process can be a little slow to configure and get started. It's a cloud platform and you're trying to run automated tests. It first has to be configured, then the test downloaded, then you have to configure the environment that you're going to run on, then kick off the test based on what your needs are. All of this takes time.

    We have faced challenges with the availability of mobile devices. There was once or twice where there were no mobile devices available. These instances may have been due to a service outage.

    For how long have I used the solution?

    I have been using Sauce Labs for approximately five years.

    What do I think about the stability of the solution?

    We run our testing on a daily basis and I don't think we've run into any stability issues. Nothing that I am aware of has been a concern.

    How are customer service and support?

    As we run into issues with anything that we are trying to do or integrating with to run our test automation, we reach out to their support team. Anytime that we need something and we need a quick answer, my first recommendation to anyone that reaches out to us for help is to call support and get it logged. This way, we can get someone to start responding.

    Some of the support team members are better than others. Overall, I would rate the support between six and seven out of ten.

    Which solution did I use previously and why did I switch?

    We did not use another similar solution prior to Sauce Labs.

    Also, personally, I have not used any other cross-browser testing platforms.

    How was the initial setup?

    I was not involved in the initial setup or getting it integrated with our automation test framework, but I think it was fairly easy.

    What about the implementation team?

    The initial setup was done in-house.

    As a SaaS product, it is always up to date.

    What's my experience with pricing, setup cost, and licensing?

    The pricing is definitely on the higher end, and there are other options that are more cost-effective. One of our teams that was using Sauce Labs separately decided to go with a different solution that was less costly.

    Which other solutions did I evaluate?

    Prior to choosing Sauce Labs, the company evaluated BrowserStack. I was not part of the decision process but the differentiating factor may have been the support.

    What other advice do I have?

    We have our internal framework and we basically use Sauce Labs for cross-browser capabilities and mobile device capabilities. Since we have our own framework, we have not explored any functional testing capabilities that the platform has.

    The vendor has acquired API Fortress, and it is available within our account for free to try out. That is something we're going to have a demo next week to look into because, at this point, we want to start focusing on API testing. It's helpful that we are able to run our UI tests across different platforms but we would like to focus more on the other layers of the application. We would like to test under the hood, the business layer, more.

    I would rate this solution an eight out of ten.

    Which deployment model are you using for this solution?

    Public Cloud
    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
    Flag as inappropriate
    Senior Manager - Software QA at a hospitality company with 10,001+ employees
    Real User
    We wouldn't be able to test as quickly or simultaneously on various browsers without it
    Pros and Cons
    • "From an infrastructure support perspective, the number of VMs, browsers installations and versions that we would be maintaining without Sauce Labs would be a lot. This includes not only the infrastructure costs, but also the maintenance costs and people's time. The labor cost associated with maintaining all of that would be considerably high. In terms of efficiency, having concurrent VMs with various browser combinations available has allowed us to run multiple executions by all our teams."
    • "Sauce Labs has room for improvement with its price point. Using a real mobile device, and having that dedicated to your team, costs more than actually purchasing a mobile device. We haven't tried the real devices yet. This is because of their price point."

    What is our primary use case?

    We have been using Sauce Labs to test various browsers and OS combinations as well as test our applications. Our existing automation scripts are written in various technologies, which could be Java, JavaScript, Selenium, Cypress, etc. Jenkins is the tool that we use to typically run our jobs. Through Jenkins, they get scheduled and run in Sauce Labs. This is where we choose to run them, through various browsers and OS combinations.

    We use Sauce Labs core and whatever services that we choose to go with, like browsers and mobiles. From there, we can choose the browser and OS versions, etc.

    How has it helped my organization?

    Without Sauce Labs, we wouldn't be able to test as quickly or simultaneously on various browsers. That is the capability Sauce Labs has brought to us.

    The automated functional testing, visual regression testing are pretty important to us, as those are the use cases for which we use Sauce Labs. Specifically because it is hosted in the cloud, we don't have to use as much capacity from our own servers. We don't have to maintain and install various versions of browsers nor do we have to maintain them. It is a good solution and works well for us.

    Most of our teams who have solutions to test on mobiles are using the solution's mobile emulators and simulators. Having that availability is very important for a complete testing experience.

    We use the solution's emulators, simulators, and different browser version combinations for all our testing. For every release, we validate across various browsers. If it is a mobile application, then we need to validate across various Android and iOS devices as well as the previous few versions of each of the operating systems.

    The customer base that we typically serve is vast, diverse, and varied. This makes it very common to have our applications used across various applications, systems, and screen sizes. To test across all that, we needed a system in place. That is why we are using Sauce Labs. Without it, we would either develop something in-house or rely on possibly testing in only one particular system. For example, with Android, our entire customer base who is not Android wouldn't be able to validate and reproduce the user experience.

    When we are doing releases, the releases happen across all teams and various services. This means all of them need to test for a particular release deployment in whatever environment simultaneously. Based on the number of concurrent VMs that Sauce Labs provides, we can add or reduce VMs as needed. The amount of concurrency that we have purchased is specifically to support our various teams to do deployments simultaneously. Doing these simultaneous tests has reduced our test execution time.

    What is most valuable?

    There are a lot of analytics that you can do and look at when you run your jobs. It also gives you how much throttling has been seen across various teams over the past month. It tells you which particular line of code has been failing for however many runs that you have done. 

    The concurrent usage and VM availability are its strengths. We have found concurrent throttling very useful. So, we know exactly which job or team is using more than their capacity and might need additional capacity.

    We have also found the code analysis that gets run on Sauce Labs very useful. It tells us the line of code that has been failing or not received a new command.

    Sauce Labs is optimized for automation and integration with the major CI/CD platforms and developer tools. Most of our teams are creating CI/CD pipelines. Some of our teams do many deployments in a day. Without CI/CD, that pace is not possible. Having that capability, for whichever automation tool that we go with, is critically important for us.

    What needs improvement?

    Some of the trends that it shows. It only allows you to view the last month. Having it go beyond a month, e.g., yearly trends, would be good. While the yearly trends are available, they are available to Sauce Labs administrators, who are internal to Sauce Labs. I don't know if they are available for customers yet.

    For how long have I used the solution?

    My teams use it. I have about eight or nine teams and most of them have been using it at least for a year. Since I joined the company a year ago, I have been watching it being used.

    What do I think about the stability of the solution?

    The stability is good. I haven't seen any issues with it. We haven't experienced any latency or downtime issues.

    What do I think about the scalability of the solution?

    It has been pretty scalable. We are looking at increasing some of its concurrency capacity.

    We are currently using about 40 to 50 current VMS.

    One part-time resource is primarily needed to design the user groups where jobs will be run. Initially, we had all of our jobs run under a single user name, but then we decided to segregate that and divide it up amongst the teams. So, one part-time resource is needed just to analyze and manage how your jobs get run, then analyze the trends after that.

    How are customer service and support?

    The technical support has been pretty good. We haven't seen a lot of technical issues in which to engage their technical resources. Occasionally, we used them when we couldn't see something, didn't have an admin account, or for account creation. All those things got resolved very quickly, maybe in a day or so. I would say the technical support is good and rate it as 10 out of 10.

    How would you rate customer service and support?

    Positive

    Which solution did I use previously and why did I switch?

    It was implemented before I joined the company.

    How was the initial setup?

    It was already set up when I joined the company.

    What was our ROI?

    From an infrastructure support perspective, the number of VMs, browsers installations and versions that we would be maintaining without Sauce Labs would be a lot. This includes not only the infrastructure costs, but also the maintenance costs and people's time. The labor cost associated with maintaining all of that would be considerably high. In terms of efficiency, having concurrent VMs with various browser combinations available has allowed us to run multiple executions by all our teams.

    Typically, in a night, eight or nine teams have 40 to 50 jobs running simultaneously, which is amazing. Otherwise, if that was not concurrent, then it would have to be maintained in our own infrastructure. That would be a lot of money.

    What's my experience with pricing, setup cost, and licensing?

    Sauce Labs has room for improvement with its price point. Using a real mobile device, and having that dedicated to your team, costs more than actually purchasing a mobile device. We haven't tried the real devices yet. This is because of their price point.

    The number of concurrent VMs that Sauce Labs provides depends on your purchase license level.

    Latency has not been a concern due to Sauce Labs being a cloud-based solution. This comes back to the number of VMs and licenses that you have purchased. For example, if I have a capacity of 70 VMs, but I am running 100 jobs, then 30 of them will be throttled and we will see the latency. However, if I were to up my licensing, then I wouldn't see that latency. That is why it just depends on the license tier that you have ordered.

    Which other solutions did I evaluate?

    I have not compared it with other solutions.

    What other advice do I have?

    Plan for how you want to use it and how many teams will be using it as well as the types of accounts that it makes sense to have, different access levels, and who should have it on their team. If you plan ahead, then you don't have to fix it afterwards.

    We haven't tried the front-end performance testing.

    I would rate it as eight out of 10.

    Which deployment model are you using for this solution?

    Public Cloud
    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
    Flag as inappropriate
    JoseMorales2
    Automation Architect at a hospitality company with 10,001+ employees
    Real User
    Failure Analysis helps us prioritize remediation, and we can trigger multiple tests for different teams and schedules
    Pros and Cons
    • "The Failure Analysis feature is really important for us, one of the most important aspects. What is the root cause? Is it because we have a defect or is it that we have a test case that we need to fix or modify? The Failure Analysis is one of the main functionalities that I am exploring all the time in Sauce Labs... The Failure Analysis helps us to discover which test cases we need to work on."
    • "Every time that we run scenarios where we need to discover the geolocation of our customers, by default it shows as Palo Alto, California. That's a problem for us and we need a workaround for those cases... It would be helpful if we could enter a latitude and longitude into Sauce Labs configuration and say, "When you run a virtual Chrome device or an iPhone, make this your default location. Then, provide me that device so I can run my scenarios," because we have stores in different regions across the United States."

    What is our primary use case?

    We have two kinds of applications using Sauce Labs in our company. One is the website, and we're using it to test across browsers, such as Chrome, Safari, Firefox, Edge, and mobile, to see that all the components we have in a web page work. 

    The second product we use the solution for is our driver application. We make and deliver pizza. Every time an order is received, we dispatch a driver to bring it to the customer. We have an application for driver dispatch on iOS and Android, and we run test cases for those as well.

    How has it helped my organization?

    I cannot imagine our company without Sauce Labs. We are using it a lot. It gives us a provider in the cloud for running test cases, scenarios, and validation, and that is really important nowadays. You could always run testing locally or have your own environment, but it's difficult to scale and difficult to maintain. You need to dedicate a lot of resources to do that.

    We ran, for example, 500 test cases in the last week for one specific team, and those test cases were running at 2:00 AM or 3:00 AM, for some specific scenarios. We could not do that without Sauce Labs. We are integrated for continuous integration using Jenkins and we're triggering all these tests nightly for different teams on different schedules. It would definitely not be possible to run that manually or in a different way. The cloud is the right environment to run it in, and for that, Sauce Labs is the best option in the market.

    Also, the Failure Analysis helps us prioritize remediation. If we have a very high number of test cases failing for a given feature, we can analyze them, see the root cause, see the code in our repository, and create a Jira ticket to address those failures. That feature is very helpful for us. It's pretty straightforward and doesn't consume too much time. I generally spend about 15 minutes analyzing all the failures across the teams to see which three test cases are failing most.

    Another benefit is that Sauce Labs saves us on manpower. We have manual testers, but it's not possible for them to cover all the different scenarios and all the different features before release. That means we need to rely on the automated test cases, and to do that, we need to run those test cases on a cloud platform. Sauce Labs saves us a lot of time doing those validations. It probably saves us from having to hire hundreds of people. Every team within our organization has one or two manual testers, but for every team we have hundreds of automated test cases that we run before release. It's good enough, for our operations, to have one or two manual testers. We are seeing good results with that, but to run hundreds of different scenarios I don't know how many people we would have to hire.

    What is most valuable?

    From my perspective, as an automation architect, the most helpful feature is the test history. I can easily go to the dashboard and see how many test cases we ran and how many failures and errors there were. I can segregate things by team and by specification. And I can tell a teammate, "You need to fix this test case, it is failing too much. The percentage of failure is too high." 

    The most important historical data is for the last seven days. I don't go too much beyond that period of time because my feedback is about how we did in the previous week. In our environment, everything is changing all the time. We are testing different products and running different test cases. So for me, it's the recent data that is key.

    The Failure Analysis feature is really important for us, one of the most important aspects. What is the root cause? Is it because we have a defect or is it that we have a test case that we need to fix or modify? The Failure Analysis is one of the main functionalities that I am exploring all the time in Sauce Labs. I can see a test case that is failing and on which particular platform or device it is failing. And the most important part is identifying the problem with the code. We can always go to the line of code and see the possible solution. The Failure Analysis helps us to discover which test cases we need to work on.

    Before the test history became the most important feature, the most important part was the configuration that made it easy to connect test cases to the cloud. As a result, we can easily test different devices at once. 

    And from a manual test perspective, testing in different browsers using the live test solution in Sauce Labs is very helpful.

    What needs improvement?

    Every time that we run scenarios where we need to discover the geolocation of our customers, by default it shows as Palo Alto, California. That's a problem for us and we need a workaround for those cases. It would be great if Sauce Labs didn't provide any geolocation by default, and we could provide the geolocation that we want. It would be helpful if we could enter a latitude and longitude into Sauce Labs configuration and say, "When you run a virtual Chrome device or an iPhone, make this your default location. Then, provide me that device so I can run my scenarios," because we have stores in different regions across the United States.

    That is the only "ask" that I have for a feature, a geolocation that we can set by configuration. That would be awesome.

    For how long have I used the solution?

    I have been using Sauce Labs since I started with this company, back in 2018, so that would be about three years and six months.

    I am an automation architect and my work includes providing cloud testing to different teams. Sauce Labs is one important piece for us.

    What do I think about the stability of the solution?

    It's pretty stable. We don't have any problems with the solution. We probably need more virtual machines internally, but that's not related to the performance or availability of Sauce Labs. When it comes to stability, everything is going great.

    What do I think about the scalability of the solution?

    The scalability is connected to our budget. The virtual machine concurrencies that we have are not great for our demand. I am talking with management about increasing the budget for 10 or 20 more virtual machines. But Sauce Labs itself provides the opportunity to scale very easily. For us, it's just a matter of budget.

    What's my experience with pricing, setup cost, and licensing?

    Sauce Labs has a pretty reasonable price and is worth it for the solution and what we are doing with it.

    Which other solutions did I evaluate?

    There are a lot of other options on the market and I have tried some of them in the past. But Sauce Labs has a lot of advantages. For example, one feature that I really love is that when we are doing troubleshooting or have a problem with test cases, they answer every single question so quickly and are very helpful. The customer experience using Sauce Labs is great in terms of support. That is an aspect other companies don't really have.

    I started with our company as an automation engineer and their support was able to help me with every single configuration, every single problem, and every single question. I cannot remember a single time that they were unable to help me. They are the best.

    What other advice do I have?

    The usage of Sauce Labs was not as great, when I first started using it, as it is now. We have a great approach to testing using Sauce Labs. We test mobile and we do live testing and we are running automated testing. We're integrated with CI. Nowadays, it's better than three years ago and I look for it to improve even more in the next year.

    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
    Flag as inappropriate
    Sarah Barefoot
    Senior Quality Assurance Manager at a financial services firm with 1,001-5,000 employees
    Real User
    The platform is easy to use and intuitive, so I can quickly navigate it
    Pros and Cons
    • "I like the dashboard and seeing the test results. As a manager, I like to see the insights of the people using it, understanding the total path and run. I can see all of that as a manager. I also know team members love seeing the dashboard and seeing the test results in real-time."
    • "I can't remove team members that have left the organization. I can only set them as inactive. It would be really nice to clean up my data and delete them from the team management."

    What is our primary use case?

    We use it for a lot of end-to-end UI test automation.

    We really just use the visual test automation, not the performance, for our product teams.

    How has it helped my organization?

    There are a couple of products using this solution, so it has been nice having a visual result. We can step through the tests, if necessary, and showcasing to the business that we are doing automation has been very helpful.

    What is most valuable?

    I like the dashboard and seeing the test results. As a manager, I like to see the insights of the people using it, understanding the total path and run. I can see all of that as a manager. I also know team members love seeing the dashboard and seeing the test results in real-time.

    Sauce Labs is optimized for automation and integration with the major CI/CD platforms and developer tools. That is just what they do. It is an easy, one-click solution versus having to manually run scripts.

    The platform is easy to use. I can quickly navigate it.

    Sauce Labs provide access to automated functional testing and visual regression testing from a single platform. It is very important to have one tool that can do this versus having multiple. Just integrating Sauce Labs within the application and seeing the pass-fail results right away is extremely helpful for teams. You don't have to go to some place else to find them.

    Everything is so intuitive.

    What needs improvement?

    I can't remove team members that have left the organization. I can only set them as inactive. It would be really nice to clean up my data and delete them from the team management.

    For how long have I used the solution?

    I have been using it for several years.

    What do I think about the stability of the solution?

    I haven't heard anything about stability. It has been pretty fantastic.

    Latency has never been a concern with Sauce Labs. Being in the cloud is super fast for any application, and we take advantage of Sauce Labs being in the cloud. If there was any latency, then it would be on the coding of Selenium versus the Sauce Labs application.

    What do I think about the scalability of the solution?

    We have about 60 products and only three or four are using it. We have started getting other team members' products to use and integrate Sauce Labs in their test automation, though I am 100% sure of what every single product is doing.

    How are customer service and support?

    The support is fantastic. We have monthly meetings with Sauce Labs. Even if I have questions, I can just reach out to them. If I have any questions, then all I have to do is reach out to them for help, and they are very responsive. If I have any questions on training, they are willing to set up training calls and train the team on Sauce Labs.

    The feedback is super fast. They are always there when I need help. I have never had an issue with their technical support.

    I would rate their technical support as 10 out of 10. They are constantly reaching out to see if we have any problems or need anything, but we are perfectly integrated with the Sauce Labs application and don't have any issues.

    If Sauce Labs doesn't suggest training, make sure that you ask for it, if needed. They are always willing to come out or even do virtual training. Whatever they can do to help with the process, they are always there. So, if you don't hear about the training, make sure you ask.

    How would you rate customer service and support?

    Positive

    Which solution did I use previously and why did I switch?

    We did not use a different testing tool before Sauce Labs.

    How was the initial setup?

    The initial setup was very straightforward. They make it so easy to be able to integrate Sauce Labs with Selenium. There were no issues at all, and if we ever had issues, the technical support would just help right away.

    What about the implementation team?

    We partnered with Sauce Labs. They came to our office for a few days and gave a demo to a large group of people on how to use Sauce Labs, but also provided a little training on Selenium.

    I liked the training. As a manager, some people hesitate using test automation because they just don't know how to start. By having some hands-on training with Sauce Labs team members, we were able to give that technical training to people so they understood, "Oh, I could do this. I can start this."

    They saw how easy it was to integrate Sauce Labs within Selenium code. The training just helps make it possible because I can't make people do things, when it comes to test automation. By being able to train hands-on and seeing the results, that made it possible for us to do test automation.

    What was our ROI?

    We have definitely seen ROI. We are utilizing Sauce Labs specifically for one of our product teams with several more being integrated with Sauce Labs. Seeing the benefit from even one team continuously using it, we are happy with the results that they are getting and having them sharing that knowledge. That is really how we will get other products on it, when people see that it is working for teams who are getting positive results. 

    What's my experience with pricing, setup cost, and licensing?

    I am not in charge of the licensing costs. They come and ask me, "Do we still need this?" Then, I tell the teams why we need XYZ and the licenses.

    We did initially go with Sauce Labs because of the pricing and integration.

    Which other solutions did I evaluate?

    I didn't make the decision on Sauce Labs. It was recommended by some other team members that did the research. People were looking at, "Can it help us from a visual point of view? Can it add tests, see all the tests through portals in the cloud, and make testing go quickly?" It offered the possibility of being able to run on multiple browsers and operating systems as well as had integration with Selenium.

    What other advice do I have?

    I have no complaints nor issues with Sauce Labs at the moment. It meets all of our expectations.

    Teams are continually using the product. I would hear complaints if there were any issues. I am trying to get teams to spread the word to other teams. So, the teams are very happy with it and trying to get other people to use Sauce Labs as well, then integrate that into their product team.

    Because our applications are not mobile, there are only certain browsers that we need to support. So, we are really not taking advantage of the number of browser/OS combinations, mobile emulators and simulators, and real mobile devices that it offers. I am just trying to get team members up and running with test automation and having some tools to be able to help them with that. I know that they are very competitive with other products in this regard, but that has not been an important factor for us.

    I would rate the solution as 10 out of 10. I love the capabilities of it, the support, and the service. 

    Which deployment model are you using for this solution?

    Public Cloud
    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
    Flag as inappropriate
    reviewer1398594
    Senior Software Development Engineer in Test at Autodesk, Inc.
    Real User
    Top 10
    Helps us in reducing the number of manual testing
    Pros and Cons
    • "Live device testing. As we all know, It's really hard and challenging to find/purchase many real devices to test because it will be costly and not all the team can be able to purchase all of the devices out there. We used to have a lot of real devices under our labs. However, it is really time-consuming to maintain those devices and make sure they are up to date with the testing requirements."
    • "As a web product QA team, we sometimes need to spot check some new child site on multiple browsers and OS(es). It was a little time consuming for us since we need to click on each of the browser/OS combinations and start a new session to test. Every sprint, with new features and child pages being added, we mostly need to do the same steps over and over again."

    What is our primary use case?

    We use Sauce Labs for mostly our automated testing on cross-browser and emulator devices. Our team mainly focuses on web product testing so using a third party vendor to help with the external resouces is a must. There are many cases that we need to run our test on multiple browsers like Chrome, Firefox, IE, Edge etc. Sauce Labs has all the options for us. Of course, because our products are mostly web based, we need to ensure the cross-browser testing for every release cycle. Instead of doing it manually on many real devices, we use Sauce Labs since they provide many emulator devices.

    How has it helped my organization?

    Sauce Labs has been helping us a lot in reducing the number of manual testing. By moving most of our test to the cloud and not running on the EC2 instance, we not only save time for the service but also the maintenance for the external servers. Sauce Labs is the best out there. They beat their competitors because of its price, customer support, and an amazing forum where you can ask and get answered really quickly. They never stop expanding. Whenever we have a new update for any browser, devices, etc. they make sure it will be in their devices and browsers farm so users can use it immediately.

    What is most valuable?

    Live device testing. As we all know, It's really hard and challenging to find/purchase many real devices to test because it will be costly and not all the team can be able to purchase all of the devices out there. We used to have a lot of real devices under our labs. However, it is really time-consuming to maintain those devices and make sure they are up to date with the testing requirements. Sauce Labs knows that and already has a solution for their live device testing which allows us to test our app on the actual devices out there and see the real result.

    What needs improvement?

    As a web product QA team, we sometimes need to spot check some new child site on multiple browsers and OS(es). It was a little time consuming for us since we need to click on each of the browser/OS combinations and start a new session to test. Every sprint, with new features and child pages being added, we mostly need to do the same steps over and over again. We would like to see if Sauce Labs can be able to open multiple different browsers on the same website at the same time, just by 1 click. It will reduce a lot of time for manually click and change the devices/OS combination.

    For how long have I used the solution?

    2+ years.

    What do I think about the stability of the solution?

    We rarely had any downtime with Saucelabs, except sometimes our tests cannot connect through the tunnel. It's not always but once in a while and with some random tests only. Their support is really great, they always help us to diagnose and find the root cause of the problem. 

    What do I think about the scalability of the solution?

    The performance is great, we never had any slowness or trouble with running our tests.

    How are customer service and technical support?

    Amazing! They are very quick in response and very throughout. They even go way beyond to help troubleshoot our code and give the coding example to help with all the questions that we had.

    Which solution did I use previously and why did I switch?

    We did use another third party vendor for cloud testing. However, their price increased a lot, and their support was a little slower than Sauce Labs. That's why we switched.

    How was the initial setup?

    It was very easy. They have a wiki and a GitHub repo with all of the codes for every programming language. We can either use their template page or inherit it to our use cases.

    What about the implementation team?

    We developed our test framework based on Sauce Labs in-house. There was no other vendor team.

    What's my experience with pricing, setup cost, and licensing?

    We have a team that manages all the licenses, and they share the pool with the rest of us. However, we haven't had any problem of running out of concurrent devices/connection because we agree to not running many test in parallel at the same time.

    Which other solutions did I evaluate?

    There are only two big companies in this field and since we were moving away from the other one, we obviously did not consider any other options.

    What other advice do I have?

    If you are looking for a cloud devices testing provider, look no further. They will beat their competitors on price, support, and a lot more. 

    Which deployment model are you using for this solution?

    Private Cloud

    If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?

    Amazon Web Services (AWS)
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    reviewer1396422
    Senior QA Automation Engineer at Bleacher Report
    Real User
    Top 20
    Analytics have drastically improved over the years and you can dive into your test results (pass/fail) to get a better understanding of how things are working
    Pros and Cons
    • "I have found the live test section with Sauce Labs to be extremely valuable. When you can't quite figure out why a test is failing, you can go to the live test results section within their tool and launch your test (specifying a given OS/browser, or device) manually and step through the test to see the issue more clearly, usually opening up the developer's tool console and watching the network calls and console (within Chrome) to usually find the underlying issue."
    • "Overall, I think Sauce Labs provides us with a valuable tool and resource. As far as what could be improved, I would say the overall test execution time. Some of the calls take a bit longer than I expect, for example in web browser tests; while the execution time isn't obnoxious, it could be improved so that overall tests/test suites finish faster."

    What is our primary use case?

    Our CI/CD (Continuous Integration / Continuous Deployment) QA Pipeline interacts with Sauce Labs when it goes to run the necessary automated tests. Our automated tests exist for normal web browser tests (we specify to run on the latest Chrome version, on Mac OS) as well as mobile tests where are mobile tests run on various Android and iOS devices to verify that our apps are working properly on multiple systems. We use a 30 VM (Virtual Machine) farm from Sauce Labs, which gives us enough VM's to get through all our tests in a very reasonable amount of time.

    How has it helped my organization?

    I use it to maintain my own VM (Virtual Machine) farm of 25 VMs. The admin time I had to spend on maintenance and upkeep was a few hours per week, coupled with the fact that the up-time on the VMs was probably somewhere around 75% due to issues coming up in off-hours when I wasn't available to due maintenance/fixes/repairs. When I finally moved over to Sauce Labs, all this burden of doing the management was uplifted off my shoulders, and the up-time is nearly 100% for Sauce Labs VMs. On top of that, we can specify from an abundance of device types, OS's, browser versions, etc. when running our tests. Furthermore, the analytics side of things within Sauce Labs has drastically improved over the years and you can dive into your test results (pass/fail) to get a better understanding of how things are working over a period of time. Overall, Sauce Labs has provided us with a VM farm that is virtually always up and running and has taken the burden off of us/myself of maintaining the VM farm.

    What is most valuable?

    I have found the live test section with Sauce Labs to be extremely valuable. When you can't quite figure out why a test is failing, you can go to the live test results section within their tool and launch your test (specifying a given OS/browser, or device) manually and step through the test to see the issue more clearly, usually opening up the developer's tool console and watching the network calls and console (within Chrome) to usually find the underlying issue.

    I also frequently use the insights section within the tool, to analyze trends of the pass/fail rate of our tests, so I can see what suites of ours are more failure-prone than others, etc. The tool allows you to filter up to the last 30 days' worth of data, along with many other filter options (OS type, tag, build, etc.) so you can really produce insights into your tests.

    What needs improvement?

    Overall, I think Sauce Labs provides us with a valuable tool and resource. As far as what could be improved, I would say the overall test execution time. Some of the calls take a bit longer than I expect, for example in web browser tests; while the execution time isn't obnoxious, it could be improved so that overall tests/test suites finish faster.

    Within the insights section of the Sauce Labs tool, I think it would be nice if you could go back farther than 30 days worth of data. I like all the custom filters available, but being able to go back say 90 days (or more) would show more statistics on our data/test results.

    In the next release, I'd like to see continual new additions of OS's. This area is usually pretty up-to-date, please keep at it!

    For how long have I used the solution?

    Five plus years.

    What do I think about the stability of the solution?

    Very good, up-time is near 100%.

    What do I think about the scalability of the solution?

    Very good, you can add Vs if necessary.

    How are customer service and technical support?

    We've only usually contacted customer service (our account manager) to add VMs, it's been seamless and pleasant.

    Which solution did I use previously and why did I switch?

    Manually maintained my own VM farm. We switched to Sauce Labs as the cost/time to maintain my own VM farm was too high.

    How was the initial setup?

    Straightforward and easy, with good instructions in their knowledge base.

    What about the implementation team?

    In-house.

    What was our ROI?

    I'd say it definitely provides a good ROI, vs maintaining your own VM farm.

    What's my experience with pricing, setup cost, and licensing?

    The cost is worth it. Start with a VM farm that you think suits your needs, and then add/subtract VM's if necessary.

    Which deployment model are you using for this solution?

    Private Cloud

    If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?

    Amazon Web Services (AWS)
    Disclosure: I am a real user, and this review is based on my own experience and opinions.