What is our primary use case?
We create banking software and we use this product for testing. We have different business units and we set up an enterprise license, so everybody feeds into it from each of our business units.
We have about 750 websites and approximately 50 mobile apps, and we will test the different types of browsers against our automation. A good section of the work we do is running automation against different combinations, and that will expand into mobile devices once we kick off the new year.
For the most part, it's heavy automation, but there is also testing that is manual, where they can log in and pick their devices or browsers.
Our environment includes VMs on the cloud, as well as public and private devices. we have the CrossBrowser and we have the private and public cloud.
How has it helped my organization?
Sauce Labs allows us to expand our coverage without having to run them separately against different browsers on your local machine. You can just spin up an instance and run against whatever browser and operating systems combinations you need.
Especially on the mobile side, things are improving. We had an on-premises lab, where we were managing the physical devices on a device cart. That's going away, as we are now able to use the public cloud offering, which has 2,000 plus devices.
For locking down devices that we prefer to use without registering with other clients, we use the private Cloud. We split up the distribution 50/50 between private and public.
Sauce Labs is what enables our different testing combinations. We run our different suites that include smoke tests, regression tests, and functionality testing. All of these types of testing need to be verified against different browsers and mobiles, and this is our solution to that problem.
This product helps us to increase test coverage by testing both functional and visual aspects of the UI. This is very important to us because we have different types of customers. Some are end consumers, whereas others are banking customers. We won't release software without validating all of our use cases.
Sauce Labs helps us to deliver new features and products to the market. It's part of the use cases that we have to validate, so we need tools to complement our test cases. We're trying to eliminate manual testing or how we kick off scripts. We want to be able to automate testing of our suite of use cases against our devices and operating systems.
What is most valuable?
The most valuable feature for us is the browser. That is all we're using right now, although come to the start of the year, we will be using the mobile feature.
The most critical thing is that this software aligns with our Agile and DevOps way of doing things. It integrates with kickoff scripts through DevOps. Time to market is key, which means that we can't have a lot of manual interventions, and this fits into our automation program.
What needs improvement?
We had some specific features that we opened tickets on, although they were not earth-shattering. For example, the way the menus scroll could be improved because it does not have a bar, the way that people are used to, where you can move up and down. It is something that you have to flick in order to activate.
For how long have I used the solution?
We have been using Sauce Labs for between three and four years.
What do I think about the stability of the solution?
This solution is very stable. It's generally up 24/7.
What do I think about the scalability of the solution?
We have about 12 business units that use the service.
Scalability is good on the public Cloud. You've got all of the different devices and most of the operating systems, including beta. It's all that we can ask for in that respect.
The browsers are pretty straightforward. They usually deploy any new versions within 48 hours of the operating systems for the browser side.
We have a thousand registered users and their roles are a combination of traditional QA testers and developers. To support them, we have a small team that I call admin support. There are only a couple of us on my team and if there are any issues, we centralize that before deciding if we need to open a ticket up to the Sauce Labs support center.
Our small admin team works very well. Two people are sufficient to handle the training, education, and any configuration changes. In our company, one of them is the lead over all of the different tools and we license from different vendors. This person is a little bit more technical when it comes to the Sauce Labs configuration setup. The second is an admin for the end-users, teaching them how to use the tool appropriately. This person is also responsible for guiding end-users on how to test applications properly.
How are customer service and support?
The sales group was easy to work with. We had a service account representative, which is a nice setup. They handle the day-to-day activities and support needs, which helps to make us successful by answering questions and giving us solutions or enhancements.
We meet with our service rep on a weekly basis and if we have any issues, we can discuss those solutions and opportunities. Basically, we centralize the communication flow between a small admin team under my team for the organization, and it seems to work very well.
Overall, I would rate the customer service an eight and a half out of ten.
Which solution did I use previously and why did I switch?
We merged as a company two and a half years ago, and we had licensing agreements with BrowserStack, Kobiton, Sauce Labs, and SeeTest, which is part of Digital.ai. Some of the solutions were on-premises for the devices and we wanted to move to the cloud to simplify the infrastructure.
We moved forward with Sauce Labs because they offered both public and private cloud options. The other ones did not necessarily have that combination. It's a win, having private and public both, as it is important to companies like us.
How was the initial setup?
The initial setup was very straightforward.
We get a service rep that guides us on how to set up the tunnels to access our intra-sites, and then the device procurement is pretty straightforward. You give a list of what devices you want and they are put into each of the data centers, San Jose or Germany.
They do offer training as part of the agreement, as well, and we're in the midst of doing that because we're rolling out the mobile component in January.
Everything that I've needed, they've been there.
In terms of how much effort and how long it took to deploy, because it's on the cloud, the majority of the effort is on the Sauce Labs side. We've had some basic internal procedures to follow, but that doesn't really align with Sauce Labs. This included things like security approvals and stuff like that but overall, it was very little effort on our side.
Our implementation strategy was pretty straightforward. They make it available to us and then we do the training for the end-users. We have a registration list of about a thousand people, to which we have provided demos and documentation. Sauce Labs has a very robust document site. It's a combination of self-service education for the testers and developers in training. So, that's essentially our rollout plan.
We had to rightsize how many licenses we needed, based on user consumption, which was part of the planning phase. We did a full pilot where Sauce Labs gave us an area to access the mobile environment, and we ran our use cases before selecting the software.
What about the implementation team?
We deployed in-house and we are happy with the support provided by the vendor.
What was our ROI?
The licensing agreement that we have has reduced our cost through solution consolidation by 30% to 40%. I don't have a precise measure of how much it saves the end-user because I don't have a good metric for it. That said, it expands our coverage and saves us from having to kick off the scripts against each of the different combinations. I would estimate that we save maybe 20% in this regard.
In total, considering the bundled discount, we're going to save several hundred thousand dollars per year in licensing fees. The vendor has a tiered pricing structure and by going to one vendor and having enough licenses, we were able to get a more competitive price.
What's my experience with pricing, setup cost, and licensing?
With respect to pricing, they did a bundled discount because we went with Sauce Labs for both mobile and browser. They were very competitive on pricing and provided a bundle discount for us as a larger customer.
I like the licensing model because it is a system of shared licenses. This is different from BrowserStack, for example, which I didn't like because they charge on a per-user basis. This matters to us because we have a situation where there are heavy users and light users. In this case, we prefer to have shared licensing.
Shared licensing is like having a seat at the table and when we have a thousand registered users, it is easy to understand that some are heavy users and some are very light users. With the shared licensing, we don't have to manage the registration of whether they're heavy users or not. We don't worry about that. Rather, we just worry about the consumption of the licenses, which are shared amongst all.
There are no costs in addition to the standard licensing fees.
Which other solutions did I evaluate?
We recently went through consolidation to get rid of BrowserStack, Kobiton, and SeeTest. We chose Sauce Labs over the other vendors and consolidated to one.
With Sauce Labs, we did a full pilot before selecting it. The selling point and the reason that we ultimately chose this product is because we had our own service account representative that helped us to be successful.
The features met our needs, which was the other evaluation point against the other software providers. They had both browser and mobile solutions, and it's easier to work with one vendor versus four.
What other advice do I have?
We have usage reports, so we're taking the logs and evaluating that over time. Because we're consolidating from four tools to one, we considered the usage reports of all four of the vendors and came up with what our go-forward is. We expect our usage of the product to increase a little bit over time; maybe a 10% increment as we move forward and get more teams engaged.
Although a 10% increase is in order, we won't purchase licenses without having the data to support that. We will have detailed usage reports for each business unit, which is what will help to determine how much we expand our usage of Sauce Labs.
My advice for anybody who is considering this product is that they need to understand their solutions and match it up against what is provided. Sauce Labs is leading in the area of both mobile capabilities and CrossBrowser and I would highly recommend looking into Sauce Labs. My decision was to choose Sauce Labs over those other leading contenders.
The biggest lesson that I learned from implementing this solution had to do with what our options were. We had to do a comparative analysis that included breaking down all of the different features and comparing them against those other tools, to make sure they satisfied the features and functionalities we needed. Not having that knowledge across those vendors completely, we were able to do that through a pilot. Essentially, we came away with knowing the nitty-gritty by running the pilot and doing the feature analysis. As such, we were able to make sure that Sauce Labs met our needs.
Overall, we are happy with this solution. We validated most of the functionality when we ran our pilot. Nothing critical stands out in terms of improvements that it needs.
I would rate this solution a nine out of ten.
Which deployment model are you using for this solution?
Public Cloud
Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.