What is most valuable?
- The ease of use of the object repository.
- How quickly you can get up and running using a powerful programming language.
- Powerful IDE and Visual Studio integration.
- Pricing, particularly the runner license.
How has it helped my organization?
Six years ago, the only real alternative was QTP. Having previously used other tools, including Rational Robot, Winrunner, TestComplete and others over the years, we wished to move away from the typical VB scripted approach. Selenium was in its infancy and still a basic tool. QTP, whilst still good, was struggling to keep up with new technologies, despite still being a market leader. For us QTP’s antiquated IDE was no longer adequate and VBscript did not offer the power of newer programming languages.
Ranorex at the time was the only solution able to easily handle the new Ajax web implementations of the day. Our company had just moved to ExtJS 2 and QTP did not provide support for that technology. HP support was not able to rectify the issue whereas Ranorex worked out of the box.
Ranorex support was great also. Support responses were always quick and they were able to work with us to find solutions, weather they were solutions out of the box, or construct solutions for us if they weren't yet available.
For people starting out in automation or even people who have been doing it for years, Ranorex provides an automation solution that can get you up and running with limited fuss. Whilst we no longer use the solution in the way they would intend i.e. we’ve abstracted all of their functions within wrapper classes and simply use Ranorex for interacting with the application, I still believe it is a great tool that caters for a wide range of technologies for testers that are new to automation.
For how long have I used the solution?
I've used it for six years.
What was my experience with deployment of the solution?
I’ve always been impressed with how easy it is to install and maintain. It’s a quick installation and you’re on your way. These days they have inbuilt test management which previously was not available. As such we wrote our own test management tools and framework so I can’t comment on that side of the application as I’ve never used it.
How are customer service and technical support?
I’ve always found their support second to none. Responses to my questions were answered promptly and their technical staff are exactly that, extremely technical which is refreshing given the generally basic support previously experienced from other vendors.
Which solution did I use previously and why did I switch?
Over the last 18 years I’ve used many products ranging from Rational Robot, Winrunner, Test Complete and QTP to now using Ranorex and lately Selenium. I’ve also “played around” with other tools such as WATIR, RFT, SilkTest and lately cucumber etc but I ultimately chose Ranorex because of its price point, hence ROI. It was a fraction of the cost of other commercial tools, yet had the features we needed and was quick to get up and running.
It had a “modern” IDE that was user friendly, I could develop in Visual Studio and it was in a powerful language that most of my team was familiar with.
Our company has recently made a decision to use Selenium also. This was purely based on its cost however. Whilst Selenium these days has become a powerful automation alternative, it is still really limited to people with previous automation experience and with a strong programming background if you wish to achieve similar ROI compared with the commercial tools. For us this is now the case, hence the move.
Selenium however lacks the inbuilt IDE and tools that the commercial solutions have and is still essentially a group of libraries. It does have a strong user base however, hence lots of examples are available in Java. If you don’t have a programming background it can be time consuming to come to grips with however. Selenium really requires a framework to utilise it efficiently, so if you don’t already have one you’ll either have to learn and use an existing framework or build one which will be time consuming. You also need to hook into other tools such as TestNG or similar to get consistent reporting approach (these days UI testing is but a small part of a bigger picture; Unit, Web Service, UI testing etc). As such the tool which started off sounding good because it was free is now incurring significant cost as a result of the lack of inbuilt tools to get you up and running quickly, and the skills learning curve.
I think skilled users loose sight of how much they’ve learned over the years, so whilst Selenium is great and easy to pick up if you’ve already got a strong development background, it’s not a good choice for teams that lack those skills. The commercial tools allow users to walk before they can run so to speak.
Ranorex still has the best pricing point for bang for buck I believe with the runner license availability being a big selling point. When we shifted from QTP to Ranorex six years ago we did so because we were able to purchase 39 Ranorex licenses with the budget we had for the maintenance of our four QTP licenses, and the tool worked on our technologies whereas QTP didn’t anymore. It was a simple decision to move.
How was the initial setup?
The initial setup was straightforward. Just install it and you’re off and running. The software installation was also quick compared to other products. Admittedly we had an inbuilt framework that we’d built available to us so we just wired Ranorex into that framework.
What about the implementation team?
We implemented using an internal team. One thing I would say is that if you have the choice I would always get your automation completed by a professional team rather than manual testers looking to become automated testers. Whilst the costs may seem higher initially I’ve yet to see a manual team introduce robust, maintainable automation anywhere near the timescales that a professional team can achieve. Automation is an art.
What was our ROI?
ROI isn’t just limited to pricing/licensing. Whilst it is an initial selling point, resource availability, skill requirements etc is what I see as the fundamental cost savers toward your ROI.
Maintenance is a cost killer in automation, so if you haven’t implemented a modular, data driven framework (which requires a skilled team) then I still believe the commercial tools will provide you with a better ROI. As I’ve said above however, if you do have a choice, hire a skilled team as then you will be able to get away with using free tools such as Selenium. The resource will have experience with those products, hence will also be able to implement a maintenance efficient framework in a cost effective manner.
What other advice do I have?
Skilled resources! I can’t push that point enough. You need at least one highly skilled resource to be responsible for the architecture of your framework as long term maintenance will be your largest cost, followed by actual implementation time.
A skilled resource can then help to transfer their knowledge across to less technical resources. Ultimately a good architect will aim to abstract the technical as much as possible to enable non-technical team members to also assist with the automation process. People ofter refer to this as a “script less” approach. Whilst this is nothing new, there is still a lot of contention around this topic, particularly from automation “guru’s” as generally speaking most framework still required a lot of coding, and “scriptless” frameworks simply don’t provide the flexibility of their coded counterparts.
I do believe however that it will become the way of the future and is achievable by continuous abstraction of functionality within your frameworks to eventually get to a point where hardly any code is required to “build your script”. Essentially only data is required to run your automated scenario. Not just data driven in a sense of providing your input/expected result data, but data driven at the object/automation artefact level. Also the more that you can automate the process of producing that data, the better. It is similar to the modularisation of code, only you’re not modularising data.
This is something we’ve achieved for our company. I refer to it as “Model Based Automation” as we use a model hierarchy for managing all of our application objects. A data dictionary as such. This model can be built manually, or automatically scraped from an application using rules. This has the hidden advantage of also enabling you to automatically track object interaction coverage across your application under test via automation. This has enabled us to reduce our code footprint from over 250,000+ lines of code to just 6,000, whilst automating most of what would usually be a manual process of producing code.
Once again however as you can see we’re now referring to frameworks, and not the tools themselves. Obviously building the frameworks themselves takes time and skill. The real skill is making the frameworks generic enough that they are no longer application dependant.
Disclosure: My company does not have a business relationship with this vendor other than being a customer.
