The True Cost of Free Testing Tools

Open source testing tools carry a certain appeal to testers – there’s no doubt about that. However, there is a pretty intense debate out there between open source testing tool advocates and in-house/vendor testing tool advocates. Both camps provide equally valid points.

But, we have to wonder, what is the true cost of using a free open source testing tool? We are going to present the reasons why open source tools are appealing, why “free” comes with risks and the importance of choosing the right tool for your tests.

The Appeal of Free Tools

1. No upfront cost

Obviously, this is the reason that most testers try out free load testing tools. The tools require no licensing fees and so, when you are running on a tight budget trying to push multiple applications out the door, “free” appears to be a great option.

And, some people would agree with you – like Goranka Bjedov, a capacity software engineer at Facebook, who slams vendors like LoadRunner and SilkPerformer in her Google Tech Talk for being overly expensive and using proprietary scripting languages. This gives people all the more reason to seek out freebie options.

2. Open source allows for customization

Sometimes the tool you are using doesn’t do exactly what you need. Or perhaps there is a unique aspect of how your application behaves that demands a slightly different approach to performance and load testing. When you are using an open source tool, you have the ability to open up the inner workings of the tool and build out custom functions and behaviors as you need them.

In an active open source community, many of these enhancements will get fed back into the main body of code, making customizations available to everyone. This frequently adds new and interesting features into the mix – especially when it comes to adapters, plugins, language translation, and system interoperability.

3. Works for a lot of cases

If the architecture of your application is fairly simplistic, you are testing a relatively small number of concurrent users, or the function of the application isn’t critical to your business, an open source load testing tool might work fine for you. Of course, you still have to decide from where you’re going to generate the load and be sure the tool supports the protocols used in your application in order to get realistic results from your tests.

Bjedov says, “The vendor tools support far more protocols than the open-source tools, but as long as you are staying in the web, and you’re looking at HTTP/S and POP3, then open-source tools are pretty good.”

Why Free Can Be Risky

But, there’s a flipside to everything. Although free open-source tools have their upsides, they also come with their own set of risks you should consider.

1. Free tools might not be as free as you think

Free testing tools are like free puppies. Think about it. You get a puppy as a gift, but that puppy lives for 18 years or so and requires food, vet visits, training classes, visits to the groomers and a whole lot more basic necessities to keep your furry friend healthy and happy.

Well, free testing tools are not far from that. There isn’t an upfront cost, but over time you need to pay for the upkeep. Many organizations run into high development, administration and implementation costs over time using free testing tools.

What it comes down to is determining where you want to invest your time. There is a tradeoff involved in using open source testing tools and it’s paying someone to upkeep it for you, versus dedicating the necessary time to upkeep it yourself. And, if you see time as a valuable asset to your organization or development team, then open source tools probably aren’t for you.

2. Script maintenance can turn into a nightmare

As described above, the ability to customize an open source testing tool can be a great advantage. But it also has a downside because maintaining those customizations can be super-difficult for testers.

Typically testers using an open-source tool spend most of their time on the test scripts they write. But from time to time you may actually need to change the behavior of the testing tool itself to accomplish your objectives. To do this, testers edit the source code of the tool itself – or get a developer buddy to do it – and use this customized version of the tool to test their application.

But people tend to be less deliberate when making changes like this that are not part of the core application code. They don’t always document the changes they make adequately, or they avoid using descriptive comments. After some time goes by – and maybe you’ve cycled through a few people in the organization – any knowledge about your customizations is lost.

Furthermore, occasionally your changes will conflict with changes in new releases of the tool, making upgrading and maintenance a challenge. You may have to manually merge your old, uncommented, undocumented code into a new release before using it. What a pain!

This is a very common scenario, especially in organizations that aren’t active participants in the open source community. In the end, you lose a lot of time just trying to figure out why that person before you did what they did – and how to maintain it going forward.

3. Lack of professional tech support

There are open source communities out there that are very responsive and many argue new features are rolled out faster than vendors. But, you can’t really count on the community too heavily because it’s not their job. They aren’t getting paid to work on the tool, and if you want the tool to have a feature that does X-Y-Z, you have to wait until someone in the community decides to implement that feature.

A vendor, on the other hand, has a dedicated team whose number one job is to roll out the specific new features that customers are asking for. With a vendor, there is a high incentive to keep the tool as up-to-date as possible because it’s what the customer wants. While in an open source community, there is no high incentive to keep the tool functioning the way others want it to, making it extremely difficult to use the testing tool efficiently.

Tips For Finding the Right Tool

Looking at the appeal and risks of free testing tools you now have a better picture of what kinds of tools are right for you. Here are a few of our tips to help you make the right decision.

1. Always get experts on your side

No matter what tool you choose for your testing needs, open source or proprietary, always consider who is on your side. A community of fellow developers or established experts in the load testing industry might make the difference between a successful application and a failure.

2. Matching the tool to the skill of your team

It’s important to consider the skill level of your team and the type of tool you are interested in using. If you have a lot of skilled developers on your team who are familiar with the open source community and can deliver changes to code quickly, then open source might work for you. If you don’t, open source load testing tools could be a nightmare if you need something more from those tools than what they currently offer.

If you have a complex and extremely important load test and your developers don’t have the extra time to maintain the open source tool, a vendor might be the better option for you.

3. Determine what’s more important to the organization

What’s more important to you? Time or money? What this choice basically comes down to is this: do you want to avoid the licensing fee but potentially dedicate a ton of time towards making the free tools work for you? Or, do you want to have someone committed to managing all the pesky maintenance upkeep for you, allowing you and your team the time to focus on results analysis, application development or anything else that is valuable to the success of your application? Hopefully you’ve invested in talented people who you think can deliver more value when they’re allowed to focus on their core competencies.

We are here to help

No matter what tool you choose to use, vendor or not, we hope this post gave you a little more insight into the appeal and risks of using some of these open source tools out there. If you find yourself stuck, we’ve got a team of load testing experts who’d be happy to answer your questions.

No Responses
  1. March 6, 2014

Leave a Reply

Your email address will not be published. Required fields are marked *