The True Cost of Free Testing Tools

The open source movement has opened up a floodgate of software that costs nothing more than the time it takes for it to download. Free open source applications like LibreOffice have become viable alternatives to the mainstream Microsoft Office. More programmers are using Visual Studio Code (free from Microsoft), as their integrated development environment preference. Offerings such as JMeter and Selenium are commonplace among QA/testing community. The number of free applications and tools available today can, at times, seem endless. But, just because it’s free, doesn’t mean that it comes without cost.

Free testing tool usage usually brings with it a trade-off. You save on the purchase, but the costs can be significant in the “time” it takes to get to acceptable levels of productivity. Sometimes testing software can cost less. It all comes down to context.

Free Open Source has Benefits

Free open source testing tools offer benefits. The primary value is that you don’t have to make a cash purchase to obtain the tool. This can be good, especially if you’re a startup with limited capital.

Your benefits increase when the free tool is easy to use enabling the tester to be up and running quickly. “Short period” context will vary by company. For some, it’s a half a day. Others can expect fruitful results in a few days.

Testing activity can be complicated. Hence, supporting ease of use goes beyond merely firing up the tool to start your testing. Education is required. So, for a free testing tool to be cost-effective, there needs to be clear educational documentation as well as training how-to video. It is important to note that the documentation must be accessible. Many companies tend to miss this simple, straightforward concept. And, providing an impromptu screencast isn’t going to cut it. Clear intention and instruction need to be articulated.

Another free testing software advantage is that it provides a means to avoid vendor lock. Many commercial tools are proprietary. For instance, one tool might have a unique scripting language or a specific model to organize and conduct testing. In this example, as the tool becomes a mainstay of the testing environment, a company can find itself at the mercy of the particular vendor. The vendor can add, modify, or remove a feature without any notification to the user. They can even raise the price. Given the proprietary nature of the product, a company is limited in how it can address these types of changes. The choice is often black or white – accommodate the tool or abandon it. In either case, the result will impact operations.

On the other hand, by design, open source testing tools adhere to general standards, and many are extensible. For example, a preferred open source option such as JMeter allows third parties to develop plugins that extend the capabilities of the tool. In the case where support for current need is absent, a developer will work with the JMeter API to create a plugin to meet the demand.

Although a commercial vendor may contribute to an open source project (e.g., Microsoft’s Visual Studio Code), the project is independent of the vendor. Developers are free to modify the project to meet their needs. And, the project is not at risk of collapse should the vendor stop contributing. The long-term viability of the project depends on the strength of the community the project serves. In the end, companies using open source testing tools avoid vendor lock because no one vendor owns the project. The company who use these tools commits to the community, not the vendor. The project’s community is the organizing agent.

Sometimes it Pays to Pay

As compelling as the free open source argument may be, when it comes to operation and support guarantees, you’re on your own. Open source is the butt of the joke in many IT shops today: “Here’s some code. It might work, it might not. You figure it out.”

As cynical as this humor is intending to be, it’s true. Most open source code comes with no guarantee, be it operation or support. Typically, support is the way that companies associated with an open source project make money. MongoDB delivers an open source version of its popular document database, which you can download for free directly or from GitHub. The company sells an enterprise version of the product that provides a host of additional features. One of the highlights is 24/7 technical support with a one hour response time guarantee.

Many companies are happy to pay the price for enterprise code, especially if it means that when an engineer gets a 3 a.m. distress call will be able to get things back on track quickly. There are fewer situations more exasperating than a piece of open source code failing during a mission-critical case. The usual fix when disaster strikes is to post a query on Stack Overflow or other community sites asking for help to solve the problem. Hopefully, a correct answer will come in time. You never know. But, when you’re paying for support, you do!

And then there’s the question of operational guarantee. A company selling broken software won’t be around very long. A developer might tolerate bugs and defects in an open source product. For something purchased, well, that’s another story. Given the nuances of a licensing agreement, should a piece of commercial software fail, there are real legal ramifications. The manufacturers buy insurance to help minimize the risk/stress associated with product malfunction. The purchasing companies are protected. Open source projects are typically not insurable and tend to be used at the operator’s risk. The more popular open source licenses such as ISC, MIT, and BSD explicitly release the creator from having any product liability, pushing the exposure back onto the user.

Putting it All Together

The open source movement is a game changer in the landscape of software development. It’s placed a lot of great code in the hands of welcoming programmers, and motivated, gifted developers to make significant contributions. One such developer is Doug Cutting, the creator of Lucene and Hadoop.

“Open source seemed to offer the option to have the software that I’d written, this particular one, Lucene, live on and have the opportunity for people to use it.”

However, open source software is not a panacea. There are risks, particularly around code associated with projects that have neither a long history nor large user base. In such cases, the software can be a crap shoot. The download may be free but could cost hundreds of thousands of dollars in the labor it takes to get it to work.

As compelling as open source software might be, sometimes it pays to pay for it. This is true for mission-critical undertakings. In many ways, funding for the expertise and guarantees accompany a software purchase is like buying an insurance policy. Some say that the best insurance money buys the kind you never use. Should the need arise, few people are sorry to have paid the purchase price.

Learn More

Discover more load testing and performance testing content on the Neotys Resources pages, or download the latest version of NeoLoad and start testing today.


Bob Reselman 
Bob Reselman is a nationally-known software developer, system architect, test engineer, technical writer/journalist, and industry analyst. He has held positions as Principal Consultant with the transnational consulting firm, Capgemini and Platform Architect (Consumer) for the computer manufacturer, Gateway. Also, he was CTO for the international trade finance exchange, ITFex.
Bob’s authored four computer programming books and has penned dozens of test engineering/software development industry articles. He lives in Los Angeles and can be found on LinkedIn here, or on Twitter at @reselbob. Bob is always interested in talking about testing and software performance and happily responds to emails (

Leave a Reply

Your email address will not be published. Required fields are marked *