Open Source vs. Commercial Testing Tool TCO Comparison

When Scott McNealy of the late Sun Microsystems said in 2005 that “open source software is free like a puppy is free,” he was onto something. The most costly thing about a puppy isn’t the initial cost; it’s everything else that it demands during its life. For this blog, we will focus on open source software (OSS) and how testing tool evaluation requires a total cost of ownership (TCO) analysis.

Initial OSS testing tool cost can look attractive to an organization, but the TCO  what it costs over its lifetime – can be overwhelming once evaluated. The hidden costs of using open source may not be evident at the beginning of the decision process, but those costs will always be present over the software’s lifecycle.

The thing about TCO is that it reflects more than just the balance of the direct qualities of competing for software products (price, functionality, reliability, etc.) and the relationship of the software to the organization’s broader set of technology platforms, installed systems, culture and skills base, and strategic goals. The ability to access market and community-based services and support is also a part of the equation.

Initial Costs

The first cost associated with OSS is due to the need for change, so it fits into a company’s specific situation and platforms. While a vendor will usually provide a product that does not have to be modified at the start, open source software will most likely require configuration or extra development to make it fit into what it will be needed to do. OSS is general in scope by design. Making it fit into a particular situation will require that efforts in programming, testing, and validation be made. The organization will bear the cost for all of that.

Vendor products include appropriate tech support baked right into the purchase price. If any tweaks need to be done to the product before deployment, they will do it, and the customer won’t have to. Vendors also have an incentive to provide any needed support because they want the customer to be satisfied and continue to purchase their products. There is no such incentive for the developers of OSS. The effort they contribute stops when the software is published.

As part of their ongoing effort, vendors are more likely to innovate around newer, emerging technologies, platforms, and protocols as they affect the product. Changes in the IT landscape can change how a vendor’s products will be utilized by the customer. A vendor is far more likely to respond to changes via customer request vs. OSS because they want to keep the customer engaged with the solutions they provide over the long-term. They feel the pressure of competing with other vendors in the marketplace, and it motivates them to fulfill requests promptly.

The pressure of competition between vendors can also provide an organization with a decrease in latency when a vendor responds to their requests. It’s always going to be a faster path to the needed information when a motivated participant is involved in the discussion. A quicker response can end up solving a problem without the need for a company’s staff to have to spend the resources to reinvent the wheel out of sheer frustration.

OSS Support

OSS does have its support communities that cluster around each project. The communities are notoriously variable, with some acting in responsible manners and others far less so. The behavior that each community displays will affect how long it takes end users to receive answers to their support questions if the support community can even answer it in the first place.

Since support is done by a community, conflicting answers to the same question may arise. One member may think one way, but another will likely have a different perspective. In turn, this can lead to confusion within the requesting organization about the correct way to proceed in a situation. Resolving this will add to the TCO since the hidden cost involved will be the staffing effort the organization has to spend on evaluating the applicability of the answers it receives to its unique situation.

OSS support may also give incorrect answers. The support community may not have the needed expertise in the organization’s platforms, for example. Instead of the notion of intentionally prescribed inaccuracy, the wrong answers may be symptomatic of inexperience in using the OSS on those platforms.

Being able to respond may be limited in OSS. In many ways, support for changes in the field will occur when the support community has enough motivation to execute the updating. There are usually no formal upgrade paths specified in the lifecycle of an OSS project. Upgrades are reactive, tend to happen when they are “felt” to be needed. This stymies organizational growth and upgrades planning because they have no definite idea when an OSS upgrade will occur.


An organization takes the security challenge on itself when it deals with OSS. Vendors will consider this as part of their lifecycle effort and allow for it. But OSS requires the continual evaluation of its security status considering how new exploits are discovered on a daily basis.

This adds to the TCO an organization experiences because someone in the organization has to be aware of evolving security threats, not to mention deal with them. OSS community response to security issues can be a haphazard practice, of course. But what may concern an organization, even more, is that any security issue that arises will quickly disseminate publicly by those communities. That makes the risk of an attacker exploiting them far greater since they will have unfettered access to the details of the problem.

The TCO Equation

TCO has many components that need to be considered. Vendors may minimize TCO since they wrap many of the costs the organization would otherwise have to bear by itself into one predefined fee. OSS presents a low initial cost but has the additional costs of any needed modification that will appear during the software lifecycle. An organization must be cognizant of this TCO cost mix when it considers the path it should take.

Learn More about the NeoLoad Testing Tool

Discover more load testing and performance testing content on the Neotys Resources pages, or download the latest version of NeoLoad and start testing today.


Larry Loeb

Larry Loeb has written for many of the last century’s dominant “dead tree” computer magazines including BYTE Magazine (Consulting Editor) and the launch of WebWeek (Senior Editor). Additional works to his credit include a seven-year engagement with IBM DeveloperWorks and a book on the Secure Electronic Transaction (SET) Internet Protocol. His latest entry, “Hack Proofing XML,” takes its name based on what he felt was the commercially acceptable thing to do.

Larry’s been online since UUCP “bang” (where the world seemed to exist relative to DEC VAX) and has served as editor of the Macintosh Exchange on BIX and the VARBusiness Exchange. He lives in Lexington, Kentucky and can be found on LinkedIn here, or on Twitter at @larryloeb.

Leave a Reply

Your email address will not be published. Required fields are marked *