#NeotysPAC – Performance Testing in a DevOps World: Don’t Spin in the Middle, by Stijn Schepers

[By Stijn Schepers, Accenture]

I was honored when Neotys asked me to present at the first Performance Advisory Council in Scotland.  I do not often have the opportunity to meet world-class performance engineers to exchange ideas and to talk about the future of our profession. So a big thank you to Neotys who organized a unique event at a superb location. Borthwick Castle, a medieval castle built in 1430! A place where whiskey tastes even better than anywhere else in the world! This must be the perfect setting to talk about the future of performance engineering.

My presentation was about Performance Testing in a DevOps world: Don’t spin in the middle. This was a personal story about how DevOps has transformed my professional life in a positive way. DevOps has brought a lot of change to the way we work; when you embrace this change, and you use this changing landscape as an opportunity to learn and grow, you can become a trendsetter of your future profession!

Waterfall Approach

The majority of my career as a performance engineer, I was involved in huge programs of work which were delivered based on a waterfall approach. These projects took a very long time to deliver (months to years), were costly and always went live after the official go-live date (delayed). Performance load testing would happen after functional testing, and quite often this meant that you would start with performance test executions just days before go-live. So there was no time to do decent test execution and test analysis and no time what-so-ever to solve any bottlenecks. If testing does not bring added value, it should not be done at all.

DevOps and Performance Testing

With Digital Transformation the change from Waterfall to DevOps was needed so business could release more efficiently and quickly to meet customer demand. Instead of having one major release as a big BANG, in a DevOps delivery model, smaller features are being released in iterations. Iterations are typically 4 to 6 weeks but can be daily or even hourly! This move to DevOps meant that we had to change the approach of performance testing. We could not spend weeks and weeks of scripting and clean data, and modifying frameworks. The focus of Performance Testing changed to adding value as quickly as possible. Performance Testing moved from E2E Performance Testing to Low-Level Testing (Shift Left) and Operational Monitoring with Application Performance Management (APM) tools (Shift Right). Performance Testers should not only spin in the middle but should move more to the left and the right.

Shift Right

When defining a Performance Test Approach in a DevOps delivery model, I see Shift Right as low hanging fruit. Implementing a decent Application Performance Management (APM) tool is a no-brainer. An online business that cares about performance and therefore cares about revenue and branding and who puts their client first should look at implementing a modern APM solution (Dynatrace, AppDynamics, New Relic). APM solutions measure the performance of an application by measuring the performance of a “business transactions” throughout the system components. You can easily pinpoint which IT component is causing delays. You can drill down to code level or SQL query level. These insights are required to lower the risk when you often release to production. You get immediate feedback after a release. You know if the feature is used (bonus!) and if the release caused a degradation in performance. A test engineer should have access to APM frameworks in production, so he understands what happens in production (workload, slow transactions, error rate) and which functionality performs badly. Based on this knowledge he can conduct specific tests to improve the software. APM provides a test engineer with the means to turn testing from a problem raising activity into a solution providing activity. And think about it… is it not much nicer to provide solutions rather than nagging about problems?

Shift Left

Time and knowledge are limited, so we need to spend it wisely! Test features that pose the highest risk. A risk assessment of the features to be released during the next iteration is a great start. Typically you can use a scorecard system for the assessment. Scores can be provided based on business importance, technical complexity, load, front-end facing, new or changed code, etc. The features with the highest score require the most attention.

Is it not funny that Test Engineers are sometimes blamed for issues they detect. Excuse me! Who wrote the code?! So is it not time that developers take more proud of the code that they write and start doing code profiling? For JAVA there is some great profiler like VisualVM and JProfiles. Profilers can help the developer to detect performance bottlenecks in their code. With Rapid Delivery it is important to profile the most crucial code.

E2E Web-based load test scripts are very sensitive to change and take time to write. This is wasted time as creating load test scripts don’t add value. Executing load tests and analyzing the results add value. Tools like Neoload focus on reducing the time to scripts which are very useful for DevOps projects. Instead of writing (only) E2E scripts, it may be more efficient to create an automated test framework based on API calls (rest/soap). Rest and SOAP calls are easy to create and typically don’t change so often. Building a regression framework to benchmark a release or a build (CI) can accelerate testing and increase efficiencies.


The image below provides a great view of what performance testing in a DevOps world means. A test engineer should not limit testing to end-to-end testing but should look into Shifting Right and Left.

As a performance engineer, you need to “dance the dance” and not “fight the fight.” When DevOps is done well, and a culture of cooperation and trust is created, a test engineer will start to love his passion again as I did. It will be easier to dance the dance and less frequently you will hit your head against the wall. I love DevOps and look forward to what the future of digital transformation will bring.

For more detailed information about testing in a DevOps world, the following three blog posts can be of help:

Learn More

To know more about Stijn Schepers, see his biography!

Do you want to know more about this event? See Stijn Schepers’s presentation here.

Leave a Reply

Your email address will not be published. Required fields are marked *