#NeotysPAC – Automatic Result Analysis: The missing link in continuous load testing, by Stijn Schepers

DevOps remains a hot item in our industry. You may call it a buzz word but it sure is still hot! Businesses are looking at ways to add features to production in an ever-faster pace. Adding more business features faster online will create more revenue. In a continuous load testing perspective, a well working automated CI/CD pipeline is crucial in this story. And performance and load testing cannot be left out. So how can we automate load testing and embed this discipline of testing into a CI/CD pipeline? Automating load testing is not just about the automated execution of a load test. It also includes analyzing load test results in an automated way and trust your automated framework to take a PASS/FAIL decision.

Goals and requirements of an Automated Framework

So how do we start designing an Automated Framework for load testing? Before starting to create your automated framework, take time to define the goals. What do you want your “robot” to do? And what are the requirements? At the Accenture Innovation Center for Smart Services, we designed a Robotic Analytical Framework (RAF) to automate load testing. The goals we defined were focused on accelerating and improving the quality of testing.

• Fully automate performance testing into CI/CD pipeline (execution, analysis and decision taking
• Fully automate performance testing into CI/CD pipeline (execution, analysis and decision taking
• Improve the quality and consistency of testing
• Increase the velocity of performance testing
• Decrease the cost of performance testing
• Liberate the Performance Engineer from repetitive tasks

Clear requirements that support the design are crucial for the success of your framework. Too many times I have seen frameworks “exploding”. Lots of developers are driven by technicality and may create a complex beast that does way more than it should do and is so complex no-one will use it and all the investments are wasted. When creating a framework, the following requirements maybe useful for you as well.
• Keep the framework as simple as possible:
– Don’t make the framework complex because you can!
– Don’t over-engineer, stay focused on business goal
– Prime measurements must be based on response time
• The analysis must be based on raw data
• Don’t reinvent the wheel:  Use existing software to extend the framework with
• The framework must be configurable, customizable and scalable

The importance of raw data

It is funny that Henrik Rexed, the face of this event and a great ambassador for performance engineering, refers to me as Mister Raw Data. I actually don’t like this title… I’d rather have the title “Digital Super Hero or DevOps Insights Hero”. Why? The only correct way to get clear insights into the system performance is by analyzing every single response measurement (=raw data). These insights will provide you with the correct information of the application performance and will help you to improve the digital experience of the application. By doing this, a performance tester can become a Digital Super Hero. If you would like to get more insights about the importance of Raw Data, you can read my blogpost: “Performance Testing is not an average game”.

Most of the commercial solutions base the decision taking on trends, aggregation or averages. This is simply not good enough. These types of measurements, even if you compare two identical runs (baseline/benchmark), don’t provide you with enough detail to base your decision on. The risk exists that your framework will PASS or FAIL a build automatically and incorrectly.

Steps to create a Test Automation Framework for load testing

Step 1: run a load test in Non-GUI mode

So where do I start? If you can’t do it manually, don’t attempt to do it in an automated way. Your first step in automating a load test is analyzing how you can run a Non-GUI load test (sometimes referred to as Command Line Mode-CLI) with the tool you are using. This is a must if you want to automate.

• Demo_RegressionTest.nlp the name of the NeoLoad project
• Demo_Regression is the name of the load scenario you want to execute

Step 2: Export Raw Data for Smart Analytics

When you automate a load test by scheduling an automated test as part of your build process, it is important that you can export the raw data for smart analytics.

With NeoLoad Non-GUI you have an option to specify the path where you want the raw data to be exported to: -exportRaw.

Step 3: Use a central repository for all your results

It is important to transfer your raw data into a central repository. This provides a Test Engineer with the historical data of all previous runs. You may want to compare the overnight run of last night with a run you did a month ago. Having a central repository provides you with this ability.

The automated framework we created at Accenture (Robotic Analytical Framework – RAF) detects when the raw data is exported into a file share and will automatically push this data into a MySQL database. We use Tableau Software to create smart dashboards so we can easily compare runs and make customizable dashboards. The figure below displays a comparison of the last 12 runs we executed.

Step 4: Smart Analytics

Designing a smart analysis engine is the trickiest part of building an automated framework. This is also the most important part of your framework. You want to trust your framework to pass or fail a build automatically. As explained in the previous chapter, it is recommended to use the raw data for analysis. When we developed RAF, we integrated smart analytics and machine learning into the engine to calculate a Test Execution Score (TES) for every run. The higher the score, the closer the raw data (read performance/response times) of the benchmark matches with your baseline. This TES can be used to PASS or FAIL a build.

Step 5: Smart Dashboards

Visibility and insights are key aspects of DevOps. This is also the case for fully automated testing frameworks. When you’ve managed to automatically gather the results (raw data) into a central data repository, a Business Intelligence tool can be used to create smart dashboards. Personally, I am a huge fan of Tableau Software. Tableau is one of the market leaders of BI Tooling. You can customize your data in such a way that it tells the right story to your stakeholders.

RAF offers a main dashboard in which we focus on the Test Execution Score (TES), trend line, raw data, errors, throughput, and percentile graph. We also provide a Business Transaction Health graph which details the number of business transaction groups that have a performance that did not meet the thresholds.

Step 6: Tune your framework in the context of the application under test

It’s important that your framework is configurable. Automatic Analysis is a learning process. Your framework should be configured correctly based on this learning process. It is important to tune your thresholds, so you PASS or FAIL a build correctly for the application under test. You do want to avoid – at all costs – “false positive” decision taking. You do not want to FAIL a build when it should be PASSED. You can lose a lot of credibility very fast if you do get it wrong.

Conclusion about automatic result analysis in continuous load testing

Clever automation is a must in a digital world where features are being deployed into production at record speeds. This is also true for Performance Testing. When you develop your own automation framework, the following rule is an ingredient for success.
Don’t aim to develop a perfect automation framework. Understand the imperfections so you can make smart conclusions and add value quickly, progress fast and be cost-effectives in an iterative way.


Learn More about the Performance Advisory Council

Want to learn more about this event, see Stijn’s presentation here.



Leave a Reply

Your email address will not be published. Required fields are marked *