#NeotysPAC – Continuous (Early) vs System Level Performance Tests, by Ramya Ramalinga Moorthy

[By Ramya Ramalinga Moorthy]


DevOps Performance Testing Best Practices – Early vs System Level tests

With the popularity of DevOps culture, increasingly continuous performance testing is adopted by most of the organizations. This brought in tremendous improvement in proactive performance measurement & analysis possibilities, but performance testing team always had/have challenges to adapt to the new needs of the DevOps environment. In the interest of bringing up early performance testing methodologies, sometimes performance testers fail to focus on system level realistic performance tests which are still important to certify the system for target scalability & capacity using realistic workload patterns.

My Neotys PAC session covers the best practices for early (continuous) versus system level performance tests.


Is Agile & DevOps Performance Testing are different?

As the shift from traditional performance testing (waterfall) to agile performance testing (iterative) approach started spreading the seed for early performance analysis, Continuous Integrated (CI) automated Performance Testing became the obvious choice during recent years.

In Agile performance testing, the focus is on early performance measurements & feedbacks during each sprint (or target sprints) followed by system level performance tests during hardening sprint. In DevOps performance testing culture, focus is on early & continuous performance analysis during each sprint (vigorously on nightly builds) through the power of automation. This can be made possible by close collaboration of Dev, Testing & Ops teams. The level of automation adopted by the performance testing team will facilitate providing quick feedbacks thereby reducing the time taken to nail down & fix the problem quickly.

As this culture brought Performance as everyone’s responsibility, it significantly removed all tool platform silo’s & open source tools became the popular option, be it for maintaining code & version control, building, deployment, testing, release, production monitoring, etc. Hence, most of the popular commercial performance testing tools & application performance management tools provide plug-ins for integration with popular CI servers (like Jenkins, Bamboo , etc), to manage performance test execution & server performance monitoring activities from CI server. The checked in code (upon code commit), gets automatically build & deployed in various test environments to carry out functional & non-functional testing test & upon test pass criteria being met, gets deployed in production enabling continuous delivery & continuous deployment to production environment.


Level of Automation in Continuous Integrations vs Continuous Delivery vs Continuous Deployment practices

Continuous Integration practice streamlines code integration using shared repository and has automated build & testing process to validate & find problems quickly to provide rapid feedback to developers. Automated early (API) load testing is conducted as part of CI pipeline to measure the build-wise performance degradations & reported quickly. Test pass/fail criteria based on KPIs are used to automatically promote the build to next stage & performance dashboards are made available online for all the teams.

Continuous Delivery practice encompasses develop, build, test followed by making the software release ready, though production deployment is handled manually. Managing different type of tests on various environments including benchmark tests, client-side browser web performance tests, load tests followed by various other system level performance tests on production similar environment as part of CI/CD pipeline is practiced. Some long duration tests are carried out independently apart from pipeline tests. The performance test infrastructure setup for tool / load generators installations are managed on the fly with the help of containerization solutions. Use of APM tool statistics as part of CI/CD pipeline to promote the build is practiced to facilitate quick feedback cycle to developers.

Continuous Deployment pushes the code that has been successfully accepted in the CI/CD cycle into the production environment without human intervention. This practice requires highest level of automation for handling Performance Test activities, for providing deep dive performance diagnostics to identify the problems quickly using APM tools & for continuous monitoring of production environment to provide feedback cycles to make improvements in the performance test suite for upcoming releases.

Hence, the Performance Testing Framework setup for DevOps environment should be based on the organization / project team maturity & the CI / CD / CD practices adopted by the project team.


Shift-Left Performance Testing

From early sprints, depending upon the product’s development roadmap, identify foundational unit testing pieces (APIs or DB queries or micro-services, etc) to start exploring on strategies to measure its performance. APIs & Microservices are the most common foundational units considered in early CI performance tests. And the choice of the unit should be in such a way the performance test script development & maintenance time should be very less. Remember ONE-SIZE-DOESN’T-FITS-ALL. Upon finalizing the methodology to measure the performance of the foundational units of the system, make performance measurements continuously as part of CI pipeline to qualify the build. This helps to quickly identify the performance degradations immediately & give feedback that enables developers to relook at the code changes done in the last build.

Doing automated continuous performance testing without engineering activities (root cause analysis), will not help in quick performance optimization activities. Using an APM tool independently or as part of CI pipeline to provide detailed analysis will actually make it a Performance Engineering exercise instead of Performance Testing exercise.  It has become a common practice to use APM tools (like Dynatrace / AppDynamics / New Relic) on both performance test environment & Production environment to enable continuous quick feedback cycles. Ideally, both Performance Testing & Engineering (Deep dive diagnostics) pieces have to get into automated CI pipeline to achieve the full benefit of quick root-cause analysis & verification.


Early Performance Testing KPIs

To make quick comparison on the build performance, identify Key Performance Indicators (KPIs) like Response time, Server Throughput, Server Resource Utilization (CPU & Memory utilization) & Error rate. Don’t over measure & toggle additional metrics only on demand basis. The CI build pass/fail criteria should be automated based on the SLAs set for these KPIs. If the test pass criteria are met, the build is promoted to next stage, else fail the nightly build & the automated performance dashboards are available for the entire team to understand the reason for build failure. Providing detailed root-cause analysis feedbacks on the code performance with the support of APM tool can help developers to nail down the issues quickly to address them.


Shift-Right Performance Feedback

By using Application Performance Management (APM) tools & Real User Monitoring (RUM) tools on production environment for continuous monitoring, feedback loops can be established to optimize the workload used for carrying out system level performance tests. The measurements from production monitoring tools, real time user experience monitoring tools & server performance metrics collected from real user load can be used for optimizing the system level performance test strategy for upcoming sprint releases.


DevOps – Early vs System Level Performance Testing Strategy

Early Performance Tests

System Level Performance Tests

The key objective is to measure the performance of identified units & provide early feedback & automate the performance measurements on nightly builds.


The key objective is to measure the system performance for realistic workload on production similar environment to make judgments on scalability levels & server capacity requirements
Measure performance for 1 user & small concurrent user loads (on nightly builds or targeted builds) Measure System Performance , Scalability, Availability & Capacity characteristics for realistic usage patterns
Unit Test early whenever available (DB queries / API / Microservices) and automate as much as possible.


Ongoing system level performance tests can be conducted on targeted sprint releases (may or may not be part of CI pipeline) along with newly added features. (Regression test suite evolves continuously for carrying out system level tests).

Final system level tests should be carried out with ideally 6-8 critical use cases for realistic workload characteristics considering peak traffic hours.


Tests can be executed on 1 or more small scale environments (Dev / QA / Staging Environment) Tests can be executed on production similar environment (Performance Test Environment or Pre-Production Environment)
Do not bother about realistic think time / production data volumes, etc Create realistic usage patterns on production similar environment configuration / settings. Test data development evolves continuously & realistic test data should be used atleast during last level system level performance tests before production deployment.


Carryout short & crisp performance tests as part of CI pipeline Conduct System level tests on the integrated builds during every release or targeted sprint release (some tests can be part of CI pipeline as downstream job & some tests can be run independently)
Early unit tests should be run with Server Monitoring tools in place. Having APM tools would be preferred to provide detailed view of performance root-cause analysis quickly. System tests should be run with Performance Diagnostic tools (preferably APM tools) on the production similar performance test environment
Performance analysis should primarily focus on code performance optimization activities (including identification of costly methods, CPU hotspots, Memory leaks, etc).

AI/ML techniques can help in build-wise performance trend analysis & anomaly detection.


Infrastructure performance analysis (by mapping of test results from TEST to PROD env or by running few load tests on PROD env).

Performance modeling analysis (QN) to predict system performance for untested load levels.

Performance analysis should focus on system scalability assessment & capacity planning.

AI / ML techniques can help in quick performance anomaly detection

The objective & test measurements during early performance tests & system level performance tests are completely different. Because early performance test analysis culture is getting prevalent, sometimes Performance Engineers underestimate or perform incomplete system level performance tests. CI Performance tests can never be considered as a replacement for system level performance tests.

A successful Performance Engineer should understand the system internals & thereby the ‘Big Picture’ of the system under test & maturity of the project environment (Agile/CI/CD/CD) to finalize on the proactive performance assessment strategy & learn the art of connecting the dots without compromising any of these for assuring system performance: Early Performance Measurement à System Performance Measurement à Scalability Assessment à Application Capacity Assessment.


Learn More about Continuous (Early) vs System Level Performance Tests

Do you want to know more about this event? See Ramya Ramalinga Moorthy’s presentation here.


Leave a Reply

Your email address will not be published. Required fields are marked *