#NeotysPAC – Automated Quality Gates in Performance Testing, by Roman Ferstl

Performance testing involves many steps, but the evaluation and analysis of results are usually time consuming and repetitive. Therefore, fully automating this process is desirable to reduce the duration of your test cycles and to enable different DevOps use cases. Creating automated quality gates is one of the goals that you can achieve with Keptn, an open-source project initiated by Dynatrace.


In this blog we show the necessary steps to implement such quality gate automation with NeoLoad, Dynatrace and Keptn.


Why automation? What are the benefits?

Improve performance test quality – repeatability

In order to compare results of two performance tests, you need to analyze individual metrics over a period of time, under the same conditions, and follow the exact steps for every test cycle.

Seamless CI/CD flow

Performance test automation enables the integration of additional quality measures in existing pipelines, which adds automated feedback for developers in the development process.

Less time in test evaluation

As stated above, test evaluation takes a lot of time when done properly. This time could be spent defining new targets to increase overall testing coverage or test frequency.

Prerequisite for PTaaS – Performance Testing as a Service

When fully automated, performance tests can be triggered and evaluated by anyone in the engineering department.

Foundation for chaos engineering

To test remediation tasks, one can introduce chaos in a pre-production environment. Those remediation tasks may be triggered and afterwards verified by automated performance tests.


Building blocks

Our scenario covers: 

  1. A pipeline (Azure DevOps) that initially triggers NeoLoad Web to run a load test.
  2. During the test, our application is constantly monitored via Dynatrace’s OneAgent.
  3. NeoLoad Web informs the pipeline after the test is completed.
  4. The pipeline triggers Keptn to start the evaluation.
  5. Keptn pulls all metric data from Dynatrace via an sli-provider.
  6. Keptn automatically evaluates the SLIs with the given SLOs.
  7. Keptn reports the test score of the load test back to the pipeline.


  • NeoLoad Web
  • Dynatrace Tenant

Keptn on Kubernetes (Keptn on k3s)


Quick reminder on SLIs, SLOs & SLAs

For anybody not familiar with Service Reliability Engineering (SRE), here’s a short introduction on the topic. If you want to learn more, consider reading Site Reliability Engineering or get a good overview by watching SLIs, SLOs, SLAs, oh my! on YouTube, which are both produced and advocated by Google.

SLI – Service Level Indicator

Is basically any metric, something you can measure, like response time, error rate, CPU usage, etc. Additionally to “what” to measure you also have to define “how” you measure it. Examples are Average, Min, Max, or any Percentile (P1,…,P99).
Example: P95 of the CPU usage in % of the host, in short host_cpu_usage_perc_p95

SLO – Service Level Objective

As the name indicates, this defines the objectives of an SLI. In other words, the boundaries in which the SLI should lie. If we continue our example, we could say that the upper limit should be 75% and the lower limit 25% (a lower limit could indicate the potential to downscale).

SLA – Service Level Agreement

A SLA is a business-level agreement, based on your SLOs, that defines the service availability, including penalties.


Automate the load test

NOTE: A pipeline is not necessary. For instance, everything could be achieved with a shell script as well. 


We used the Azure DevOps Keptn Integration, written by Bert van der Heyden, for sending events to Keptn.


Even if your existing pipelines may look different, it should be easy to integrate this approach: 

  1. Install NeoLoad CLI (NL-CLI)
  2. NL-CLI: Login into NeoLoad Web
  3. NL-CLI: Upload NeoLoad project
  4. NL-CLI: Configure load test
  5. Save start time
  6. NL-CLI: Run NeoLoad test
  7. Save end time
  8. Prepare Keptn
  9. Start Keptn evaluation
  10. Wait for Keptn to finish

To have a working example by hand, you can copy everything from our gist:
GIST: ttt-pipeline.yaml

Define SLI/SLOs via Dynatrace dashboard

Previously to use metrics from Dynatrace as your SLIs, one had to parse the dashboard JSON (Edit > Settings > Advanced Settings) to find the correct metric name. 


# dashboard JSON



  "name": "Custom chart",

  "tileType": "CUSTOM_CHARTING",


  "tileFilter": {},

  "filterConfig": {


    "chartConfig": {

      "legendShown": true,

      "type": "TOP_LIST",

      "series": [


          "metric": "builtin:containers.cpu.usageMilliCores",

          "aggregation": "AVG",

          "type": "LINE",

          "entityType": "CONTAINER_GROUP_INSTANCE",

          "dimensions": [],

          "sortAscending": false,

          "sortColumn": true,

          "aggregationRate": "TOTAL"



      "resultMetadata": {}


    "filtersPerEntityType": {}





In the above example it would be 

"metric": "builtin:containers.cpu.usageMilliCores”


Afterwards you had to follow this guide to write a proper SLI.yaml. This process of gathering more than one metric is time consuming and error prone!


Therefore, during our Keptn Workshop in July with Andi Grabner, an idea for improvement was born:


Why not use a dashboard to gather all SLI information and create the SLI.yaml on the fly? And to go even further, why not add the SLO definition to the dashboard too?


Thanks to the awesome community of Keptn, the capability to gather SLI/SLO definitions from a Dynatrace dashboard to the dynatrace-sli-service was added three weeks later with version 0.5.0!


Prepare the Dynatrace dashboard

The dynatrace-sli-service looks for a name beginning with KQC – Keptn Quality Gate to find the correct dashboard. In addition, the project, service and stage from Keptn must be included in the dashboard name. For the above example, the dashboard would be used for

  • Project: ttt
  • Service: ttt-app-pac
  • Stage: dev


For SLI/SLOs we follow a similar approach. Everything after the first ; in a tile name is used for our definition. We will use our example from above:

MY TILE NAME;sli=host_cpu_usage_perc_p95;pass=<75;pass=>25

We would then repeat this procedure for all our relevant tiles/metrics, and we are good to go.

Note: You only have to add a definition on tiles you want to evaluate afterwards. You can have additional tiles without a definition on your dashboard too. The sli-provider will ignore all tiles without one.

As you can see in the screenshot above next to the “Total Pass” field, we added a global criterion to define when the test should pass based on the Total Score computed by Keptn. However, this is not mandatory, because there are predefined defaults available.
To get a better understanding of what you can achieve with those SLI/SLO definitions, take a look at the documentation page of the dynatrace-sli-service on GitHub.


Automate the evaluation

In this section we will focus on how to configure Keptn for the Quality Gate use case.

Create Project

Create shipyard.yaml (GitHub: Shipyard Spec)


- name: "dev"

  deployment_strategy: "direct"

  test_strategy: "performance"

Run: keptn create project <PROJECT-NAME> --shipyard shipyard.yaml or

with git upstream:

keptn create project <PROJECT-NAME> --shipyard shipyard.yaml \ 

--git-user=<GIT-USER> \

--git-token=<YOUR-GIT-TOKEN> \


Add Service to Keptn Project


keptn create service <SERVICE-NAME> --project=<PROJECT-NAME>

Create Kubernetes Secret with Dynatrace Credentials


  • Generate Dynatrace API Token (Settings/Integration/Dynatrace API)

Create Kubernetes Secret:

DT_TENANT={your-environment-id}.live.dynatrace.com OR {your-domain}/e/{your-environment-id}


kubectl -n keptn create secret generic dynatrace  \

--from-literal="DT_TENANT=$DT_TENANT"  \


Note: We named our secret “dynatrace,” but this could be any other name too.

Create dynatrace.conf.yaml

The dynatrace.conf.yaml defines which credentials should be used for accessing the Dynatrace API:

# dynatrace.conf.yaml

sepc_version: 0.1.0

dtCreds: dynatrace

dashboard: XXXXXXXX-XXXX-4XXX-XXXX-XXXXXXXXXXXX # Dashboard ID - uuid_v4 <optional>

Note: You can easily find the dashboard id for an existing dashboard as part of the URL.

You can add one per project or even at a stage level:

keptn add-resource --project=<PROJECT-NAME> --stage=<STAGE-NAME> --resource=dynatrace.conf.yaml --resourceUri=dynatrace/dynatrace.conf.yaml

Set up Dynatrace monitoring

keptn configure monitoring dynatrace --project=<PROJECT-NAME>

Install dynatrace-sli-service

Check releases for latest stable version: GitHub: keptn-contrib/dynatrace-sli-service releases

kubectl apply -n keptn -f https://raw.githubusercontent.com/keptn-contrib/dynatrace-sli-service/<LATEST-STABLE-VERSION>/deploy/service.yaml

Important: If you start from scratch with a new project, make sure you upload an empty SLO.yaml to your service. Otherwise the Lighthouse service will skip evaluation and never trigger the dynatrace-sli-service.

# slo.yaml

spec_version: 0.1.0



keptn add-resource --project=<PROJECT-NAME> --stage=<STAGE-NAME> --service=<SERVICE-NAME> --resource=slo.yaml --resourceUri=slo.yaml

Run evaluation

To test evaluation of Keptn, send some traffic to your application and trigger the evaluation via the Keptn CLI:

keptn send event start-evaluation --project=[PROJECT-NAME] --stage=[STAGE-NAME] --service=[SERVICE-NAME] --start=[ISO-DATE-STRING] --end=[ISO-DATE-STRING]

# Example

keptn send event start-evaluation --project=ttt --stage=dev --service=ttt-app-pac --start=2020-10-07T08:27:53 --end=2020-10-07T08:31:21


We truly believe that the automation of performance-based quality gates will play an important part in the steadily growing DevOps world. We would like to thank Andi Grabner for all the inspiring sessions we had on this topic.

Hopefully, we were able to inspire you to create your own automated quality gates. If you have any questions, or want to talk about performance in general, do not hesitate to contact us @triscon.

If you want to know more about Roman’s presentation, the recording is already available here.

Leave a Reply

Your email address will not be published. Required fields are marked *