#NeotysPAC – Automate Test Analysis with a Custom Keptn SLI Provider, by Andreas Grabner

Keptn is an event-driven open source project that orchestrates the process of artifact or configuration change delivery, functional or performance testing, deployment & test validation, artifact & change promotion as well as auto-remediation. You can also find us on https://github.com/keptn/keptn or join our Slack channel via https://slack.keptn.sh

Keptn’s SLI/SLO-based Quality Gates

One of the most popular automation processes is around deployment and test validation through Keptn’s SLI/SLO-based Quality Gate capability. If this is the first time you’re hearing about SLIs & SLOs, don’t worry — read up on it through my SLO Validation as Self-Service with Keptn blog I wrote for a previous PAC event or watch the where I also covered the basics of SLIs & SLOs.

Essentially what Keptn Quality Gates do is automate the process of extracting indicators (=metrics) from one or many data sources (testing report, results database, load testing tool, monitoring tool …) and compare them against your release objectives (=absolute thresholds or relative against previous releases). This was also my opening explanation of “The problem keptn solves: Automates Data Analysis” at my recent Neotys Jurassic PAC presentation :

 

automate test analysis

Keptn automates the analysis of data, e.g, test results, monitoring data, and delivers a clear result on whether your SLIs (Service Level Indicators) meet your SLOs (Service Level Objectives).

Based on what our community tells us, Keptn Quality Gates helps them save about 80% of time in analyzing build results. One company did the math and said those automated evaluations give them about 42000 hours per year of efficiency gain!

To trigger such an evaluation, one can simply send an evaluation request event to Keptn via its API. This makes it easy to integrate it with your existing automation tools such as Jenkins, GitLab, Azure DevOps or others. Keptn’s event engine takes this event to the Lighthouse service, which first reaches out to the configured SLI provider to retrieve data from the source system. Once data is received, it compares the data against the SLOs and sends the result as an event which can either be seen in the Keptn’s Bridge (=Web interface) or retrieved through Keptn’s API.

Like any process, Keptn’s Quality Gate capability is also event-driven, allowing it to pull data from any data source through an SLI provider and comparing it with SLOs

While the Keptn community has already built a handful of SLI providers to pull in data from data sources such as Prometheus, Dynatrace or NeoLoad — and while some other SLI providers are already in the works (see the keptn-sandbox repos on GitHub) — I walked through the process of building your own Keptn SLI provider in a matter of minutes. This allows everyone to apply Keptn’s SLI/SLO-based evaluation approach on any type of data source!

The PAC SLI-Provider as Reference Implementation

For my demo at implemented a very simple SLI provider that was pulling data from a JSON data source. In my case I decided that the JSON is holding information about previous PAC events such as the number of speakers or average rating of sessions. The PAC SLI provider basically enables me to use Keptn to pull data from that JSON file in a fully automated way:

The PAC SLI provider I built for Neotys Jurassic PAC 2020 showcases how to have Keptn pull in data from a JSON data source for SLI/SLO-based Quality Gate evaluation

I encourage you to have a look at my implementation — modify it — or give me feedback. I also think it’s a great way to learn how to build your own SLI provider!

During my presentation I also gave more insights into the internals of Keptn, which events are really sent, what type of configuration option you have and how you can leverage configuration files in Keptn’s Git repo. Here is a screenshot of one of my slides — but please be sure to watch the recording for all the details:

I covered all internals in my Neotys Jurassic PAC talk. So – make sure to check it out.

With this you should have all information you need to build your own SLI provider 😊

Writing Your Own SLI Provider in 5 Minutes

If you have a data source that contains valuable data that you want Keptn to analyze automatically, then nothing should hold you back from writing your own Keptn SLI provider. Only thing you need to do is:

  1. Clone the template repo: https://github.com/keptn-sandbox/keptn-service-template-go/
  2. Replace the template name placeholders with your data provider name
  3. Implement the method HandleInternalGetSLIEvent in eventhandlers.go
  4. Run the test suite to validate that your code works
  5. Build the container, push it to Dockerhub and let the Keptn community know!

 

Building your own SLI provider is really easy. With the template project it will only take minutes to add your own code.

If you need any help, simply reach out to the Keptn community. Best way is via our Slack which you can sign up for via https://slack.keptn.sh

 

If you want to know more about Andreas’ presentation, the recording is already available here.

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *