This article is the second in a multi-part series on how to enable performance testing to fit into” continuous performance testing” and automated pipelines seamlessly.
- Store all your assets: code, test artifacts, pipeline, config, etc. using SCM (e.g., Git)
- Develop common branching/tagging strategies that align across all your assets
- Branch before you start debugging CI testing/reconfigurations, then merge back
At Neotys, we work with a lot of diverse teams around the world. Not every company implements performance engineering practices at the same volume level or time intervals along the software development lifecycle. That’s okay. The important thing to keep in mind when making decisions about your Agile, DevOps, or otherwise overgrown software forest journey, “What gaps will prevent us from progressing beyond what we’ve already decided as imperative to do?”
Many have decided that building sustainable automated and continuous processes across the entire lifecycle, including performance engineering and testing, is a modern imperative.
Avoid “Hurrying Up Just to Slow Down” Syndrome
There are a million causes of why we waste time. “Rework” isn’t the only category in your value streams, not even close. Agile and iterative product teams often build prototypes to learn about the problem and to produce a useable product. While we could all improve planning and requirements gathering, we often find ourselves blocked from closing out work items by unanticipated dependencies and misaligned expectations.
In performance testing, numerous things must be in place to get started: non-functional requirements/criteria, testing environments, system/service specifications, clear definitions of Service Level Objectives (SLOs) aligned across product, development, and operations teams, test data, etc. These are usual suspects, but how is this different when you begin to implement performance testing in continuous delivery processes?
From all of our experiences with customers and partner teams, here are a few key lessons we’ve learned about what needs to be in place to ensure your journey doesn’t start with critical blockers:
- Test storage, branching, and versioning should match the development and operations patterns and cycles (e.g., feature work, sprint hardening phases, etc.)
- A clear glide-path for promoting changes in test artifacts during sprints to release-ready branches used in continuous integration and orchestration platforms
- Debugging issues during testing and differentiating actual system defects from testing and deployment process problems
Let’s start with the first blocker – your test storage strategy.
Developers Demand a Performance “Dumb Button,” and that’s Fine
Your performance tests need to be somewhere that your automated CI job can access clearly and quickly. Many organizations have already switched to some form of Git-compliance Source Code Management (SCM) system: GitLab, AWS CodeCommit, Azure Repos, Bitbucket, etc.
Consider that storing your performance tests in a monolithic silo such as Micro Focus Performance Center isolates performance engineering activity from the rest of your modern automated lifecycle (ALM) in CI tools (e.g., Jenkins, TeamCity, Bamboo, CircleCI, and XebiaLabs XLRelease). This approach makes performance testing activity something that only a select few are capable of doing.
Some Simplified Processes Encourage Autonomy, Some Don’t
We recently met with the DevOps/cloud engineering, scalability, and performance team at an F20+ organization moving off of all Micro Focus products. While they elaborated on their current experiences to provide greater autonomy to their app and service development teams by creating deployment playbooks to spin up various tech stacks for Go, Node.js, and Java-based systems, they had a fantastic point – making something easy for someone also makes it hard for them to learn how to fix things if they go wrong.
Simplifying Git Semantics for Non-CLI Savvy Folks
That’s why NeoLoad provides direct integration with Git sources, so everyone can collaborate on the details of performance tests, version their changes in a manner that matches feature/sprint branching in development source code, execute them from anywhere, all in a matter of minutes not hours.
For contributors unfamiliar with Git command-line interface (CLI), Git capabilities in the NeoLoad desktop workbench simplify things, making it easy to clone, modify, push to a particular branch, and eventually merge with (and promote) those changes to other mainstream CI/CD events.
Git in NeoLoad encourages both autonomy and collaboration between engineers at all levels.
NeoLoad also supports all-code-based YAML descriptions of load tests. Regardless of the method for test construction and maintenance, via a graphical interface on a workstation or with the IDE or code editor of your choice, NeoLoad accelerates API load testing and SLO alignment across teams by visualizing and agreeing on performance expectations early on in cycles.
If you want to try this out yourself, you can get started here: https://www.neotys.com/as-code.
Now, let’s turn to the topic of what to do with these versions and branches – “test promotion” in CI.
Avoid Hitting Every Branch on the Way to Production
We want our code in production because that’s where it makes money. Performance testing is vital to this process, namely because you don’t want to ship crappy code that fails in front of users. But, this kind of testing comes with versionable artifacts that must match the app or service code you’re testing.
These assets should be in lock-step with each other. When multiple teams and features are flying out the door simultaneously, how do you ensure that versions are the same across code and test assets? Do you tend to “test all the things” as some vendors use as marketing speak?
One ubiquitously implemented answer to this question is to use short branches in Git to simplify verification and validation of changes to testing and automated processes definitions.
Branching Accelerates Test Promotion
For performance tests in CI, pipelines are often constructed such that humans or other processes (e.g., other master pipelines, JIRA, webhooks, and chatbots) can trigger them with automation. You build, get them working for test suites that match the current version of your app or service, then what? The code/environment configuration changes and tests similarly have to change as well. You’ve already got working pipelines and tests on the working version, how do you verify that the proposed version of the software and the new tests work too?
Case Study: F20-something SREs Building Scalable CI Conventions
In the conversation previously cited, I asked the DevOps engineers what represents some of their most painful work. Almost unanimously, they told me that it’s pattern and convention building across multiple cloud platforms. While they hadn’t deployed containers internally in a scalable way yet, between moving to Microsoft Azure, existing Pivotal Cloud Foundry, huge amounts of VMware automation, and Google Cloud Platform, the idea of creating playbooks for development teams and helper scripts to deploy new services was much of their team’s daily challenge.
What was different in this case was that they’re using TeamCity and eventually CircleCI to automate these playbooks, in essence allowing the goal of fully automated to drive the motion to improve their entire organization’s practices. They had Neotys around because, in their opinion, the NeoLoad platform was the only load testing and performance solution that could fit all enterprise aspects of their modern, legacy, and hybrid apps using this automated model. Git support, access to data via Open RESTful APIs, and flexibility to run tests whenever, wherever they want, suits their needs nicely.
As they scale up these patterns across various cloud and container providers, automated pipelines that use simple configuration toggle to control unit, integration, functional, regression, and performance testing execution allow multiple development teams to comply at a high level with risk requirements. Of course, a boolean toggle needs tests and data to complete, so they created convention-based repository and directory structures which the SRE team can expect in each pipeline project to contain these project-specific artifacts. This leaves it to development and performance engineers to use NeoLoad Git and YAML-based test projects to elaborate on test semantics and goals together.
Ignore the Urge to Fix Test Issues in the “Master” Branch
When performance tests fail, understanding why they failed is critical not only for traditional deep-dive analysis processes but to inform automated go/no-go indicators in pipelines as well.
The difficulty comes in when teams don’t feel like they can trust these pass/fail warning flags.
Being able to determine which part of your automated process needs fixing is critical to keeping them up and running. In our experience, having separate versions of pipelines to diagnose issues helps engineers fault isolate, correct, verify, then integrate their changes back into the production version of the automated process.
The number one reason we’ve seen continuous performance testing fail is when people who haven’t learned the value of testing their changes – update these pipelines inadvertently making them seem flaky or oft broken.
Branch Before Changes; Rapidly Drive to Mergable Fixes
Before you make changes to your tests, automation/pipeline code, or infrastructure descriptions (Dockerfiles, Kubernetes deployment files, Chef scripts, etc.), create a new branch. They’re cheap, disposable, and hopefully, short-lived.
However, this creates a problem for classic monolithic automated pipelines: do you have to create “candidate” copies of all your CI jobs and configuration? The answer is, “usually not.” Take, for instance, Jenkins Multibranch Pipelines, which allow you to point to a single repo but maintain separate pipelines per branch automatically.
When also configured with support for Pull/Change Request, you can create workflows that allow test reconfiguration and validation to occur separately from the production version of the pipeline until a final fix is known to work and ready for the rest of the teams to run.
Differentiating between “Hard” and “Soft” Failures
Performance tests contain lots of data, and not every failure means the same thing. Here are some of the top reasons why they fail:
- Your app/service can’t handle the pressure of the load test
- The load infrastructure provisioned can’t perform the test
- One or more service level objectives (sometimes called SLAs in tooling) are violated
- Networking or connectivity between load infrastructure and target system(s) is faulty
- Validations to prove the scrips complete transactions fail (as they should)
In each of these cases, a high-level go/no-go indicator used by your CI job is essential, but also insufficient to diagnose what happened and if something needs to be fixed. For instance, process indicators such as NeoLoad’s command-line interface (CLI) exit codes provide early failure level detail to inform pipeline try/catch and orchestration event semantics. In this first example, the test was successfully executed, but a few critical SLA violations occurred:
In this second example, our load infrastructure (Docker-based Jenkins build nodes) had some issues attaching to Jenkins due to new security constraints, thus invalidating the test entirely:
Going further, NeoLoad API summary data can be used to determine success or failure context:
Additionally, specific SLA data can be retrieved in a similar fashion from different API endpoints:
With the right level of detail on hand, even in a simple bash script with curl, teams can quickly identify the most significant contributing factor to why pipelines might not be working. Small, understandable changes to these automation artifacts will save you a ton of time in the end.
Next: Modern Fast Feedback Loops, Analysis, and Trending
In the final article of this series, we’ll look at what putting continuous performance practices in place gets you. In short, better systems, faster fail/fix cycles, and less technical debt.
We have seen all sorts of teams and challenges, but we’ve also worked through them with engineers to benefit from having guard rails in place around system performance.
Find an engineer and an hour or so, download NeoLoad, and contact us to figure out how to get your performance ball rolling in a better direction. You won’t be sorry you did.
Learn More about Continuous Performance Testing