“Right-fitting” Performance Feedback Loops into Automated Pipelines

“Right-fitting” Performance Feedback Loops into Automated Pipelines is the first blog in a series of 3 that discusses a modernized approach to performance testing in DevOps and automated delivery models.

The main points of all 3 posts are summarized in the whitepaper A Guide to Getting Started with Continuous Performance Testing.

What do we want? Performance Feedback! When do we want it? It depends.

Feedback loops are critical the faster you deliver software. Automated approaches to feedback loops really require right-fitting the who/what/when/where aspects to implement effectively. Typically, automated feedback loops live in pipelines and CI. Anything that can’t fit into pipelines looks inefficient to the business (i.e. long cycles, toil, “waste” thinking in values streams). But what about performance testing? Isn’t it too complicated to fit into DevOps and automated contexts?

The short answer is, of course not, but it takes a little engineering thought and setting the right goals first. Why do performance engineers need pipeline-driven performance feedback sooner? A few solid reasons:

  1. It’s often cheaper to address obvious faults early in product and feature lifecycles.
  2. Long cycles introduce delays and context switching, which increases the likelihood of faults.
  3. Automating feedback requires us to drive unnecessary variance out of our processes.
  4. Having feedback helps us make better-informed decisions moment by moment.
  5. Frequently generating performance data allows us to understand trends over time, not just “big bang” point samples.

3 big challenges in automating load and performance testing

Traditional load testing and performance testing focused on large, complex end-to-end testing of fully baked pre-release systems. Most automated build and packaging cycles take minutes, not hours, so traditional performance test cycles don’t fit well into delivery cycles from a time perspective. Is end-to-end testing still necessary? Yes. Is it necessary for this process to inherit plenty of flaws, faults, anti-patterns, and bad assumptions that we could have addressed earlier? Absolutely not.

“An intelligent approach requires “right-fitting” load testing into automated pipelines so that we get early signals that are actionable (i.e., feedback loops).”

An intelligent approach requires “right-fitting” load testing into automated pipelines so that we get early signals that are actionable (i.e., feedback loops). From an engineer’s perspective, in order to not produce a mess of end-to-end late-cycle validation, we need to break down the performance verification process into meaningful smaller feedback loops that fit into shorter work cycles. This way, developers and teams can receive performance feedback earlier and more often about their changes so that by the time we test end-to-end, it’s not a garbage dumpster fire.

performance feedbackSource: Automating Go/No-Go Performance Decisions in Continuous Pipelines webinar


There are 3 main challenges when migrating from traditional performance testing to the modern automated approach:

  1. Not all systems and components are created equal, so no one single approach applies across the board. Fitness and candidacy for early vs. late testing requires discussion; pick the right targets and apply a progressive testing strategy
  2. The more frequently you do something, the “tricky bits” sum to a volume of toil (unless automated). Specifically for load testing, manual pre-configuration of infrastructure, data, and environments traditionally often takes a lot of time.
  3. If we only do something infrequently, it takes more effort than if we did it regularly. Flaky environments, brittle test scripts, out-of-sync data . . . because we don’t exercise them continuously, our ability to address these kinds of problems doesn’t improve.

A proper performance testing tool belt includes a number of practices that inherit these challenges: load testing (i.e., simulation), monitoring, mocking/virtualization, test data management / sanitization, experience sampling, and others. Many folks I work with on and off have either partially or fully automated many of these components with the help of tools like Delphix, Mockiato, NeoLoad, Graphana, and Selenium. The good news is that there is nothing insurmountable to any of these approaches when automating them; having a solid vision and an approach aligns and organizes our efforts.

Automated load testing is an approach to these challenges

It’s easy to say that the goal of automation testing is to “go faster” or “accelerate delivery.” That applies when it works reliably and produces actionable outcomes, but to get to “reliable” and “actionable,” it takes more than load testing tools. As I often say, “the problem/solution is an N-body space,” like the DevOps “people, process, technology” Venn diagram. If you’re only solving for one of these aspects of the problem, you’re not really solving the whole problem. So how do we solve for the “people” and “process” elements involved in continuous load and performance testing?

Enter automation. Automating our processes puts us up to a challenge to articulate our goals, requirements, and activities in a way that machines can execute on our behalf. It shines light on gaps in our processes, technologies, and skills . . . but that’s a good thing. We need to know our gaps in order to properly address them.

To automate performance tests that produce meaningful, actionable outcomes, people need to communicate about goals and outcomes. Discussing SLAs, SLOs, and SLIs up front is an important part of “left shifting” performance practices. Neotys customers often have “intake forms” that are pre-filled for product teams to encourage these discussions. These digital forms produce baseline automation artifacts like SLA definitions and API test details.

To automate performance tests that produce meaningful, actionable outcomes, people need to communicate about goals and outcomes.

As you combine these artifacts into a pipeline (i.e., our “process”), you will also have to automate the load infrastructure provisioning/deprovisioning process so that you have resources to run your load tests that are properly isolated from the Systems Under Test (SUTs). These days, containers and Kubernetes have come a long way to providing a standards-based approach to automating infrastructure, but there are always a million other ways to manage these resources too. Whatever provisioning strategy you employ, it shouldn’t be complicated for teams to run a test. Autonomy to obtain feedback when necessary is a key component to accelerating delivery.

Get the full story in the whitepaper A Guide to Getting Started with Continuous Performance Testing.

If your performance practices aren’t automated, you’ll scale-fail

Finally, automation shouldn’t just be in the service of a single team. There’s no force-multiplier to that kind of work in larger organizations when it’s just a local efficiency (for a Performance CoE team). Conversely, automation that many teams can use, even if they aren’t the experts that set it up, is what really helps organizations go faster. 

In DevOps, the work of performance and reliability engineering isn’t just running tests or analyzing/consolidating results (re: “tester vs. engineer”); it’s providing product teams a way to do these things themselves, while also providing safety guardrails and “good practices” around these process so that teams can grow their own performance competencies over time.

In my Customer Engineering work at Neotys, I see many mature large-scale performance practices that have transitioned from the “performance [team] as a [silo] service” mindset to a “performance processes as self-service” offering. In part because there’s just no way to scale “people” performance expertise to hundreds of teams by adding more full-time bodies, but also mainly because we now have the means in automation to be able to formalize (i.e., automate) our performance testing practices into work where machines can help with the heavy lifting.

In many ways, automating your load testing practices appropriately “brings the pain forward” and puts the forward tension on organizational aspirations to accelerate delivery. Rather than the lazy approach to leave behind “traditional practices,” we need to re-think them, inherit their useful points of wisdom, and adjust to the constraints and challenges of today.

Where do we go from here?

I’ve written a series of three articles that dive deeper into how to enable performance testing to fit into continuous and automated pipelines seamlessly.

At a high level, the emerging theme is:

  • Establish “right-fit” automated performance feedback loops into pipelines
  • Run performance tests early and often at smaller volumes to establish longitudinal data
  • Start small (targeting APIs) build automation competence and confidence
  • Make results easy to understand and actionable for product teams and developers

Again, all these principles are discussed in detail in the whitepaper A Guide to Getting Started with Continuous Performance Testing.


If you’re really interested or have feedback, feel free to reach out to us regarding how things are working for you today.

Leave a Reply

Your email address will not be published. Required fields are marked *