Nowadays everyone is trying to board their software projects on the Agile and Continuous trains. Almost every organization is trying to embrace those practices. Sadly, some even if the practices do not apply to the nature of specific software projects. And even on the ones that do apply, many are struggling to embrace the practices, succeed in the methodologies, and reap the benefits that they bring.
A few manage to somewhat embrace the practices. Even fewer manage to implement testing practices together. So you can imagine that doing QA processes well in any Agile/continuous environment is somewhat rare. Or that organizations are still struggling to get it right.
The sources of problems are many. For starters, many are trying the methodologies without chasing the goal of multiple and constant releases to production, thinking they are Agile or continuous by releasing into PRODUCTION (this is key) once a month, or every 2, 3, or more months!
Sticking to old test practices is another source of frustration. Many organizations have not been able to let go of old tools and methodologies for QA, with all sorts of mistakes from automating all tests in the front end tier with E2E tests, trying to manually test many things, and, the topic of this post, not automating their automations well. Or not automating them at all.
In other words, enabling QA automations to automatically be triggered in response to another process, scheduled, or in a flow of steps. In other words, in a pipeline.
But what is this pipeline thing I’m talking about? Let’s see what the deal with it is.
Pipelines in real life are just a set of tubes. They keep things (usually water) flowing, and they control the flows by redirecting them, distributing them, or limiting their throughput. But the most important is that they keep things flowing from the source until the intended output.
IT continuous pipelines work the same way. They help to keep your SDLC flowing by maintaining such flow, distributing it, helping to control where it goes and where it shouldn’t, deciding if it should go or not, as well as measuring the flow. In general, it is a harmonic structure that will guarantee that you rarely think about it, if it is well implemented, tuned, and maintained when needed. Or do you constantly think of the pipelines at home bringing you water and taking the disposable liquids away?
As you noticed, a key element is to build them well, think of their design ahead of implementation, and keep updating them as needed.
In modern continuous platforms, these pipelines are often referred to as just a set of instruction files. They trigger a process, wait for it to finish, and then trigger the next one. Some can run in parallel; some validate the output to decide what to do and many other things.
But the concept of pipelining doesn’t just refer to these instruction files. It can be about any mechanism that streamlines a set of processes you may need to be executed in a chain, to optimize that your software ends in production as easily, optimally, and uninterrupted as possible.
And that is the key goal for pipelines. They are automations that move your software through all the needed processes to get safely to production. This enables the teams to push code constantly, reaching this ephemeral state of continuous integration.
Now let’s see about some of the main elements we need to be careful of, or provide to our pipelines to be successful.
As a starting point, your pipelines must ensure that things keep flowing. We must create the steps by taking into account all the possible states they can pass through to ensure they won’t stop. Taking care of the necessary structure means checking that every piece is taken into account.
We must consider the inputs and outputs of each. Of course, in the steps where there will be inputs expected or outputs needed for further steps. But on those, a key element is that we must ensure that the information can be passed through — by directly sending an output value, by putting the output in a reachable repository, and avoiding tools that do not share.
That is key because elements in your pipeline must integrate well to main the aforementioned flow. That means that the elements should be able to be automatically invoked, share inputs and outputs, not be sealed or exclusive, need the least human intervention, and overall not stop the flow.
Another element to keep the flow is to create the pipelines with every possible outcome in mind You must be able to see different futures and program into your pipelines the behavior they should have on every possible scenario and branch where the flow could take you.
And last, your pipelines should be like Rube Goldberg machines. Everything chained, triggering other actions. From the initial kickoff, it can be scheduled or triggered. To every other action afterward, everything must be linked and keep running like the beautiful machinery that it is supposed to be.
WHY PERFORMANCE PIPELINES?
Performance is an element that we must be thinking of and implementing already in our pipelines. Since most aren’t checking for it, anything goes through in terms of bad performance. Pipelines must have steps in themselves to ensure that what passes through is performing well.
Every step in our pipeline must be guaranteeing good performance in general. Again, not just load tests. Every type of performance should be assured and in every element, from unit, integration, development, debugging, production releases, etc. Even load testing should be included as a step, but more on that later.
As mentioned, you must implement performance measurements at every step. And each measurement must be paired with a validation. You must implement performance gateways on each step. With this, you will generate historical records of the performance of the element flowing through the pipeline. You will have clear SLOs. With that, you will be able to push back if the performance deviates or degrades from past metrics.
Not focusing only on load tests, you must include automations according to the automation pyramid principles. Those automations should be included in your pipeline flow. But each element and automation needs to adapt and evolve from old practices, as they cannot be easily included and maintained in our pipelines.
And another performance element is the good performance of the pipelines themselves. The flow should not be slow. We must ensure that the processes that we chain and link on perform well. The point of pipelining is to enable fast and automatic release into production. Bad-performing pieces will stop the flow. We must ensure everything flows fast and well.
HOW TO PIPELINE PERFORMANCE
Now it is time for some recommendations on how to enable performance assurance in your pipelines.
- Stop thinking of performance assurance as just load testing. Load testing is a slow process, and in itself is very inefficient. Implement all the other performance test and assurance practices before.
- Implement automatic performance measurements and tests everywhere and on every tier. Be it in requirements meetings, while developing code, in QA, in staging, and even once in production.
- Applying performance everywhere needs synthetic triggering through automations. For that, again make sure that you follow the automation pyramid principles and that you integrate them as much as possible in your pipelines. Everything covered in unit tests, lots on the services tier, and a little bit in front-end or E2E tests. Manual tests do not play well in pipelines.
- Enable performance measurements everywhere. Again, this should be done in every environment through APM, instrumentation, and telemetry. All that observability should be funneled to your pipelines to aid in the decision-making process.
- Speaking of the decision-making process, make sure you implement all the SLOs and relevant outcomes based on performance results. Make sure that code that doesn’t perform well is pushed back in your pipeline. And, of course, let the good code pass freely and happily. You should verify deviations as you will be constantly checking the performance quality of everything.
- Share performance with everyone, making sure that every performance automation or measurement is accessible, easy to retrieve, and manageable by everyone. With that, you will avoid changes in the code to break the automation and hence break the flow. Everyone should be able to fix and maintain the performance assurance elements of the pipeline.
- Train your team on the performance components. This should no longer be the responsibility of just the QA or performance specialist. Make sure you share the knowledge with the team, train them, and share the relevant toolset.
- By enabling everyone to maintain, you must ensure that whoever introduces a change can update the relevant automation. Whoever makes the change should be the most indicated person to maintain whatever is impacted by the change. Especially performance assets. Make sure that whoever breaks something fixes it right away.
- Also the performance metrics and results should be accessible by everyone and everything. Every team member should be able to verify the results. Every pipeline component should be able to access and query any metric needed as well.
- Make your performance automations small and quick. With this, you will not be stopping the pipeline or slowing it down. That is why load tests are not really recommended here. But even if you need to, you can include small, quick load tests. Do not slow the flow, please.
- Automate your data needs. You can embed them in your pipeline. To review that there is data automatically. Or in the same way, generate needed data automatically as a pre-step of your pipeline.
- All your performance automations should be invokable and not require too many or complex tools. Everything should be as transparent and efficient as possible.
And I could list more recommendations. But this post should not be War and Peace-length material. Even so, these wae some of the most important recommendations I can give you to embrace your performance testing practices in your pipelines.
Thank you very much for reaching this point. Leandro, out!
If you want to know more about Leandro’s presentation, the recording is already available here.