#NeotysPAC – Should performance testing grow fur or feathers to be saved from extinction? by Alexander Podelko

At the 2020 virtual PAC I discussed “Should performance testing grow fur or feathers to be saved from extinction?” — you may check the video and slides for more details and interesting historical facts. Basically it was about the past, present and future of performance engineering. Here I will try to summarize it in a more streamlined way.

I believe that we see two major industry paradigm shifts: one, more platform/infrastructure-oriented; another, more process-oriented.

If we look at the changing computing paradigms, it started from highly centralized mainframes with full control of workloads (including instrumentation, scheduling, monitoring, chargeback — which we may trace back at least to 1966). Then it moved to distributed systems — where we didn’t have much control, and performance engineering mostly boiled down to system monitoring and load testing. And now the paradigm moved back to centralization with the introduction of web and, especially, cloud. We may see many parallels with mainframes (including more control and charging for resource usage) — but now we have open, unlimited workloads for many systems, and their sophistication skyrocketed. That is where we are — and performance engineering catching up. We may have the next paradigm on the horizon — another wave of decentralization with IoT, FOX, edge computing, peer-to-peer communication, blockchain, and other similar trends — but it doesn’t look like it is seriously impacting performance engineering yet (although some technologies such as IoT definitely are getting mature and their impact becomes noticeable).

Another major shift with iterative (agile) development and DevOps is rather process-oriented.

During the times of waterfall development, performance engineering (and some other groups — for example, functional testing) was between development and operations. There were well-established processes, including, for example, performance testing and capacity planning to move systems from development into production during product releases. Iterative (agile) development and DevOps merge everything into the single continuous process. There are a lot of discussions on what exactly DevOps (and other modern trends) is, but from the performance engineering point of view it is important that it is not between development and operations anymore — it must re-define itself as a part of a continuous DevOps process. That is happening as we speak — but far from being set in stone. The exact names and variations of that continuous process are not so important in our context, so we won’t dive into terminological discussions here. 

It is important that performance engineering gets merged into the single continuous DevOps process. Merging with development is often referred to as “shift left,” and merging with operations is often referred to as “shift right” — but, in essence, it is the same process and it is rather merge than shift per se. And the main question is whether performance engineering remains a separate discipline integrated into the DevOps process  or whether it will disappear, with some functions going to development (for example, to developers and  SDET) and some functions going to operations (for example, to SRE and FinOps). Unfortunately, we see both trends — and if the second wins, we may lose very important parts of performance engineering. So it is really whether performance engineering (and performance testing in particular) adjusts or goes extinct.

How should performance engineering adjust itself? I considered saying “re-invent itself,” but decided that it might be misleading. Yes, all processes and roles get changed — but the foundation remains the same. Almost all performance engineering books, even those published in the 1990s, are mostly relevant (of course, you need to make a lot of adjustments due to changes in technologies and processes — but the main ideas are still quite relevant). So I believe that it is rather an evolution and adjustment rather than a revolution — but it is still a very major evolution and very major adjustment. Where will it go and where will it end? It is rather a large topic. I wrote a few posts where I share my thoughts. In particular, I would like to mention  Shift Left, Shift Right – Is It Time for a Holistic Approach?Context-Driven Performance Engineering  and  Integrated Performance Management discussing my view of different aspects of this adjustment. How performance engineering should adjust — whether it should grow fur or feathers to be saved from extinction (if we use the dinosaur theme from PAC), we don’t know for sure; we are still in the middle of the process.

We clearly see some trends:  continuous performance engineering / performance testing with seamless integration with both Dev and Ops; performance as code (rather a new concept that is not well defined yet); a holistic view of performance to cover all aspects, including resilience, scalability, etc., being the center of performance expertise. We do see some interesting advances here, but it is still far from a clear picture where it will end. For example, see Automated system performance testing at MongoDB and The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System (I have joined the MongoDB performance team recently and am very excited to work with these technologies) or the Keptn project. There quite well could be other interesting developments around that not shared publicly yet. We still are rather in the beginning  — and I am looking forward to seeing how it will work out. We definitely are in interesting times for performance engineering (which, as we all know, is not always a blessing).

If you want to know more about Alexander’s presentation, the recording is already available here.


Leave a Reply

Your email address will not be published. Required fields are marked *