#NeotysPAC – The Forgotten Art of Performance Modeling – By Alexander Podelko

 

As promised during my Performance Advisory Council presentation (which was devoted to the future of performance), here is a list of resources. What I am providing is just a portion of all the content. Check “The Forgotten Art of Performance Modeling” (video) and (slides) for details.

We see that the scale and sophistication of systems are exploding. We recognize that full-scale tests become less frequent, while continuous and component-level performance testing spread widely. The main point is that with all these trends, “white box” performance modeling (to be able to analyze changes in the underlying architecture) should be an integral part of every serious performance engineering efforts.

Performance modeling was a robust discipline, but it appears that it is now somewhat in decline. Not because it was based instead on advanced math (mostly queueing theory) and not because it is no longer needed (we have even more significant performance challenges today). It appears as if it just didn’t adjust to the changing technology paradigm.

The situation is better with production-level, “black box” modeling mostly based on data extrapolation: time series forecasting, trending, machine learning, etc. Actually another PAC session Predictive Performance Modeling using Machine Learning (ML) Algorithms covered that topic.

It doesn’t appear that the problem is that it is too difficult. There were multiple (and sound) attempts to explain performance modeling more straightforwardly, for example, Queueing Theory, In Practice. Performance Modelling for the Working Engineer by Eben Freeman or Applied Performance Theory by Kavya Joshi.

There are excellent courses in some universities – see, for example, Mark Friedman’s performance engineering class slides and videos (at the University of Washington).

There are good introductory books:

There are opinions that they are going too far in simplification, but they are an excellent place to start. The authors, by the way, worked for Hyperformix.

Books by Dr. Connie U. Smith focused more on software development, introduced Software Performance Engineering (SPE):

And, of course, we have a lot of great books about performance where the application of queuing theory is discussed in detail (including explanations of queueing theory itself). Here is a list of these in chronological order:

There were (and still are) quite a few modeling tools which we may trackback to 1977 when BGS Systems released BEST/1 (seems there are few traces left); even the Wikipedia entry for BGS Systems disappeared (Jeffrey Buzen reminds us of this). BGS Systems was acquired by BMC Software in 1998. BMC Software regularly renamed and re-shaped their products.

BMC TrueSight Capacity Optimization has some modeling functionality.

Two other products with a rich history are still alive (although both were acquired relatively recently):

There were simulation tools – Hyperformix was probably the most advanced tool to use for “white box” modeling as it had an advanced modeling language. Hyperformix was acquired by CA Technologies in 2010, and its traces disappeared there a long time ago. (well, the company may probably say that such products as Broadcom CA AIOps may include some hints of Hyperformix capacity management products).

I mention Hyperformix not only because of the tool (it is gone a long time ago anyway), but because they developed the approach I am advocating. For example, see some other pieces from Richard Gimarc and Amy Spellmann as “Moving Beyond Test and Guess” slides (CMG 2004) and “Full Lifecycle Performance Engineering” (CMG 2006); more articles can be found in the CMG resources section for CMG members.

The main advantage of the above-mentioned tools was they had a library of performance characteristics of different hardware – which allowed what-if analysis for different configurations without the need to benchmark all configurations you are considering (it was far from ideal, but better than nothing). See this and another important aspect of performance modeling discussed in my old post “Two Main Challenges of Performance Modeling and System Sizing” (considering that the IDC product mentioned it was gone a long time ago).

There are several tools open source tools, for example, PDQ Performance Analyzer from Dr. Neil Gunther and Java Modeling Tools a suite of applications developed by Politecnico di Milano and released under GPL license.

The following tools have introduced some notions that allow simulating more complex architectures, so maybe more appropriate for “white box” modeling:

  • SPE·ED – designed specifically to support the SPE methods and models defined in Connie Smith’s book
  • Palladio – an open-source tool using a software architecture simulation approach
  • QPME, (Queueing Petri net Modeling Environment) – an open-source tool for stochastic modeling and analysis. It is based on Queueing Petri Nets.

There are also many general-purpose simulation tools (offering discrete event simulation). However, they don’t have much IT-specific logic built-in. A couple of more known are OMNeT++, an extensible, modular, component-based C++ simulation library and framework, free for academic and non-profit use, and MathWorks SimEvents, a discrete-event simulation engine and component library.

I think we have a robust foundation and great history to reflect upon and learn from. However, it appears that modeling should be adjusted to today’s challenges. Maybe we don’t need such sophisticated models for a standalone machine when we have multiple processors, even in the most straightforward machines, and don’t load test them to the limit. But we need a way to model complex systems with dynamic configurations. Modeling is an integral part of performance engineering – and even more so when full-scale tests become rarer. Let’s revive the art!

Learn More about the Performance Advisory Council

Want to learn more about this event, check out the presentation here.

Leave a Reply

Your email address will not be published. Required fields are marked *