As promised during my Performance Advisory Council presentation (which was devoted to the future of performance), here is a list of resources. What I am providing is just a portion of all the content. Check “The Forgotten Art of Performance Modeling” (video) and (slides) for details.
We see that the scale and sophistication of systems are exploding. We recognize that full-scale tests become less frequent, while continuous and component-level performance testing spread widely. The main point is that with all these trends, “white box” performance modeling (to be able to analyze changes in the underlying architecture) should be an integral part of every serious performance engineering efforts.
Performance modeling was a robust discipline, but it appears that it is now somewhat in decline. Not because it was based instead on advanced math (mostly queueing theory) and not because it is no longer needed (we have even more significant performance challenges today). It appears as if it just didn’t adjust to the changing technology paradigm.
The situation is better with production-level, “black box” modeling mostly based on data extrapolation: time series forecasting, trending, machine learning, etc. Actually another PAC session Predictive Performance Modeling using Machine Learning (ML) Algorithms covered that topic.
It doesn’t appear that the problem is that it is too difficult. There were multiple (and sound) attempts to explain performance modeling more straightforwardly, for example, Queueing Theory, In Practice. Performance Modelling for the Working Engineer by Eben Freeman or Applied Performance Theory by Kavya Joshi.
There are excellent courses in some universities – see, for example, Mark Friedman’s performance engineering class slides and videos (at the University of Washington).
There are good introductory books:
- Fundamentals of Performance Engineering; You can’t spell firefighter without IT Perfect
- Every Computer Performance Book: How to Avoid and Solve Performance Problems on The Computers You Work With by Bob Wescott, 2013
There are opinions that they are going too far in simplification, but they are an excellent place to start. The authors, by the way, worked for Hyperformix.
Books by Dr. Connie U. Smith focused more on software development, introduced Software Performance Engineering (SPE):
- Performance Engineering of Software Systems, 1990
- Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software by Connie U. Smith and Lloyd G. Williams, 2001
And, of course, we have a lot of great books about performance where the application of queuing theory is discussed in detail (including explanations of queueing theory itself). Here is a list of these in chronological order:
- The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling by R. K. Jain, 1991
- The Practical Performance Analyst by Neil Gunther, 2000
- Scaling for E-Business: Technologies, Models, Performance, and Capacity Planning by Daniel A. Menasce and Virgilio A.F. Almeida, 2000
- Measuring Computer Performance: A Practitioner’s Guide by David J. Lilja, 2000
- Capacity Planning for Web Services: Metrics, Models, and Methods by Daniel A. Menasce and Virgilio A.F. Almeida, 2001
- Computer Systems Performance Evaluation and Prediction by Paul Fortier and Howard Michel, 2003
- Performance Assurance for IT Systems by Brian King, 2004
- Performance by Design: Computer Capacity Planning By Example by Daniel A. Menasce et al., 2004
- Guerrilla Capacity Planning: A Tactical Approach to Planning for Highly Scalable Applications and Services by Neil J. Gunther, 2006
- Software Performance and Scalability: A Quantitative Approach by Henry H. Liu, 2009
- Analyzing Computer System Performance with Perl::PDQ by Neil J. Gunther, 2011
- Solving Enterprise Applications Performance Puzzles: Queuing Models to the Rescue by Leonid Grinshpan, 2012
- Performance Modeling and Design of Computer Systems: Queueing Theory in Action by Mor Harchol-Balter, 2013
- Foundations of Software and System Performance Engineering: Process, Performance Modeling, Requirements, Testing, Scalability, and Practice by André Bondi, 2014
There were (and still are) quite a few modeling tools which we may trackback to 1977 when BGS Systems released BEST/1 (seems there are few traces left); even the Wikipedia entry for BGS Systems disappeared (Jeffrey Buzen reminds us of this). BGS Systems was acquired by BMC Software in 1998. BMC Software regularly renamed and re-shaped their products.
BMC TrueSight Capacity Optimization has some modeling functionality.
Two other products with a rich history are still alive (although both were acquired relatively recently):
There were simulation tools – Hyperformix was probably the most advanced tool to use for “white box” modeling as it had an advanced modeling language. Hyperformix was acquired by CA Technologies in 2010, and its traces disappeared there a long time ago. (well, the company may probably say that such products as Broadcom CA AIOps may include some hints of Hyperformix capacity management products).
I mention Hyperformix not only because of the tool (it is gone a long time ago anyway), but because they developed the approach I am advocating. For example, see some other pieces from Richard Gimarc and Amy Spellmann as “Moving Beyond Test and Guess” slides (CMG 2004) and “Full Lifecycle Performance Engineering” (CMG 2006); more articles can be found in the CMG resources section for CMG members.
The main advantage of the above-mentioned tools was they had a library of performance characteristics of different hardware – which allowed what-if analysis for different configurations without the need to benchmark all configurations you are considering (it was far from ideal, but better than nothing). See this and another important aspect of performance modeling discussed in my old post “Two Main Challenges of Performance Modeling and System Sizing” (considering that the IDC product mentioned it was gone a long time ago).
There are several tools open source tools, for example, PDQ Performance Analyzer from Dr. Neil Gunther and Java Modeling Tools a suite of applications developed by Politecnico di Milano and released under GPL license.
The following tools have introduced some notions that allow simulating more complex architectures, so maybe more appropriate for “white box” modeling:
- SPE·ED – designed specifically to support the SPE methods and models defined in Connie Smith’s book
- Palladio – an open-source tool using a software architecture simulation approach
- QPME, (Queueing Petri net Modeling Environment) – an open-source tool for stochastic modeling and analysis. It is based on Queueing Petri Nets.
There are also many general-purpose simulation tools (offering discrete event simulation). However, they don’t have much IT-specific logic built-in. A couple of more known are OMNeT++, an extensible, modular, component-based C++ simulation library and framework, free for academic and non-profit use, and MathWorks SimEvents, a discrete-event simulation engine and component library.
I think we have a robust foundation and great history to reflect upon and learn from. However, it appears that modeling should be adjusted to today’s challenges. Maybe we don’t need such sophisticated models for a standalone machine when we have multiple processors, even in the most straightforward machines, and don’t load test them to the limit. But we need a way to model complex systems with dynamic configurations. Modeling is an integral part of performance engineering – and even more so when full-scale tests become rarer. Let’s revive the art!
Learn More about the Performance Advisory Council
Want to learn more about this event, check out the presentation here.