Learning to Expect the Unexpected: Load Test Before It’s Too Late!

At times, I can be a bit of a procrastinator. If there is no pressing need to clean out the garage this week, shop for holiday gifts, or do my taxes, I can easily be convinced to let it go until next week or next month. Inevitably, though, something happens that makes me wish I hadn’t delayed quite as long as I did. The garage gets flooded, for example, and the boxes that I had been meaning to organize get soaked and destroyed… or my Valentine’s Day gift doesn’t get shipped in time and I’m left holding the bag. I’m sure many of you can think of your own examples.

Unfortunately, many organizations tend to treat load testing as one of those activities that they can put off until a later day. You can almost imagine what they’re thinking: “Our website is up and handling the daily load with little difficulty. Yes, there’s a major advertising campaign—or new product launch, or major news release, or something similar—coming, but that’s a few months away, and we have plenty of time to prepare for it.” Then, inevitably something happens, testing doesn’t get completed and the website is flooded with traffic that it is unable to handle. If you’re the one responsible for the fall-out from such an event, it’s time to stop procrastinating and start load testing. And by load testing, I’m not just talking about throwing a bunch of simultaneous users at your website. I’m talking about generating a realistic load so you get results your developers and infrastructure team will actually pay attention to. This involves modeling real-world factors in your user loads, including users accessing your website and applications from mobile devices and from lots of different geographies. With proper load testing you can be prepared when the unexpected deluge of users hits your site. You’ll know what your site can handle, and how it will respond. More importantly, you can put mechanisms and procedures in place to help ensure your site will scale—or at least fail gracefully—and avoid the organizational embarrassment and opportunity cost of a major outage.

Perhaps you’re still not convinced that some event could happen tomorrow that sends a tidal wave of users your way. That’s what Florida Gulf Coast University (FGCU) thought in March of 2013, when the school’s basketball team scored an unprecedented and unexpected series of victories in the 2013 NCAA Men’s Division I Basketball Tournament. Suddenly, what was a relatively unknown school outside of Florida was thrust into the national limelight, and millions of people wanted to learn more about it. What happened? The university’s website crashed. At exactly the time when FCGU could present a positive image of itself to a wide, interested audience, the website failed under load. To be clear, I have no inside information about what tests FGCU may have performed if any, but I can infer that the tests were either inadequate or that the results were not taken seriously. As a result, what could have been a public relations slam dunk turned into a missed opportunity.

Sure, you say, but that’s a once-in-a-generation occurrence, and you’re a business not a university. Consider this. While watching the Super Bowl, I saw a rather cryptic commercial for a company I hadn’t heard of before. Like many people around the country, I wanted to know more so I visited the company’s website. What did I find? It had crashed.

crash

And they weren’t alone. Coke, Acura, and other big names experienced similar outages. It’s true that running a Super Bowl add does not really qualify as an “unexpected event”. Surely, many individuals in each of those companies knew the ads would run, yet still the web team was left looking unprepared. Perhaps they had not tested with a realistic, geographically distributed user load? It is not inconceivable that in some cases, the web teams were not aware that the ad would run. Can you say with 100% confidence that you know exactly when your company’s next big campaign will launch? What if a mistake is made—one outside the control of anyone in your organization? Is your website ready today to handle the traffic generated by such an event? If not, how will it fail?

So far, the examples I’ve cited have resulted in maybe a little embarrassment, a bit (or more) of revenue loss, and some brand tarnishing. But, it’s easier to imagine a more dire scenario. People depend on the web today like never before. Websites that serve the public interest and public safety are expected to perform and deliver up-to-date information that the public needs at all times.

During the terrorist hunt after the tragic marathon bombing in Boston earlier this year, the FBI had held a press conference alerting the public of the identification of the suspects. Among the discussion was the release of a URL that contained photos of the suspects. Living just a few miles from where the bombing took place I, like most of the people watching the press conference, tried to bring up the website and only to find a “website not available” message. And while the outage seemed to be resolved promptly, I can only imagine the ramifications if the site had remained down.

And while the latter example might be extreme, all of these situations had one thing in common: The organizations did not expect the causes of the failures to ever occur in the first place. If they had, obviously they would have done more to protect against it. So, instead of wracking your brain to think about all the possible causes of traffic spikes such as a water main breaking, a major storm approaching, or a competitor’s site going down, just test it so you know if and how you can handle the spike. This will ensure you have the answers to those questions you will be sure to get should an unforeseeable event impact your site. The fact is unlikely events occur all the time. Our job, as load testers, is to understand how our software will perform when they do. Since learning to expect the unexpected, I realize it’s all about preparation, not procrastination.

This weekend, I’m cleaning out the garage… maybe I’ll even get a jump on my holiday shopping.

No Responses
  1. January 30, 2014

Leave a Reply

Your email address will not be published. Required fields are marked *