The Heat Is On: Performance Testing in a Bind

We all have times that we’re in a jam. How familiar does this scenario sound? Your company is going to be announcing something big tomorrow, and a lot of people will take notice. It’s getting late in the evening and everyone would normally be home by now, but today the entire office is buzzing as everyone is preparing for the news. Your responsibility: make sure the website can handle all the traffic that’s coming. You hear every tick of the clock as hours stretch later into the night, and you see your manager popping in with a death glare – obviously eager to know how things are going, but not wanting to get in your way.

The pressure is on! But you can’t complete of the tests you’d normally execute with the limited amount of time you have. So where do you focus? This is a classic question for performance engineers, so we thought we’d help with some key strategies you can use when you are under short time constraints.

Settle on Your Metrics

When it comes down to it, if you can only concentrate on a limited number of metrics when conducting performance testing, you’ll want to focus on these three things:

  • Concurrency: how does performance behave when multiple users access the application simultaneously?
  • Transaction rate: how many transactions can the application process as load increases?
  • User experience: how does the user experience change or degrade as activity on the server ramps up?

As we’ve written about before, these key performance indicators are the ones to watch when testing in a hurry. They are the most basic variables in almost any load or performance test, and when you’re under pressure, metrics like these can act as your beacon. By identifying these KPIs and focusing on them up front, you are setting clear goals, and you know that you’re doing meaningful work and not just wasting time.

Clarify the Scope

As a load tester in a situation like the one described above, you may be tempted to jump right in and start testing when you know you have a lot to do and not much time to do it in. However, a better approach in a time crunch is to narrow the overall scope of the testing focus. Don’t let the pressure get to you. Instead, step back and ask, “What parts of the system need testing, what exactly needs to be done, and how much?” The answers to these questions will help guide you and ensure the testing activities you perform are high in value.

Context becomes important. Specifically, the type of traffic you’re expecting should influence how you prioritize your load tests. For example, if it’s a press release that’s going to be published, you are probably right to anticipate heavy overall traffic to your website in the form of people browsing pages to read the news and learn more about you. In this case, visitors will probably enter the site via the home page or a news link, and then spend a short amount of time exploring. Or let’s say you’re offering a marketing promotion (like Amazon Prime Day) – then you might get a lot of eCommerce traffic. If your marketing promotion isn’t well received (again, like Amazon Prime Day), you might want to expect a lot of activity on your social forums and customer support pages.

So before you begin, exercise those brain cells and prioritize which tasks are likely to be the most common – or the most problematic. The nature of the activity dictates where you need to spend your precious time.

Hand-Pick Your Scenarios

Despite the pressure from your deadline, it’s essential to include some variability in the load test design. However, while a typical load test might be more exhaustive, in this scenario you really want to pick and choose which tests you run. At a minimum, make sure the load generators are simulating both mobile and desktop users. You can rely on your website analytics package to tell you what kinds of browsers and operating systems your visitors are using, and then model that in your load test design.

It’s also a very good idea to have the load for your test generated outside your firewall to test the full network path. The cloud is a great option in this situation – not only because of the realism of the test, but also because you can quickly spin up different load generators with different scenarios without having to provision or configure any hardware. With the right tool, you can easily distribute loads across multiple geographies without adding too much work to your plate, giving you a more realistic test.

Emulators, Not Devices

The number of devices and mobile operating systems may be one of the biggest hurdles. Now is a good time to make heavy use of emulators as opposed to real devices. Again, variability is important. You want to include a variety of operating systems, screen sizes, and network types. Don’t just stick to one device type when generating a load. This is another good opportunity to leverage the information in your website analytics, and use those metrics to prioritize the top few devices that should be emulated.

You certainly don’t need to cover every combination and option when you are under a time constraint. Mix and match just enough to give you the confidence that you’re getting good coverage, even if it’s not comprehensive.

Pulling It All Together

Each performance test can take a long time, so it’s critical to stay organized. Once you figure out your metrics, environments, device strategy, and key user paths, write down what you are going to do in a simple plan – or even a checklist. Coordinate your tasks so that all the important user paths are taken care of, then walk through and distribute all the environments and device types to make sure everything is covered at some point in the plan.

Finally, estimate how much time each test will take before the deadline and make any tweaks you need to. With your plan in place, you are ready to make it happen. You’re bound to beat the clock with a strong strategy on your side.

Beat the Heat

Even when the heat is on, you don’t need to panic as long as you have a sound strategy on your side. Armed with a plan like this, you can stay two steps ahead of your visitors, one step ahead of the competition, and be right on target for an optimal user experience.

Photo: numb3r

Leave a Reply

Your email address will not be published. Required fields are marked *