How to Make Your Load & Performance Testing More Continuous

It’s great that everyone seems to be becoming more Agile these days with their software development workflows, and testing is becoming more continuous to follow suit. The problem comes when people are telling you that you should become more Agile and automate more testing without giving you practical advice on how to do it.

Previously, I wrote about the challenges and benefits of doing load and performance testing in an Agile environment. These four recommended best practices can help you maximize those benefits while helping you overcome the challenges:

Put Performance SLAs on the Task Board

Every application has minimum performance service level agreements (SLAs) to meet, but Agile teams are often more focused on adding features and functionality to an application than optimizing the application’s performance. User stories are typically written from a functional perspective (e.g. “As a user, I can click the ‘View Cart’ button and view the ‘My Cart’ page.”) without specification of application performance requirements (e.g. “As a user, I can click the ‘View Cart’ button and view the ‘My Cart’ page in less than 1 second when there are up to 1,000 other users on the site.”). This may not be the best way to write a user story, but it illustrates the point: Performance needs to be somewhere on the task board if the team is going to give it attention.

One way to get performance on the board is to use your SLAs as acceptance tests for each story so that a story cannot make it to “Done” if the changes for that story cause the application to miss those SLAs (this may require new SLAs to be defined for new features and new apps, e.g. all searches for a new search feature must return results in less than 2 seconds). This approach works well when the changes made for the story affect a relatively small section of the codebase and performance issues would therefore be confined to a small section of the application.
For SLAs that are general across the entire application (e.g. all pages must load in less than 1 second), tests should be added to a larger list of constraints (which may include functional tests) that get tested for every story in order to determine that story meets minimal “definition of done” without breaking any of these constraints.

Work Closely with Developers to Anticipate Changes

One of the benefits for testers working in an Agile environment is that they typically learn about updates on development tasks during daily standups or similar meetings. In order to get the maximum benefit from this level of collaboration, testers should constantly be thinking about how the stories that are currently being coded will be tested. Will these require completely new load tests? Will they cause errors in current test scripts? Can you get away with slight modifications to current test scripts if you plan ahead? Most of the time, these are small changes, so testers can stay ahead of the curve if they keep engaged with the team.

Integrate with Build Server

Even if you aren’t completely on the Agile bandwagon yet, you probably have a build server that kicks off some automated tests, unit tests, smoke tests, regression tests, etc. In the same way that performance goals need to be added to the task board, performance tests should be among the tests that occur with every build. This can be as simple as setting up a trigger to have the build server kick off the test, but could include displaying test results within the build tool depending on how sophisticated the integration is. Ideally, you want the person who kicked off the build to instantly see the results and know which changes went into that build so they can be fixed if there is a performance issue.

CI + Nightly Build + End of Sprint Load Testing

The difference between Continuous Integration builds, nightly builds, and the builds produced at the end of sprints can be huge. We’re talking the difference between a single change committed to a version control server versus all the changes committed in a day versus all the changes committed during a sprint. With this in mind, you should adjust your load tests to the type of build you’re running.

The best practice here is to start small and internal. For CI builds that are getting kicked off every time someone commits a change, you want these tests to run quickly so that you can get results back to that developer about how his/her changes affected the system. Consider running a small performance test with the most common scenarios covered with the typical load on your application being produced from your own internal load generators. For nightly builds, ramp it up to include more of the corner case scenarios and increase the load to what you see at peak times to see if any performance issues were missed during the CI tests. At the end of the sprint, you’ll want to go all out: consider generating load from the cloud to see what happens when users access your app through the firewall. Make sure every SLA on the constraints list is passed so that every story completed during the sprint can be marked as “done”.

The best thing about working in an Agile environment is being able to make changes iteratively. You can take each one of these practices and try them out for a few iterations. See how they work for your team before starting some of the more difficult practices around automation. Remember, Agile is all about delivering incremental value.

Not sure if your tool can keep up with your Agile workflow? Stay tuned for my next blog on how to choose a load and performance testing tool that fits the needs of your Agile teams.

No Responses
  1. February 21, 2014

Leave a Reply

Your email address will not be published. Required fields are marked *