Are you attempting load and performance testing in an Agile environment but find yourself asking, “Am I doing this right?” Maybe you have a number of tests that are running with your continuous integration builds but haven’t been able to effectively incorporate load and performance testing into that automation.
Don’t worry – you’re not alone. Many organizations today are struggling with questions about load and performance testing in an Agile or continuous testing process. To help you out, we have collected answers to common questions from our performance guru Steve Weisfeldt, taken from our webinar Load and Performance Testing at the Speed of Agile: Best Practices and Automation Tips.
How do you handle code churn? Are you constantly maintaining your scripts?
The short answer is yes. Think of it as a development effort. The idea is: when we maintain scripts in an Agile environment, the changes should be small to implement. That’s the challenge we have as performance testing engineers if we sit and wait until the end of the cycle, especially if our cycles are far apart.
In the traditional waterfall approach we constantly get asked by clients, “OK, we have a new release of the app and it’s so different that we need to rewrite the scripts.” But if I take that and move it up earlier in the cycle or become more Agile, the application changes are significantly smaller. So if I can have that process built-in, the impact I can have on my performance testing becomes much smaller as well, which is ultimately what you want.
How do I choose SLAs (Service Level Agreements) for my application?
That’s a great question. Again, this isn’t specific to Agile but more to load performance testing in general. And that is how do I even know what I’m looking for? There are a number of different ways to do this.
First of all, have a conversation with the business. They know what the users expect and what they think will be good or bad performance.
Second, talk to the folks who have developed the requirements in the app. It may be there are performance requirements built into the app already before the app was developed.
If it’s an app in production, talk to your end users and ask them what is good and bad about the application. Don’t make the mistake of just looking at industry standards because I believe each application has different SLAs. I see people who fall into the trap of Googling SLAs and saying, “OK, 8-seconds response time is my SLA. I’ll use that as a part of my testing,” instead of asking their end users what good performance actually looks like.
It’s almost impossible to get a realistic production environment. Do you suggest scaling as a resolution?
It’s a bad assumption to think that performance is linear; very often you can’t get results that way. And that’s the challenge we have with performance testing, companies don’t always have the ability to mirror production. That’s what I want you to have, but typically that’s not the case.
How do we get around that? Unfortunately, we do the best that we can. And that’s why I made the comment before that performance testing is not an exact science, we can’t eliminate risk, but we can mitigate it. So if your production environment has four web servers and three databases on the backend, and your QA only has two web servers and one database, the question is: can you extrapolate those results? The answer is no. And that’s where we start to see more of testing in production.
And I grimace when I say that because I know there’s a lot of negative ramifications, but if it’s done right, it can work. Having said that, there is a lot of value found if you can test small instances of your environment. If your staging environment is smaller than your production environment, then you have a lot of opportunity to knock out a lot of things. But I don’t want you to assume that if your staging environment can handle 200 concurrent users and your production environment is twice as big, then that means you can handle 400 concurrent users. No, it doesn’t work that way.
Any More Questions?
We hope these answers give you a better picture of Agile and performance testing. If you have any additional questions regarding Agile or performance testing leave us a comment below or check out our whitepaper Load Testing at the Speed of Agile.