As a tester you are often placed in many difficult situations. You act as the gatekeeper between your company and your consumers, which can create a lot of pressure. You are after all the person who must tell the team, “No, we can’t put the app into production because it can’t handle the load.” And who wants to hear the word “No?”
As a gatekeeper, you are an easy scapegoat if something goes wrong. At that point, when all eyes are on you, you might find yourself making the #1 excuse testers everywhere make, “Well, we didn’t have the right testing environment.” This explanation is often the product of a testing team under too much pressure and, as a result, compromising the quality of the work.
Compromises are sometimes necessary, but you have to make them with eyes wide open. In order to avoid sacrificing performance and giving into pressure, we will break down the #1 excuse by presenting some of the common reasons you might be blaming the testing environment.
Reasons Why You Might Blame the Testing Environment
You don’t ever want to be in a situation where someone comes to you after a disaster and says, “Why did your testing not pick this up?” So the more you can understand the trade-offs you make that put you in that awkward spot where you are forced to blame the testing environment, the easier it will be to spot them and address them up front. So let’s dig in a little bit:
“We didn’t have the right testing environment because…”
1. It wasn’t representative of real users
Trying to create a testing environment that is representative of your users is tricky, even if you have all the time in the world. User behavior is complicated – often erratic. Try simulating the patterns of thousands of irrational users in a series of automation scripts – it’s next to impossible!
So before you rely on those test scripts to tell you how well your application will perform, take some time to get as close to real users as possible. Start by monitoring the behavior of users on your website and make sure that your simulated traffic patterns are reflective of real usage. You may also want to record this traffic and play it back as part of your performance testing to get even closer. And of course, don’t forget the importance of making your tests geo-realistic.
2. It’s too different from our production environment
When your performance testing environment was created, was it meant to look like your production environment? Or was it just an environment where you could test specific aspects of scalability and stress? Sometimes we don’t see the forest through the trees. It can be easy to continue performance testing the way it’s always been done without realizing that as your application matures, so too must your processes.
Revisit the goals and objectives of your environments and procedures on a periodic basis. Start with your needs first: what risks are you trying to prevent through healthy performance testing? From there you can determine what tests you need to put in place to mitigate those risks, and what kind of an environment you need to support those tests. Because you don’t ever want to be hit by a performance failure you never even intended to test for.
3. The testing environment has drifted out of date
This is probably one of the most common problems with maintaining separate testing environments. Think about every time a software upgrade or hardware build-out is conducted in your production environment. Have all those same steps been executed within your testing environment in the same way? Even the most robust data centers with highly coordinated change management processes struggle with this one. Then – through in a late-night emergency server rebuild – and all hopes of production/testing synchronicity are out the window.
It may not be possible to develop and operate a bulletproof change management process. However, there are a few things you can do to avoid too much drift. Keep a list of the critical pieces of software and run a script to periodically check what versions of each of those software packages are running. Or invest in a lightweight inventory/asset tracking solution that you deploy to both your production and testing environments. Set aside a task every month or even every quarter to review these reports and confirm that nothing has drifted too far.
4. We don’t have enough resources to test properly
The amount and quality of resources you have access too might be affecting your testing environment. We have all been there before: sometimes you just have to deal with out-of-date equipment and just make the darn thing work. Or when there simply aren’t enough hours in a month to complete a critical test.
If you are being asked to make a Ford run like a Ferrari when you’ve only been given a hammer, you have two options. Option 1: make the best of it. Option 2: identify the level of risk that is being assumed and broadcast it to the world. We suggest taking both options. You have a job to do – so make sure the job gets done as best as you can. But part of your job is to communicate problems that may impact the performance of the application later on. And if you don’t have the resources you require to do that, then that itself is a problem. Make sure to let whoever is in charge know the limits of the testing environment based on the resources handed to you. Remember, the squeaky wheel gets the oil.
5. We were under too much pressure to finish up testing
Throughout this post you can get the idea that we understand you are under immense pressure from co-workers and management to get your testing done in a timely fashion. After all, the quicker you can get the application in the hands of users the faster your company can capitalize on the huge investment they made in building the application in the first place. But the pressure to release the application into production too soon can cause major problems. Perfect example: the Obamacare website (read more about that disaster here).
Stay strong in your decision to work on the application a little longer. Explain to whoever is trying to bully you into releasing the application prematurely that it will cost them more time to fix the application after production. Figure out ways to communicate that evoke a sense of teamwork and cooperation. After all, everyone is after the same thing: success.
Listen to Your Gut
Throughout your career as a tester you will be placed in tough situations where you will have to compromise with your co-workers. Do the very best you can with what you have, and build strong and trusting relationships with your coworkers that allow you to tackle these difficult issues before they become nightmares.