There is a lot more to testing than just finding bugs. Today’s testers are part detective, part forensic analyst, and part social scientist. When building a testing strategy you need to examine a pile of data, develop an understanding of how real users behave, and use that information to create test scenarios that push those limits.
This is especially important when creating a strategy to test a mobile application. Understanding the dynamics of mobile application use can be confusing at times, so asking the right questions up front are extremely important. By combining a rigorous investigative and questioning methodology with your knowledge of performance testing, you can quickly get on the path to becoming a great contributor to your application’s success.
We have identified four key questions testers should be asking when building out their mobile application test plans in order to achieve realistic testing scenarios. It’s important to note that there are more questions you should be asking at any point during the testing process, however these four questions are simply here to guide your thinking on how to make the testing process as realistic as possible. Remember, the more realistic your testing process, the more prepared your application will be for real users.
1. Where are my users located?
Understanding the geographical location of your users is important because in reality they aren’t accessing your servers from a single location. In a realistic scenario, users will be spread out across multiple locations, and those dynamics can greatly affect the way load is placed on your mobile application.
Ask yourself, from where in the world is my application being accessed? For a new app, you might need to reach out to your marketing team to understand where in the world they are marketing this application. You might find out the application is being released in North America as well as Europe, which means you need to generate load according to these locations during your testing scenarios.
By generating load according to the geography of users, you can assess the performance of your application accurately. After all, if you are testing the performance of your application you want all users, regardless of their location, to share the same experience and to fully know how your data center will react if your users are accessing the application across international borders.
2. When do my users access my application?
In order to create a realistic testing scenario you have to take into consideration the real-world traffic patterns your users face when they interact with your servers. This is a key part of mimicking behavior, and necessary for recording traffic and replicating app usage.
Maybe your application will be accessed consistently throughout the day, or maybe there is a spike when people get to work in the morning. If your application has been created for a major event, like the Superbowl, you will need to be prepared for an overwhelming traffic surge. Of course you cannot always predict if your servers will ever experience a surge in traffic. For example, the Oscar selfie that crashed Twitter’s servers was quite unexpected (at least for Twitter).
As a general rule of thumb, getting a good understanding of when users are accessing your application will lead you to better simulate traffic spikes. That way you can proactively protect against their damaging effects and be prepared just in case.
3. Are my usage assumptions accurate?
As applications undergo continued development, their performance characteristics continually change and evolve. Bringing performance testing into a Continuous Integration (CI) environment helps to keep your fingers on the pulse of your applications behavior, but what about when usage patterns change? For example, when you release a new feature, you may receive an influx of new users. This may impact how you design your test scenarios.
If your application is new, you really don’t actually know what amount of load it can handle in production. In order to find out, you need to do a capacity test. By executing a capacity test on your production hardware, you’ll develop some baselines and benchmarks about how your application behaves under various assumptions. Then, execute the same tests on your dedicated test systems. Document all these assumptions and the results, so that when you push the application out into the field you have a frame of reference for your new users’ experiences.
Once your application is out there running in production, you may not be able to run an adequate load test in the production environment anymore because it could impact real users. If testing in production isn’t an option for you, you may want to begin doing performance regression testing. This type of testing consists of doing a stress test of all the individual features of your application in the testing environment to measure its performance. Compare these results with your testing and production baselines taken earlier. As you make changes to your application, re-run the performance regression test to see if the results have changed drastically from your baseline. If these new results didn’t change much on your development server then you should be safe to deploy the application to production without your server utilization changing. If there is deviation, you can prepare appropriately.
4. Where are the bottlenecks in my application?
It’s important for any tester to study the architecture of the application to identify weak spots. This type of testing is known as glass box testing, where you are testing the software with knowledge of the internal structure and coding inside the program.any questions you find asking yourself during mobile performance testing please leave it in the comments below, we would be happy to answer them and even include them in our next blog post.
If you don’t have insight or knowledge of the code, reach out to your development team and identify bottlenecks in conjunction with them. It’s important once you find bottlenecks to build stress tests around them, as they are the weakest links in an application and must be solved strategically in order to avoid a disaster. Case in point: when Healthcare.gov failed to address the bottlenecks in their software, it ultimately led to its widely broadcasted failure and negative brand recognition.any questions you find asking yourself during mobile performance testing please leave it in the comments below, we would be happy to answer them and even include them in our next blog post.
Always Ask Questions
As a tester, you should always be asking questions; doing so will only enhance your career as you become more inquisitive and driven to make an application perform optimally. As mentioned above, these are not the only questions you should be asking. There are other factors to consider with mobile performance testing like network conditions, platforms, devices, and so on. If you have any questions you find asking yourself during mobile performance testing please leave it in the comments below, we would be happy to answer them and even include them in our next blog post.