Mobile devices bring a whole new dimension to the computing experience. Not only do they provide many of the same services found in desktop computing, but they also offer additional features such as GPS reporting and photo distribution from camera to the Cloud. The fact is, without the proliferation of mobile device services Uber and Instagram would never come to be. Mobile computing has changed the game and so too has mobile performance testing.
The key to building a great mobile performance test strategy is to remember that you are testing mobile devices. While this may sound trite, the implications are profound. Mobile devices bring a unique set of challenges to performance testing. Need I remind you that mobile devices were intended to, well, be mobile. Therefore, the mobile performance testing strategy has to support this critical deviation from typical desktop computing.
The following tips are designed to help performance test practitioners ease the transition into mobile testing in a useful and practical manner.
Tip 1: Ensure that your Test Objectives Align with the Nature of the Business
What you test and how you conduct performance testing are each dependent on the nature of the business. Performance testing a smartphone will not look the same as say, testing for an on-demand scooter picked up by a rider on a street corner. Sure, both are mobile devices. And, there are standard features that each require to be tested – e.g., data exchange latency. But, a smartphone is not built nor expected to carry passengers just as the scooter won’t let you watch Star Wars: The Last Jedi. It’s no surprise then that your approach to performance testing should be as unique as it needs to be based on the mobile device type.
Okay, this is an extreme example. However, it illustrates an essential point: the business needs to dictate what/how to test for performance. The scooter business’s success criterion is likely to safely take passengers to and fro. Smartphone manufacturers want to ensure that calls can be made and photos taken.
Takeaway: As a test practitioner, you must always be aligned with the business intention and need, and your test activities are designed to meet these requirements explicitly and implicitly.
Tip 2: Start from the Ground and Work your Way up
Every year, new versions of mobile devices penetrate the consumer landscape. Today’s iPhone X is tomorrow’s dinosaur. Ongoing version change affects all aspects of testing, and performance testing is no exception. The working code on a newer device is probably not going to work well on an older one. This drives the thinking that an appropriate rule of thumb when performance testing a physical machine would be to start with the older version, working your way up. Let’s walk through this when performance testing an application. Start with the installation of the app on the oldest version of the device that the app is specified to support. Test against the release. Then, increase one newer version at a time until you have arrived at the current flagship model. The reality is, you’ll probably catch issues in the older version, first. Assume that if the app performs poorly on an older version, that’d be the safest place to start remediation, at least if this is possible with the release in doubt.
Tip 3: Don’t Test Everything (at first)
Proper performance testing needs to be comprehensive; we know this. But, since time is money, a progressive approach is most effective. In other words, test the low-hanging fruit, first, then progress based on complexity. Keep your test scenarios limited, but focused. For instance, at the onset separate client testing from server-side API testing. Run your load tests on each before testing entire transactions from client to server, back again. Test often and early; keeping in mind that the testing needs to use the time efficiently, mainly, when it’s conducted earlier in the development cycle.
By focusing on a small number of specific, critical testing points early, you’ll get a clear idea of the types of situations your application can/not handle. If you run into a bottleneck or bug, fix it before moving on to the next scenario – you don’t want any disasters.
Tip 4: Keep it Real
At the developer level, emulators are necessary. No single developer can be expected to have access to every smartphone, tablet, or Alexa type of device available. As you move further down the development cycle, hardware will start to matter. It’s important to know when to use emulators and vs. real devices on real networks. The golden rule: use emulators when you need to test application logic, functionality, and scale at the server end. It’s devices when you need to evaluate user experience, device performance/device-specific functionality, and activities based on geolocation.
Tip 5: Don’t Ignore Interruptions
One of the most significant behavioral differences between a mobile device and a PC is that mobile users can/will experience interruption regularly. On a PC, you switch from one window to the next, but your application keeps hums along in the background. With a mobile device, users are often in the middle of a time-sensitive task when a 30-minute phone call barges in on them. Who hasn’t experienced (seemingly every day at that) getting dropped from the network connection or watching the battery drain down to 0%? If nothing else, your personal experiences with mobile devices will make keep you acutely aware of the types of interruptions that may occur such that you can easily incorporate each into your testing strategy.
Tip 6: Remember, all Carrier Networks were Not Created Equally
Today’s mobile devices generally access the server over 3rd-party carrier networks varying significantly in speed, latency, and bandwidth, etc. Baking carrier network testing into your strategy will help you identify and resolve how your mobile application works across multiple types of networks – a crucial component which shapes the end-user experience.
Speaking of the users, be sure to consider their geographic location. Expect a 4G network in a major city to behave differently than a remote connection in a rurally. Your performance testing strategy should include a cross-section of geolocations to simulate as much realism as you possibly can. You are sure to want to see if your servers can handle the load whether it’s coming from Times Square or Mount Rushmore.
Tip 7: Be Thinking about the Next Device
As mentioned earlier, it was only a handful of years ago that few could have imagined a “scooters-on-demand” business model being commonplace. Today, they’re all over the place – each a mobile device in their own right, interacting with the Cloud to report location, deliver customer billing.
Behind each of these new mobile devices is a test practitioner who designed tests that ensure that the technology works as intended, both regarding hardware and software. Safety depends on it.
Determining the next big mobile device is always a bet. Some devices catch on and some won’t. The test practitioner’s primary challenge is preparation. What will that next device be? How can I/we be ready for it?
Anyone who’s been around for a while understands that technology evolves and that the rate of evolution is accelerating daily. The history of technology has demonstrated more than once that today’s fantasy can turn into tomorrow’s reality in virtually no time at all.
Putting it All Together
Performance testing is not a stagnant practice. It can’t be. Technology changes too fast. The evolution of mobile computing attests to this. According to Statista, in 2010, there were ~62 million active smartphones in the US. This year, the number has grown to ~237 million. That’s nearly 400% growth in four years – an astounding amount.
There are significant challenges today and tomorrow for performance test practitioners. New technologies appear regularly. Old ones still need to be supported. Hopefully, these tips described above will ease the transition fueling your ability to face and persevere amid the ongoing challenges.
Learn More about Mobile Performance Testing
See our Holiday Shopping Trends & Mobile Performance Testing blog as well.