“Mobile performance testing is the same as web application testing. Right?”
The answer is no, web application testing might share the same basic principles with mobile application testing but there are more factors to consider with mobile apps.
Your users’ mobile experience will be affected by the type of device, network conditions, device features, packet loss and a whole lot more. All of these factors necessitate a sound mobile performance testing strategy. The following tips are not all-inclusive, but from our experience we believe these are the most important tips to take into consideration when building a mobile application strategy.
1. Ensure that your test objectives align with the corporate and business requirements
If your test objectives don’t align with the needs of the business, then you risk testing the wrong aspects of your application – potentially missing a use case or scenario that is critical to the success of the endeavor. Here are a few questions you can ask to align your testing practices with the business objectives:
- What aspect of the experience drives meaningful business metrics (revenue, advertising, engagement, etc).
- How do competitive applications perform relative to these metrics, and how does your application differentiate?
- What are the requirements of the user experience relative to these metrics?
- Are there any situations that require specialized logic or behavior – for example, different connection types (WiFi vs. cellular), usage while international, or experience while not connected to a network?
2. Identify your Key Performance Metrics (KPIs) for application performance and pay special attention to them
Your KPI’s will differ depending on your business objectives. Make sure to clarify your goals in the beginning to be able to provide concrete results to your team when the application is ready to be released. Here are a few important KPI’s to consider when testing your mobile application:
- Average requests per second
- Average request response time
- Maximum response time
- Average throughput
- Error rate
3. Don’t test everything… at first
Especially in the first rounds of testing, you don’t want to focus on all the different combinations of scenarios. When your account for both Initial testing and subsequent regression tests, executing all tests across every combination will be both time and cost intensive. Use a combination of your KPIs and some behavioral analytics to determine the 10-20 combinations you wish to test for the first round of testing scenarios.
By focusing on 10-20 combinations you will have a clear idea of the types of scenarios your application can and cannot handle. If you run into a bottleneck or bug, make sure to fix it before moving on to the next testing scenario, you don’t want any disasters.
4. Keep it real
Don’t ignore the importance of real devices when testing – they don’t scale very efficiently, but they do provide real user experience information. Use emulators to scale your testing operation, but remember that they won’t provide you with information on the full real user experience. It’s important to know when to use emulators and when to use real devices on real networks.
- Use emulators when you need to test application logic, functionality, and to scale at the server end
- Use devices when you need to test user experience, device performance, and device-specific functionality
5. Align your approach to your development methodology (waterfall vs. agile)
Many of the decisions you make as a tester should be made with your company’s preferred development methodology in mind because how your mobile app is developed impacts the feasibility of the testing approach that best works for your company.
If your company takes on the Waterfall methodology work on test plans. Build out the testing environment while development is taking place. Ramp up testing as development transitions towards a functional-complete state. And be ready for the onslaught of big finding, bug fixing, regressing testing, and triage that follows.
If your company uses the Agile methodology create automated processes that fit into your continuous integration platform. Turn test requirements into user stories and treat performance problems like every other bug.
Ultimately, you want to make sure that your developers and testers are working together as part of the same team as much as possible. They should share the same vision, speak the same language, and support in other in accomplishing their goals as an integrated unit.
6. Don’t ignore interruptions
One of the biggest differences between the way a mobile device behaves and the way a PC behaves is the fact that mobile users experience interruptions all the time. On a PC, you switch from one window to the next, but your application keeps on humming along in the background. On a mobile device, your users could be in the middle of a complicated task on a small screen when a 30-minute phone call interrupts the whole process and all context is lost. Be aware of all the types of interruptions that may occur and integrate them directly into your testing strategy. Here are some of the interruptions you may encounter:
- Incoming and Outgoing SMS and MMS
- Incoming and Outgoing calls
- Incoming Notifications
- Battery Removal
- Cable Insertion and Removal for data transfer
- Network outage and recovery
- Media Player on/off
- Device Power cycle
7. All carrier networks are not the same
Today’s mobile devices generally access the server over 3rd-party carrier networks that vary greatly in speed, latency, bandwidth and more. Including carrier network testing into your strategy will help you understand how your mobile application will work across multiple types of networks, which matters from an end-user experience.
It’s also important to consider the geographic location of your users. You can expect a 4G network in a major city to behave differently than a remote connection in a rural area. In your strategy you should include different geographical locations to simulate a realistic testing environment. This will allow you to see if your servers can handle load from multiple areas.
Come Up with a Strategy Before Testing
Before you go ahead and test your mobile application, please make sure to take into consideration these tips when creating your strategy. Your strategy to testing is your “plan of attack” make sure to have it planned out before any testing commences. Your team – and your users – will thank you in the long run.