So you are new to the mobile performance testing world. Maybe this is your first job as a tester and you have been tasked with the project of setting up the test framework for a mobile application. Or maybe you’ve just released a mobile app to go along with your web-based version, and you need to figure out how to properly assure its quality. Either way, it’s your job to make sure the application performs under stress so that your end-users have a smooth, interactive experience.
What you should know is that consumers have become picky with mobile applications. According to a survey conducted by Radware, nearly 85% of mobile users expect sites to load at least as fast or faster than a site on their desktop. Application developers have long understood the need for load testing conventional desktop web applications to ensure that they will behave properly under load with the expected number of users. With the advent of mobile apps and mobile websites the principles of load testing have not changed.
Your job will entail testing mobile applications for performance in order to protect the brand integrity of your company or client. In this post you will take away a general overview of mobile testing, how to construct a mobile performance testing strategy and where to begin.
How Mobile Performance Testing Fits Into Your Overall Plan
Before you get started it’s important to understand the scope of tests covered within your mobile performance test plan. The best way to do that is to understand how mobile performance testing contrasts with functional testing and with web testing.
Performance Testing vs. Functional Testing
Functional testing ensures the application is working as per the requirements. It answers the question, “Does the app do what it is expected to do?” While performance testing checks the performance and behavior of the application under certain conditions such as heavy user load, peak usage times, low battery, bad network coverage, low available memory, simultaneous access to application’s servers by several users and more.
Performance of the application can be affected from two sides: the server side and the client side. Performance testing is carried out to check both of these attributes.
Web Testing vs. Mobile Testing
Web performance testing is different than mobile testing. Desktops and laptops do not suffer from poor network conditions including packet loss and latency at nearly the rate they used to, which means testing for these failure scenarios are typically back-burnered. In mobile performance testing, however, network conditions, the types of device, packet loss, latency and bandwidth all matter. If you ignore these elements of mobile testing it will surely affect the end-user experience.
By emulating different conditions in your testing scenarios you can better understand the effects of changes in connectivity on the application’s performance. Mobile networks have limited bandwidth and high latency compared to WiFi and broadband. Which means you should test your application under limited bandwidth to see if your users will get the best user experience and ensure your servers won’t have a problem under load. To read more on the differences between mobile and web performance testing read this post.
Understanding Your Application
As you develop your mobile testing strategy, an important starting point is understanding the type of application you are running. This will guide the suite of tests you want to focus your limited efforts on. There are three types of mobile applications, each of which dictates a specific set of testing requirements.
The first type is a browser-based application. These applications are delivered directly through a mobile browser, without requiring any software to be downloaded or installed on the device.
Browser-based applications are lightweight, centrally managed, and have an added benefit of being built on the same stack no matter what device they are delivered to – iPhone, Android, Microsoft, PC, or otherwise. In fact, the responsive web design movement has it even easier to create multi-platform browser-based applications because you can render the app to different screen sizes from within a single application framework. Browser-based apps do have their drawbacks, however. They can’t be accessed without an internet connection, they don’t have access to all the features built into the device, and they tend to appear slower and more sluggish to end users.
Takeaway: While testing for performance, it is important to replicate the user server load from mobile browsers in case additional dedicated components are being deployed to cater such requests. It is also important to test Web page rendering on target devices (like CPU, memory or browser rendering engine) as it can affect end-user performance.
The second type is a native application (native app), which is has been developed to run on a particular platform using software installed directly on the device.
Because native apps are written for a specific platform, they can interact with and take advantage of device-specific hardware and software, such as a global positioning systems (GPS), accelerometers, and cameras. Native apps also tend to be faster and more responsive to the user because much of the functionality runs locally on the device. By the same token, however, native apps are more difficult to deploy and maintain when compared to their web-based counterparts, since they have to be updated directly on the device.
Takeaway: This leaves the development group with the task of having to develop and test diverging codebases – essentially different versions of the app – on each supported platform (iOS, Android, Blackberry, Windows Mobile, and so on). Since the code on each device is in fact different, testing a native application calls for thorough performance testing on each target platform as well as maintaining a set of test devices in each selected platform.
The third type is a hybrid application (hybrid app), which is an application that combines elements of both native and Web applications. These apps are typically composed of a native shell application that provides a fast, integrated experience with the device, but the content, business logic, and even the rendering of the user interface is done with an embedded browser. This solves some of the challenges inherent in pure browser-based applications by providing a faster, richer shell application, while maintaining the ease with which browser applications can be modified and deployed. You still can’t use the vast majority of the application’s functionality without internet access, though.
Takeaway: Hybrid mobile applications still require separate development efforts and pools of testing devices. However, they don’t demand as much rigor in testing since much of the codebase is in fact shared. Again, a performance test strategy has to target the load generated by users of such hybrid apps on the server-side, as well as gauge the on-device application performance from an end user experience perspective.
Building the Client Testing Environment
Another key piece of your testing strategy is the client environment you set up to simulate real users. You’ll be making choices about the types of devices running your app, as well as the geographical location of those devices.
Choosing Your Clients
In order to test the performance of your mobile application you need to construct a client environments consisting of emulators or real devices. While these approaches have their differences, one is not universally better than the other. How you mix & match clients simply depends on your testing situation.
Mobile emulators are software platforms installed on a laptop or desktop that imitate a mobile platform or OS environment. This allows testers to see how their application behaves on an emulated phone. The great thing about emulators is that you can generate a significant amount of load, which is what you want to do when you are testing the performance of your application.
Real devices allow a tester to physically switch from device to device in order to see the performance of the application on an actual device. Using real devices gives you the ability to test your app on live networks. However, it does not allow you to vary signal strength, latency, bandwidth and packet loss of multiple devices to a connected network. And it also does not allow you to generate enough load. You would need to have multiple devices handy to generate any significant amount of load to test the performance of your application.
If you are looking to test the behavior of your application on a specific device you need to use a real device. However, if you are looking to understand the performance of your application under load you will probably be best served by an emulation tool.
Creating a Geo-Realistic Testing Environment
If you are creating a testing environment without considering the geographical locations of your users you are doing it wrong. It’s your goal to try and create as close to a realistic testing environment as possible, without understanding where your users are accessing this application your application will surely suffer.
Ask yourself, where in the world is my application being accessed? Are your users operating out of one city, or are they spread across multiple continents? Once you have decided where, geographically, your users are then it’s time to place load on the application in a way that is consistent with what your real users experience. After all, if you are testing the mobile end user experience from the same building as your data center, but real users are crossing international borders to reach your data center, then how can you trust the performance characteristics you discover?
Where to Get Started
You have a better understanding of mobile performance testing and creating a strategy, but if you’re hungry for more, be sure to check out our whitepaper Addressing Mobile Load Testing Challenges. And if you think you’re ready to start testing but don’t have the tools, take a look at how NeoLoad enables mobile testers with device simulation, network emulation, Cloud load generators and more.