Whitepaper
Addressing Mobile Load Testing Challenges

Introduction

With the advent of the app economy, mobile apps have revolutionized how business gets done. Mobile applications are not only the primary channel for the millennial generation but also are a significant driver of the Internet of Things. Mobile apps and websites process billions in business transactions, provide the primary communication conduit of brands, gather reams of data that fuel analytics initiatives, and transform the customer experience. In short, mobile apps undergird much of the economy.

In the past, when mobile played a smaller role in the business application ecosystem, performance issues and outages were considered an inconvenience; this is no longer the case. Today, performance issues that impact mobile applications tie directly to revenue loss, brand damage, and diminished employee productivity. The ubiquity of mobile apps that instill delight in the minds of customers is now the benchmark of broader app performance. Besides, as organizations rely on mobile channel transactions to enhance revenues, they apply relentless pressure on development teams to deliver meaningful consumer experiences. These factors together present a moving performance target for app DevOps teams and drive the need to create tests that accurately represent real users regarding network conditions, specific devices, and geographic locations.

Application developers have long understood the need for load testing conventional web applications to ensure proper behavior under load. These same principals of load testing apply to mobile apps and mobile websites. There are, however, challenges specific to mobile load testing that must be addressed by the load testing solution.

Because mobile apps and applications accessed by desktop web browsers use the same underlying technologies, there is good news– most load testing tasks and challenges are the same. Testers don’t necessarily need a new, mobile-specific load testing tool. They do, however, require a high-quality webbased load testing tool capable of handling the nuances of load testing mobile apps. Using a tool that enables testing of both traditional and mobile web applications allows QA to leverage existing in-house skills for designing and parameterizing test scripts, running tests, and analyzing the results.

Note that there are three fundamental differences to consider when executing traditional vs. mobile load testing:

Simulating network for wireless protocols: Mobile devices can experience slower, lower quality Internet connections than their desktop counterparts depending on the speed of wireless protocols available. 4G wireless protocols and future low-latency/high-speed networks, such as 5G, will impact client-side response times and the server itself, which testers will need to account for as they define tests and analyze results. Additionally, latency and packet loss become more of a factor with mobile applications.

Recording on mobile devices: Obviously, mobile apps run on mobile devices, making it difficult to record test scenarios, particularly for secured applications that use HTTPS.

Supporting a wide range of devices: The plethora of mobile devices and their respective operating systems demands that web application designers tailor content based on the rendering capabilities of the client platform, presenting challenges for recording and playing back test scenarios. To further complicate matters, each device has a different hardware configuration (i.e.CPU, memory), cache size, the method for handling requests, etc. All these factors conspire to increase ecosystem complexity, making it difficult to ensure a consistent user experience on each device.

This paper discusses the challenges associated with mobile load testing, as well as solutions and best practices for recording mobile load test scenarios, conducting realistic tests, and analyzing the results.

Name
Mobile Load Testing Challenges
Type

Try NeoLoad, the most automated performance testing platform for enterprise organizations continuously testing from APIs to applications.

 

 

Mobile Load Testing Basics

As you may know, a typical automated functional test for a mobile application emulates user actions that include tap, swipe, zoom, and text entry on a real device or an emulator. A specialized mobile device testing tool measures the time needed to render a page on a specific mobile device. The objective of loadtesting, however, is not to test the functionality of the application for a single user. The goal is to evaluate how the server infrastructure performs when handling requests from many users and to understand how other users are interacting with the application impacts response times.

A practical and realistic load test simulates a high volume of simultaneous users accessing the application server. Using real devices or emulators for this task is impractical because it demands acquiring, configuring, and synchronizing hundreds, thousands or even hundreds of thousands of actual devices or machines running emulators.

The current, hyper-competitive market climate underscores the importance of mobile application performance as experienced by the user. Unfortunately for testers, the mobile user experience is profoundly influenced by the characteristics of the device and the quality of the telecommunication provider network, making it difficult to accurately replicate all production conditions with conventional testing tools and approaches.

Fundamental questions that can be difficult to answer include:

  • How is it possible to validate the stability of the platform, the stability of the response times, and a good user experience under load?
  • How can you measure the user experience while staying under budget for the project?

The solution is to apply a protocol-based load testing approach designed to scale as needed in conjunction with real mobile device testing tools. With a client-based approach, user actions in the browser or the native application are recorded and played back. In contrast, a protocol-based approach involves recording and reproducing the network traffic between the device and the server.

To verify performance under large loads, tools that enable protocol-based testing are superior to those that support only client-based testing, because they can quickly scale up to millions of users while simultaneously checking errors and response times for each user.

The primary process for protocol-based mobile load testing is:

  • Record the network traffic between the device and the server
  • Replay the network requests for a large number of virtual users
  • Analyze the results

However, a protocol-based approach will not retrieve every metric of the mobile user experience, because it excludes the rendering time of the mobile device in the response time metrics. Remember that every mobile device has its operating system version, cache, the method of sending requests, memory capacity, processor capability, etc. All these variables influence the user experience across IOS, Windows, and Android devices. To measure the complete mobile user experience while the application is under load, it is important to combine a small number of users from a real mobile device testing tool with the protocolbased approach. Only in this manner can testers assess the impacts of bandwidth, latency, packet loss and throughput on the actual user experience.

It may appear straightforward, but there are challenges at every step. The good news is that these challenges can be addressed with an effective load testing approach.

Automate API and end-to-end application performance testing with NeoLoad, the continuous performance testing platform.

 

 

Recording Mobile Load Testing Scenarios

To generate a mobile test scenario, you first need to identify the type of mobile application under test. Challenges associated with capturing the data exchanges between a mobile application and the server depend on the design of the application and involve:

Native Applications – These apps are coded using a programming language (Objective-C for iOS, Java for Android) and an API specific to the device. As such, they are tied to a mobile platform and are installed from an online store or market.

Web Applications – Built with web technologies (such as HTML and JavaScript), these applications can be accessed from any mobile browser. More sophisticated web apps may use advanced features, such as geolocation or web storage for data or even include customizations to match better the browser used. Two common web apps are http://touch.linkedin.com and http://m.gmail.com.

Hybrid Applications – A web app embedded in a native app is known as a hybrid app. The native component of the application is limited to a few user interface elements such as the menu or navigation buttons, and functions such as automatic login. The main content displays in an embedded web browser component. Facebook’s app, installed from an online store or a market is a typical sample.

Recording Tests for Native Applications
Because native apps either run on a device or within an emulator, testers must intercept the network traffic that originates from either the real device or the emulator to record a test.

To intercept this traffic, the equipment that records the traffic must be connected to the same network as the device. When the recording computer is connected to the intranet behind a firewall, it is not possible to record a mobile device connected via the wireless network; the device and the computer running the recorder must be connected to the same Wi-Fi network.

Most load testing tools provide a proxy-based recorder, which provides the easiest way to record an application’s network traffic. With this approach, testers need to configure the mobile device’s Wi-Fi settings to route traffic through the recording proxy. Many mobile operating systems, such as iOS and Android, as well as proprietary and open source Android offshoots, such as Samsung’s Tizen, support making this change. Note that older versions of Android may lack requisite support. Some applications connect directly to the server regardless of the proxy settings of the operating system. In any of these cases, you need a tool that provides an alternative to proxy-based recording methods based on network capture or tunneling.

You can use the following simple test to check whether the application can be recorded using a proxy. First, configure the proxy settings on the device and record interactions with any website in a mobile browser. Then, try to record interactions in the native application. If the testing tool successfully records the browser generated traffic but does not record traffic generated by the native application, then testers can conclude that the native application is bypassing the proxy settings and that an alternative recording method is required.

Start testing with NeoLoad, the fastest, the most realistic, and the most automated continuous performance testing platform.

 

 

Recording Tests for Web Applications and Mobile Versions of Websites

 
 

Because web apps use the same web technologies as modern desktop browsers, testers can record the application or the mobile version of a website from a modern browser on a desktop, which presents an easier and faster alternative to recording from the device. Many web applications check the browser and platform used to access them. This enables the application when accessed from a mobile device, to redirect the content request to a mobile version optimized with less text and compressed images. To test such an app from the desktop, testers need to modify these requests such that they appear to the server to be coming from a mobile device. When testers neglect to compensate for this variable, they risk not testing the mobile version of the application; the server may redirect to a desktop version.

You have read 39% of the article