You’re browsing YouTube and a video thumbnail catches your eye. Of course you need to watch a video of cats jumping through flaming hoops into kiddie pools full of jello! You click the link. You can feel your excitement building, when all of a sudden you’re presented with one of the most hated perturbances on the Internet: the loading symbol. Its perpetual spinning only increases your frustration, and you eventually give up to seek cat-related entertainment elsewhere.
This reaction has become commonplace. Attention spans are shrinking and more often than not, this is how customers will behave if they encounter poor performing streaming media on your website.
To combat this issue, performance testing is a must. However, the diversity of streaming media formats and performance metrics may leave QA teams wondering how to accurately test and measure the quality of the end user experience. Today, I’ll discuss some of the main differences between progressive downloads and adaptive bitrate streaming in terms of load and performance testing and explore the ways in which your organization can realistically test and ensure great streaming performance for your customers.
Progressive Download & Adaptive Bitrate Streaming
Progressive download is simply a video delivered by a regular HTTP web server rather than a streaming server. In most cases, video content delivered using this method is stored on the viewer’s hard drive as it’s received, and then playback occurs from the hard drive. Progressive download allows users to access content before the data transfer is complete—playback begins once a user is able to download only a small part of the video file.
This delivery method is advantageous for users with slower connections, as content can be downloaded before beginning playback. Oftentimes, this provides a smoother viewing experience where sufficient bandwidth is not available. However, with the prevalence of mobile devices and streaming over 3G network connections, this kind of content delivery became a bit of a nightmare. Adaptive bitrate streaming was introduced to address this issue and allow for the delivery of live video content.
Adaptive bitrate streaming is a technology specifically designed to deliver content to any type of user, whether the user is accessing it from a high speed connection or through a 3G connection on a mobile device. This delivery method actively detects a user’s bandwidth and CPU capacity and adjusts the quality of the video stream accordingly. The ever-increasing number of connected devices and variance in network connectivity created demand for a technology that would allow seamless media streaming under various end-user conditions.
Adaptive bitrate streaming is designed to minimize buffering and load times for videos delivered to any user on any device. This means you would receive a different data stream while watching a movie on your smartphone than you would on a monitor at your desk. Depending on your mobile network connectivity, you could begin playing a clip in 480p and then as the signal strengthened, switch to HD quality mid-clip. Though the quality may not remain consistent throughout the video, this method allows users to avoid, for the most part, pesky lags and frozen screens.
Load & Performance Testing Streaming Media
When it comes to load and performance testing streaming media, your team will need to adjust its approach based on your website’s video delivery method. Delivery via progressive download is as simple as placing a video file on your web server and pointing your player to the URL. Once a user clicks play, the player begins to download the file. Because you’re only dealing with a single file, performance testing doesn’t present any major challenges. Once you are able to generate the file itself, it’s just a matter of delivering it to the content delivery network (CDN) or caching layer.
Delivery via adaptive streaming is another matter entirely. One piece of content is made up of thousands of data chunks representing anywhere from two to 10 seconds of video and audio with varying quality levels. The format of the design changes depending on the streaming media technology you’re handling (MPEG/DASH, Adobe Dynamic Streaming for Flash, Microsoft Smooth Streaming, etc.)
In these cases, the interaction between the client and server is incredibly complex, making it more difficult to simulate in load tests. The first challenge testers will face is handling data capturing. This initial challenge isn’t necessarily related to performance. It deals with functionality—are you able to correctly transcode your file input format into adaptive streaming format? Is every chunk of data available?
Various load and performance testing challenges will also arise depending on the way you choose to deliver adaptive content to users:
Without a CDN
If your organization forgoes the use of a CDN, the video files will be hosted on internal servers. Your organization bears the responsibility of handling all traffic and spikes in usage on your own network and servers. This requires load testing. You need to make sure you have enough bandwidth to support any and all activity of users streaming the content.
Different kinds of streaming introduce an added layer of complexity. Your load testing tool must be able to handle a wide variety of streaming technologies like MPEG/DASH, Adobe Dynamic Streaming for Flash, Apple HTTP Live Streaming, and Microsoft Smooth Streaming. You should be able to test your backend with realistic Apple users, Android users, web TV users, browser users and more.
With a CDN
If your company utilizes a CDN, you won’t host streaming media files on your own infrastructure. Once you’ve generated the chunks of data, you’ll push them to a third-party content delivery network. This means your servers will need handle little to no load. There will be some load directed to your network if you encrypt your content. Digital rights management (DRM) calls will hit your network, and you’ll need to ensure your system can handle them. You’ll also need to validate the offload ratio of your CDN.
Some Use of a CDN
Some companies choose to host their own content while also using a CDN. The content delivery network will help to ensure seamless performance under heavy spikes in usage. If the CDN isn’t hosting the file, then it will need to request the file from your datacenter. So even with a CDN in place, there will be a small percentage of load directed onto your environment. It’s imperative to ensure that your backend can handle this load.
Measuring the End User Experience
There are different performance metrics to monitor for content delivered via progressive download or adaptive bitrate streaming. Things tend to be a bit simpler when dealing with progressive download. Metrics like bit rate, buffer fill, lag length, play length and lag ratio are directly related to the network conditions of the end user. Ultimately, if a user is experiencing poor network conditions with progressive downloads, the player will not be able to properly queue the content.
When dealing with adaptive streaming, there are a couple other KPIs to monitor. The entire goal of adaptive bitrate streaming is to eliminate lags. If your users are experiencing lags while streaming your content, there’s a significant architectural problem. You need to be able to identify and address these kinds of issues immediately.
To determine the quality level of the content you’re delivering to your users, you’ll want to measure the average bitrate. The bitrate indicates how many bits of video (or information in general) can be transmitted over a specific period of time.
You’ll also want to monitor startup time, or the time it takes from the moment a user presses play for the first frame to appear in the player. This is another important KPI as it gives you insight into how long it takes your player to detect the best playback format, queue the content, and deliver it to viewers. The percentage of frozen or black frames delivered is another KPI to track, as it will allow you to better determine the quality of content you’re delivering to users.
Executing Load and Performance Tests for Streaming Media
Due to the nature of the format, progressive downloads can be tested easily with a load and performance testing solution. However, adaptive streaming performance cannot be tested the same way. Unlike with progressive downloads, testers are not able to simply record the playback of adaptive streaming content. That wouldn’t make sense. If testers recorded an adaptive bitrate streaming scenario, they would end up with an unrepresentative list of files that wouldn’t align with real user conditions in terms of network and CPU usage. Replaying this kind of recorded traffic wouldn’t allow you to reproduce realistic use cases.
With a protocol-level solution or even a network solution, you wouldn’t be able to validate the user experience. You would need a third-party solution that could control the player at the UI level. This third-party tool would allow you to validate the user experience, while your load testing tool would allow for the generation of representative traffic on your platform.
You must account for the logic of your player in your load tests, otherwise, your tests are irrelevant. Let’s say you have 15% of users accessing the content via iPhone, 65% via web browsers and 20% via connected TVs. The video player on each of these devices will behave differently, and this will change the way load is applied to your platform. These users will also experience different network conditions that will affect playback. To reproduce realistic test cases for adaptive bitrate streaming, make sure your load and performance testing tool allows you to test from different devices under various network conditions.
Here, I’ve included a list of considerations to take into account when designing realistic test cases for streaming media:
- Don’t have a record playback approach!
- It will simply request the same chunk of data with the same quality on the same content. If you just want to hit your cache, then you will be able to do so with this method.
- On VOD platforms, users are not watching the same movie at the same time. Users will browse the catalog, select one of the available movies from the catalog and view it using different players.
- With live content, there could be several live programs running in parallel. So try to reproduce this in your tests.
- Create representative user journeys by monitoring content from:
- Different devices (different adaptive streaming technologies and different screen resolutions)
- Different network conditions: you need to request varying bitrates
- Users are free to fast-forward through the content or go directly to a specific moment in the video. The user can also choose subtitles, the language, etc. You should simulate this kind of behavior in order to generate relevant load on your application.
Your load testing scripts need to emulate the exact behavior of the real player.
Keep Your Customers Happy
Though realistically load and performance testing progressive downloads and adaptive bitrate streaming may not be a walk in the park, doing so will ensure that you’re accurately measuring and validating the end user experience.
Remember, it only takes two seconds before a majority of users will abandon an online video if it fails to load. Your organization cannot afford to lose these customers, especially over streaming media performance problems. High video quality and fast load times are at the top of consumers’ preferences. To learn how you and your testing team can treat it with the care and attention it deserves, check out our infographic: Performance Tester’s Guide to Streaming Media.