On demand Internet Streaming media has become extremely popular. About 89 million people in the United States are going to watch 1.2 billion online videos today, that number will double to 1.5 billion by 2016. Online video now accounts for 50 percent of all mobile traffic and up to 69 percent of traffic on certain networks.
Companies like Netflix, Hulu and HBO Go have seen an increase in consumers over the past year. ESPN and NBC have jumped on the online streaming video band wagon and have major deals to stream live sporting events. The notion of “cord-cutting” has become a phenomenon – which entails getting rid of your cable and substituting online video streaming subscription services.
But as more and more people demand online videos, performance starts to become a major concern. High video quality and fast load times are on the top of consumers’ preferences. It shouldn’t be surprising at all that when a company does experience a crash, as HBO Go did during the True Detective finale a couple of weeks ago, the social backlash can be deafening (our favorite from @hilmonstah: YOU HAD ONE JOB HBO GO).
What do you need to know about load testing video streaming performance? Here are the top 5 metrics to help you guarantee great video streaming performance for your consumers.
Top 5 Metrics to Measure
Here are the 5 metrics you should be tracking in order to make sure your video streaming is performing top notch.
1. Bit Rate
Bit rate might be the most important metric to measure, as it helps you understand the quality of the video that you users are experiencing. A higher average bit rate means a higher quality image (for a given screen resolution). The bit rate indicates how many bits of video (or information in general) can be transmitted over a specific period of time. Standard definition television is typically transmitted at a bit rate of 3.5 Mbps, while HDTV can be between 8 to 15 Mbps. Netflix Super HD streams at about 6 Mbps.
2. Buffer Fill
The time spent filling in the buffer when the video is first started. This is important because you want to know how long your users are waiting before a video actually starts playing. If they are staring at that rotating circle for too long, they may leave before even a frame flashes on their retinas.
3. Lag Length
After the buffer is initially filled, the video starts playing. At that point, if all goes well (meaning the download rate can keep up with the bit rate), the viewer will see their content in one smooth playback. However, this isn’t always the case – sometimes the buffer is drained, and without enough video to continue, playback halts. All the time spent waiting (including the initial buffer fill) is collectively called the lag length, and is an important metric for understanding the viewer’s experience. Ideally your lag length isn’t any greater than the initial buffer fill. If it is, you know that your consumer’s experience has been jarringly interrupted.
4. Play Length
The amount of data actually streamed out of your datacenter – the actual seconds, minutes, and hours of content that have been consumed by your users. This is an important metric to understand for capacity and infrastructure planning, and also helps you estimate peak data volumes and the overall demand for streamed data.
5. Lag Ratio
The lag ratio is easily calculated as waiting time over watching time. Because there is always an initial buffer period, you can’t ever quite get this ratio to zero. But you can get it to be pretty low. Think about it this way: the last time you watched a 2-hour movie, how many of those minutes were you staring at a “buffering…” message? How many would you tolerate before you turned it off? 3 minutes? That’s a 2.5% lag ratio.
Technologies that improve performance also complicate measurement
Clearly, consumers want video streaming services that are available (they can reach the videos they want), consistent (they don’t re-buffer), and high-quality (they look great) – probably in that order. All of this requires a high-functioning data center, high-throughput and resilient networks, and a consumer device (their TV or mobile phone) that’s beefy enough to handle the video along with anything else the user might be doing at the same time.
But you don’t always get all that, and so technologies like Adaptive Bitrate Streaming have been developed. Here, the server changes the quality of the video it is streaming when the viewer’s device can’t receive it or process it fast enough. So if you have ever noticed that a video’s quality goes down because your computer got busy with a virus scan, for example, that’s what’s happening. The bit rate is adapting – getting lowered – to make sure that the viewer’s video is not interrupted.
Adaptive bitrate streaming is built into technologies and products like MPEG DASH, Adobe Dynamic Streaming for Flash, Apple HTTP Live Streaming, and Microsoft Smooth Streaming. By doing this, they help address a big concern your users have – the sudden interruption of a video with a “loading” message. However, if you are trying to load test your system and make sure your consumers are getting the highest quality video and best possible experience as well, this technique actually complicates things.
Most load testing tools can’t accurately measure the metrics outlined above when adaptive bitrate streaming technologies are in place. So while your consumers aren’t experiencing playback interruptions, they may still be getting angry at fuzzified frames and muffled colors.
Load test your streaming video properly
The advanced technologies found in NeoLoad make it an excellent load testing platform for streaming video. With NeoLoad you can:
- Accurately assess key streaming metrics, even with adaptive bitrate streaming.
- See how a realistic viewership impacts your performance to plan for peak viewing – going so far as simulating geographically dispersed viewers all over the world operating TVs, PCs, and mobile devices.
- Design the video streaming infrastructure you need to keep consumers watching the videos they want, on the devices they want, at the quality they deserve.
Don’t make your consumers mad
Customer satisfaction has always been of concern for consumer-facing companies, but the large-scale, individual broadcasting that is made possible by social networking services – Twitter in particular – represents an entirely new force that companies must factor into account. Consider this: in the roughly 24 hours after HBO’s True Detective crash, there were 27,927 mentions of HBO GO on Twitter. That information comes from Marketwired and Adweek. But the numbers don’t capture the emotion – humor, mocking, and outright anger – that you can see in the #TrueDetective tweets themselves.
Think about the customer experience when they run into poor streaming video performance and how much a brand can be damaged by poor video streaming. According to a survey conducted by UMASS Amherst, Consumers give up on an online video if it doesn’t load in two seconds. A dissatisfied customer will tell between 9-15 people about their experience. Around 13% of dissatisfied customers tell more than 20 people. Just look at how angry Twitter got when Super Bowl ads started crashing.
HBO Go’s mishap led to some pretty angry consumers. You can learn from their mistakes. Load testing is important and not many tools can test streaming video performance well. Try NeoLoad. It will: #1 make sure your video streaming performance is top notch and #2 prevent you from making your customers angry.