What are the Challenges of Testing Streaming Applications?

Delay in streaming videos is risky

Today 62% of total internet traffic and 43% of the mobile traffic is streaming video, audio, and multimedia. And regardless of whether that rich content is distributed and consumed for entertainment or training and communications purposes, we all agree that since streaming has become one of the key channels to spread information around the world, we have become an increasingly demanding audience.

In the world of video applications, the audience decides what and when to watch, and if the content isn’t available on demand – and playable without delay – the audience quickly loses interest.

With streaming, positive user experience and sustained audience engagement are highly correlated with video playback performance.

Viewers will begin abandoning a video if their content doesn’t start streaming within 2 seconds – and for each additional second of delay, an additional 6% of the audience will leave.   After 5 seconds, over a quarter of your audience has moved on.

And once the video starts streaming, buffering even one time will reduce total viewing duration by 39%. Buffer a second time, and viewer engagement falls off by 50% – and your application and message have failed.

So when creating new multimedia applications, developers must now consider a new aspect of the user experience that must be tested during QA, and that is streaming performance under load.

What are the challenges related to formatting for streaming videos?

  • In a technical point of view, streaming is only a player requesting to “static resources.” So we all used to treat the performance of streaming like a caching problem. Is it still true? Adaptive Streaming has brought lots of value for the end-user experience: everything has been built to avoid Lags. One movie has several qualities. That means that the player is switching quality while streaming to avoid lags. The end user could start watching a movie in HD format and suddenly have low quality during few seconds. One movie is simply thousands of small multimedia files named chunks. Those chunks represent audio or video files. The player is requesting for the quality of chunk depending on the user settings: Screen of the player ( mobile, browser, etc.), quality of the network. That would mean that one media content is based on thousands of chunks.

The quality of a chunk is defined by the bitrate.

webinar 1

  • The second challenge is the Adaptive streaming Standards. We are glad to have MPEG-DASH….but since several years Microsoft delivered Smooth; then Apple has delivered HLS and last Adobe HDS. So even if MPEG-DASH is the standard, we still see lots of HLS, SMOOTH. RTMP was often used to deliver content, but it is starting to be replaced by one of the Adaptive Streaming format available on the market.

webinar 2

So depending on the computer, the device the format of the streaming will be different. That will mean that the multimedia service needs to handle the most popular adaptive streaming format to allow all users, devices to access the content.

How can my cache be efficient when 50 K users are watching different content with different quality and different protocols (hls, mpeg-dash)?

Many platforms are utilizing Content Delivery Network to speed and reduce the network time between the player and the chunk requested by the player.

Tips to properly qualify your multimedia application:

#1 Don’t have a record playback approach!

It will simply request the same chunk with the same quality on the same content. If you want to hit, prove that your cache works then you will succeed.

#2 Create representative user journey watching content

You can do it from:

    • Different devices ( different adaptive streaming technology and different type screen resolution)
    • Different network condition: you need to request a different kind of bitrate.

#3 Your “load testing” scripts need to emulate the exact behavior of the real player.

Users are free to fast-forward the content or go directly to a specific moment of the movie. The user can also choose the subtitle, the language, etc. This user behavior is also interesting to simulate to generate relevant load on the application.

On live content, users could be watching several live programs in parallel, so try to reproduce this behavior.


  • Correlate scenario reaching different content from different player
  • Correlate scenario emulating user interacting with player: fast-forward, subtitles, etc
  • Include the network constraints of the user to requests for low, medium, high-quality

Learn More

To learn more about the topic, watch the webinar we’ve recorded in partnership with Qumu and Cognizant. This is a live session with three testing experts: learn best practices, get answers to your questions and discover how to proceed to properly qualify your multimedia application.

Leave a Reply

Your email address will not be published. Required fields are marked *