One of Australia’s Largest Load Tests Ever Ensures 1 Million+ Viewers Stream Highly Anticipated Series Finale Without a Glitch

Challenge

A large media company held exclusive rights to broadcast the final season of an enormously popular TV show and needed to ensure streaming quality during expected spikes in viewership, especially for the highly anticipated series finale.

Solution

The media company turned to AccessHQ, a leading Australian IT quality assurance and testing organization, to de-risk the potential for capacity overloads during peak traffic. AccessHQ partnered with Neotys to leverage the NeoLoad cloud platform to simulate the customer journey end to end for hundreds of thousands of concurrent users.

Background

The Australian media company had exclusive rights to show the final season of one of the most popular TV shows of all time, live and on demand, over its digital IP-based video streaming platform. Viewership was expected to be on the order of 2–3X the volume of traffic of anything the company had seen before. The media company’s technology stack includes not only the core video delivery platform but also many specialized third-party media technology vendors whose services are tightly integrated into the end-to-end customer journey, from sign-up to navigation to streaming.
While such high-visibility programming offered high reward for the media company, it also posed high risk. For instance, another media company had recently experienced a failure during a major sporting event (which had been streaming to a smaller audience than the TV series), and the fallout included widespread bad press, lost subscribers and being forbidden to bid for broadcast rights for the event in the future. Six weeks before the series premiere, the media company turned to AccessHQ to de-risk potential problems during live and subsequent video-on-demand streaming.

AccessHQ Taps NeoLoad for Its Scalability, Speed and Insights

When AccessHQ was called in, they faced three challenging pressures: scale, speed and complexity. They had only six weeks to develop an effective strategy, execute one of the largest load tests ever run in Australia and provide actionable feedback to both the media company and its third-party vendors in time to optimize system performance before the season premiere.
“It was basically a best-endeavors de-risk for [the media company],” says Reuben Korngold, Consulting Director at AccessHQ. “With only six weeks, we didn’t have the time to design and script the way we’d like, but we’ll reduce the risk from a high level to a more confident level.
“Everybody said it wasn’t possible to test HD video streaming at this scale, but the [media company’s] COO said, ‘I want to make sure that if half a million people get on and stream, they’lI be able to watch HD.’”

“NeoLoad’s scalable cloud capacity was key. We used more than 1,900 load generators across AWS, Azure, Google Cloud.”
 
— Reuben Korngold, Consulting Director, AccessHQ
 
There were concerns that the media company’s existing tool’s load generation would not be able to handle the volume, and whether other performance testing solutions would be able to provide the level of scalable cloud capacity. This was a crucial factor, as AccessHQ knew it had to push the system to its limits, simulating up to triple the number of users streaming HD video beyond what was known to be the media company’s previous peak. They could not tether themselves to any single cloud provider — given the size and scope of the test, they needed a vendor-agnostic, multi-cloud testing solution that would allow them to run load tests across multiple data centers from a combination of Amazon AWS, Google Cloud and Microsoft Azure. Says Reuben Korngold, “NeoLoad’s scalable cloud capacity was key. We used more than 1,900 load generators across AWS, Azure, Google Cloud.” NeoLoad also natively interacts with container orchestrators such as Kubernetes, OpenShift, Microsoft AKS, Amazon EKS and Google GKE to provision and use load testing infrastructure automatically, on-demand.
In the end, the media company averaged 681K viewers for the season, with over 1.4 million people streaming the series finale. It was the largest single-day video player measurement (VPM) rating in Australian history.
“NeoLoad was the only product that we could identify with a specific video player component to drive the load testing. Some of our testers had been looking at how they could try to simulate (video load) themselves, and the fact that NeoLoad had this capability meant that we would already be getting video player metrics,” says Korngold. “The fact that we’d done another video streaming test previously, albeit smaller — just some movie trailers for one of the cinema chains in Australia — also gave us confidence in NeoLoad. We’d actually scripted and executed performance tests using the video capability before.”

“NeoLoad Are the Guys Who Can Help Us Make It Happen”

“We were pushing boundaries,” says Reuben. “We had that faith from our previous dealings with Neotys that these were the guys to work with who could help us make it happen. I’d just get on a call and say, This is what we’re thinking — can you support us? If you tried that with other performance tool vendors, you’d get a level-1 help desk person. . . . Instead, I’m picking up the phone and getting [Neotys VP] Laurent [Gaudy] who’s connecting me with the right people. We’d all get on a call and solve some very complicated issues in rapid time. The Neotys CTO even jumped in once to help out in the middle of the night! We had only a six-week project schedule, it could take you that long to get a hold of the right person at another tool vendor. You just wouldn’t even want to try.”

Identifying Third-Party Bottlenecks in a Complex End-to-End Flow

AccessHQ ran the load test four or five times. “If we hadn’t done that testing, there’s no way [the media company] would have succeeded. And as is always the case, we found that it wasn’t just their core delivery network that needed to be optimized, but also many third-party services.”
AccessHQ had to dig into the highly complex end-to-end flow of all the different components and systems, testing 550K requests per second. Streaming video involves a lot of moving parts. Reuben Korngold explains: “For instance, when you press PLAY, it goes and acquires a digital license for rights to play the video. Until those rights are issued, you can’t actually watch the video stream legally. There’s a process flow that went out to a third party in the US to issue this license in order for it to come back. When we did the load test it became apparent that this was a bottleneck. So there were a lot of third-party bottlenecks that got unblocked and identified as things to be optimized or to be scaled in preparation for that event.”
AccessHQ has a specialist digital experience team that uses eye-tracking technology (in fact, they pioneered use of this technology in Australia) to understand the user experience while the system was under peak load. “We interacted with the video platform as an end user while the load test was happening and recorded it to show what a baseline experience looked like and what system behavior actually looked like under load,” adds Korngold.
Company
Large Pay Television Company
Industry
Broadcasting, Internet Service Providers, Website Hosting & Internet-related Services
Employees
2000
Location
Australia

NeoLoad’s scalable cloud capacity was key. We used more than 1,900 load generators across AWS, Azure, Google Cloud.

Start testing with NeoLoad, the fastest, the most realistic, and the most automated continuous performance testing platform.