One of Australia’s Largest Load Tests Ever Ensures 1 Million+ Viewers Stream Highly Anticipated Series Finale Without a Glitch
A large media company held exclusive rights to broadcast the final season of an enormously popular TV show and needed to ensure streaming quality during expected spikes in viewership, especially for the highly anticipated series finale.
The media company turned to AccessHQ, a leading Australian IT quality assurance and testing organization, to de-risk the potential for capacity overloads during peak traffic. AccessHQ partnered with Neotys to leverage the NeoLoad cloud platform to simulate the customer journey end to end for hundreds of thousands of concurrent users.
While such high-visibility programming offered high reward for the media company, it also posed high risk. For instance, another media company had recently experienced a failure during a major sporting event (which had been streaming to a smaller audience than the TV series), and the fallout included widespread bad press, lost subscribers and being forbidden to bid for broadcast rights for the event in the future. Six weeks before the series premiere, the media company turned to AccessHQ to de-risk potential problems during live and subsequent video-on-demand streaming.
AccessHQ Taps NeoLoad for Its Scalability, Speed and Insights
When AccessHQ was called in, they faced three challenging pressures: scale, speed and complexity. They had only six weeks to develop an effective strategy, execute one of the largest load tests ever run in Australia and provide actionable feedback to both the media company and its third-party vendors in time to optimize system performance before the season premiere.
“It was basically a best-endeavors de-risk for [the media company],” says Reuben Korngold, Consulting Director at AccessHQ. “With only six weeks, we didn’t have the time to design and script the way we’d like, but we’ll reduce the risk from a high level to a more confident level.
“Everybody said it wasn’t possible to test HD video streaming at this scale, but the [media company’s] COO said, ‘I want to make sure that if half a million people get on and stream, they’lI be able to watch HD.’”
In the end, the media company averaged 681K viewers for the season, with over 1.4 million people streaming the series finale. It was the largest single-day video player measurement (VPM) rating in Australian history.
“NeoLoad was the only product that we could identify with a specific video player component to drive the load testing. Some of our testers had been looking at how they could try to simulate (video load) themselves, and the fact that NeoLoad had this capability meant that we would already be getting video player metrics,” says Korngold. “The fact that we’d done another video streaming test previously, albeit smaller — just some movie trailers for one of the cinema chains in Australia — also gave us confidence in NeoLoad. We’d actually scripted and executed performance tests using the video capability before.”
“NeoLoad Are the Guys Who Can Help Us Make It Happen”
“We were pushing boundaries,” says Reuben. “We had that faith from our previous dealings with Neotys that these were the guys to work with who could help us make it happen. I’d just get on a call and say, This is what we’re thinking — can you support us? If you tried that with other performance tool vendors, you’d get a level-1 help desk person. . . . Instead, I’m picking up the phone and getting [Neotys VP] Laurent [Gaudy] who’s connecting me with the right people. We’d all get on a call and solve some very complicated issues in rapid time. The Neotys CTO even jumped in once to help out in the middle of the night! We had only a six-week project schedule, it could take you that long to get a hold of the right person at another tool vendor. You just wouldn’t even want to try.”
Identifying Third-Party Bottlenecks in a Complex End-to-End Flow
AccessHQ had to dig into the highly complex end-to-end flow of all the different components and systems, testing 550K requests per second. Streaming video involves a lot of moving parts. Reuben Korngold explains: “For instance, when you press PLAY, it goes and acquires a digital license for rights to play the video. Until those rights are issued, you can’t actually watch the video stream legally. There’s a process flow that went out to a third party in the US to issue this license in order for it to come back. When we did the load test it became apparent that this was a bottleneck. So there were a lot of third-party bottlenecks that got unblocked and identified as things to be optimized or to be scaled in preparation for that event.”
AccessHQ has a specialist digital experience team that uses eye-tracking technology (in fact, they pioneered use of this technology in Australia) to understand the user experience while the system was under peak load. “We interacted with the video platform as an end user while the load test was happening and recorded it to show what a baseline experience looked like and what system behavior actually looked like under load,” adds Korngold.
NeoLoad’s scalable cloud capacity was key. We used more than 1,900 load generators across AWS, Azure, Google Cloud.