Sporting Index enhances website quality and performance with the help of NeoLoad
Additionally, Sporting Index provides the same services for betting from mobile devices through its mobile app, which is available on iPhone, iPad, Android, and Blackberry devices.
The Sporting Index web application has over 40,000 registered users and experiences anywhere between 2,000 and 3,000 concurrent users, many of whom have multiple tabs open in browsers to keep track of pricing on simultaneous live events (e.g. one tab for each of the five simultaneous English Premier League matches on a Saturday afternoon). The web and mobile apps have a lot of dynamic features, but perhaps the most important is the live pricing updates. “We’ve surveyed our customers and feel the 100ms response time is an important requirement, and the last thing we want to do is anger 20,000 football supporters!” said Andreas Papadopoulos, Head of Software Delivery at Sporting Index. To meet the demands of these customers, Sporting Index has to run performance tests on these dynamic elements under the most realistic conditions possible.
Sporting Index had been using HP LoadRunner to do load and performance testing for four years and they were experiencing problems. “LoadRunner was probably a great tool 10 years ago, but we found it difficult to manage setting up, running, and reusing tests,” said Papadopoulos. Tests were taking too long, scripting was cumbersome, and reporting was slow. Worst of all, when results of tests were cross-checked, it was determined that LoadRunner was producing inconsistent results making the tests unreliable. “When we submitted support tickets to HP, it took two weeks to get each ticket resolved.” Compounding the challenge, they had to script tests in open source tools like JMeter and SoapUI to test web services and other technologies like AJAX and SOAP because their version of LoadRunner was not structured to support them. “Switching between so many tools and having to manage so many scripts became a huge pain and was incredibly inefficient for us,” Papadopoulos added.
On the development side, there was a fear to change large critical components like thepricing system because performance in production could not be accurately predicted. The development organization was trying to use more Agile methodologies, but performance testing became a bottleneck and could not be accurately planned as a result.
Andreas Papadopoulos and the Sporting Index team had a history of innovation so they knew there had to be a better way. “Having been in the software delivery business for over 15 years, I knew the business always has to come first. If we were unable to support the innovation of our business with a proven high quality product, something needed to change.”
When looking for load and performance testing tools, Papadopoulos and his team found it difficult to determine which tool was the best fit for them based solely on their review of vendor websites. “They all say they will scale to a million virtual users, are easy to use, and have great support, so we felt the only way we could really determine how they could benefit our customers was to try them for ourselves.”
While trialing NeoLoad, Sporting Index also evaluated LoadRunner 11 as well as another tool. They experienced many problems with both of these other tools. LoadRunner could not complete certain tests, data buffering severely slowed down other tests, and the output of reports was not configurable. “Of course, the usability of LoadRunner was still a big concern, and in the end, it took too much effort and time to set up, test, and rewrite tests,” Papadopoulos continued, “The other tool we evaluated took half a day to set up a test, and we felt it was just too unintuitive.”
“With NeoLoad, tests were far easier to set up. Once it was installed, we were able to run a test within five minutes and get reports back immediately.”
NeoLoad required less server resources than both of the other tools so testing and reporting were much faster. In fact, they ran several tests over the course of a month and determined that they could use NeoLoad without having to invest in new hardware; NeoLoad was able to produce a larger load with the existing hardware. Testing was more efficient for the testers themselves because it was easy and intuitive to set up a library of tests for reuse. They were able to easily configure the reporting output from NeoLoad to get detailed measurement and immediately identify outliers and the 90% distribution.
Additionally, NeoLoad supported the testing of the technologies (web services, AJAX, SOAP) that they previously had to test with open source tools.
« After switching to NeoLoad, test outcomes have become more reliable – what we see on tests is what we see in production, which was not the case previously with LoadRunner,” said Papadopoulos. Overall, testing is much faster, not only because the application pulls up and runs tests faster, but also because the amount of time setting up tests, rerunning tests, and conducting maintenance is way down. “Load testing used to happen no more than once every few sprints because it took five days to run all the tests; now, load testing only takes one day and occurs every sprint.” This means the overall quality of the apps are better and teams have more confidence about how they will perform in production. “If there are ever any hotfixes that need to make it into production immediately, we can quickly load test it to guarantee the fix will not affect performance for our customers.” Using NeoLoad’s agentless monitors, Sporting Index now sees real time statistics on their servers getting CPU time, response times, system resource usage, etc. while correlating these data values with the application metrics.
This is all without having to set up separate server monitors, run tests, and analyze one by one the way they previously did with Perfmon. For their mobile app, Sporting Index had been using a downloadable java client, but they were developing their own device-specific apps and the contract for the java client was expiring soon. “We had a tight deadline to get our new mobile app out the door, but NeoLoad enabled us to shorten our testing cycle and release our new mobile app ahead of schedule.”
The Sporting Index testers are now less stressed about performance testing, and the organization has benefited from QA providing them with actionable insights.
Development feels more confident making changes to improve large, critical systems. “We were able to make two changes in one month to the pricing system that had only been previously changed twice in the previous six years!” Errors are caught quickly, an NeoLoad helps them get to root cause faster. “We know we can get information to the development and infrastructure teams that they can use to resolve issues faster each time we do a performance test,” said Papadopoulos.
“For the business, we have reduced our time to market for new features and enhancements for our customers and are able to accurately plan releases in advance.”
Additionally, NeoLoad allows them to perform capacity planning by pushing the limits of the apps to know when they will need additional infrastructure. This same capability allowed Sporting Index to move their pricing system from hardware to a movable infrastructure and know exactly how many VMs would be required, eliminating the risk of overspending on VMs they would not use. “Performance was the most difficult part of what we did in the whole software development lifecycle, but not anymore. With NeoLoad, performance has been demystified.”
In the future, Sporting Index plans to increase the number of virtual users with which they are testing and capitalize on shared licenses for NeoLoad to perform more tests in parallel. Additionally, they will look to the Neotys Cloud Platform for recreating geographic realism in their testing environment.
… NeoLoad lets us test in 1 day what previously took 5 days with [our legacy tool]. Now we can do performance testing every sprint without holding up the release.