Rundfunk-Betriebstechnik GmbH (RBT) Department of Information Technology Public/Government (Nuremberg, Germany)

“NeoLoad continuously surprises us with options that inspire even experienced test professionals to define test runs more realistic and provide clearer analysis – yet more realistic and revealing than originally expected. Features such as Rendez-Vous Point, Global Validation or Result Filter make sure that our test runs are comprehensive and thereby back our effort to quick-start even complex scenarios with everything needed. Here, Neotys has often thought further than users.”

Claus-Georg Pleyer
Test engineer (server)

Profile

RBT is a joint venture between the public broadcasters ARD and ZDF and dedicated to provide its associates with test and project engineering services. These services include but are not limited to testing IT networks and systems from a user's point of view and with a vendor-neutral approach. With know-how gained over decades, RBT works according to individual requirements or with self-developed test concepts based on up-to-date standards. In recent years RBT has intensified security and performance tests of internet or intranet-based systems, e.g. websites.

Challenges

In 2007, the ZDF (Zweites Deutsche Fernsehen) commissioned RBT with a regular performance monitoring of its web-platform. The main task was to check the performance made available by ZDF prior to the further processing of the sites via ZDF’s provider – as this is the only way to create a reliable fundament for speedy downloads which users should experience on ZDF servers. In addition to the ZDF sites including search function, Multimedia library and the news („heute“) the tests encompass the internet presence of Phoenix, 3sat and tivi, all of which also being hosted by ZDF. Load tests started with a standard Layer 7 test system that complies with relevant norms. However, it quickly became clear that for realistic tests RBT needed a tool more specialized in web performance that is able to simulate browser behavior beyond normal standards.

In a market research RBT evaluated about 15 different web test tools. The requirement of fortnightly test implementation and evaluation focused attention on NeoLoad, that was already known from a diploma thesis as technologically substantiated and extraordinarily user-friendly. The very easy and uncomplicated contact with committed and qualified Neotys staff also played an important role. With the free evaluation version of NeoLoad pretests ran positively, for example with the following outcomes:

Prior to NeoLoad we used: PureLoad, JMeter, OpenSTA.

• Easy repeatability and adaptability of test runs
• Automated comparison of results from different test dates
• Flexible aggregation of data from different test runs in conclusive graphs
• Validation of load test parameters in network recordings
• Support for the „keep alive“ parameter for realistic TCP session rates (HTTP 1.0)

Application tested:

dynamic websites; further applications: (type-ahead) search, blogs, surveys, login portals, time management system, multimedia library (audio/video streams)

Protocols/technologies tested: HTTP/HTTPS, RTMP.

Solution

In the context of the ZDF project „Cyclical web load tests“ with NeoLoad the following steps were taken:

• Load definition in collaboration with the specialists responsible for ZDF’s server management, inter alia: 6 user profiles, think-time value, load curve, maximal number of simulated Virtual Users
• Recording HTTP-request URLs, automatically excluding any other than ZDF servers
• Defining variables for a realistic random think-time between page clicks
• Configuration of a relevant multi-level scenario for a rising number of users
• Installation of three load generators with NeoLoad agents, each of them with a connectivity of 100 Mbit/s
• Administration with the NeoLoad controller by RBT via different management connections

Finally, the following assessments were carried out with NeoLoad:

• Comparison of the average and maximum page-/request-response time at different test dates
• Confronting response time with time-to-first-byte to detect bottlenecks
• Grafic illustration of page and request rates as well as throughput
• Documentation of the slowest objects on the site
• Assigning error type and rate to page objects

Test Results and Added Value

Due to the high update-frequency of ZDF-hosted websites, NeoLoad-test runs with the same maximum load are very useful to discover tendencies in design and function that are detrimental to the performance. As a consequence, ZDF takes care not to overload its site with content and thereby keeps it attractive when load peaks – sometimes being surprisingly high even for the media sector – occur. With the automated comparison of different test runs in NeoLoad, any performance alteration can be easily displayed in percentage terms and set against each other as curves in one graph.

“NeoLoad continuously surprises us with options that inspire even experienced test professionals to define test runs more realistic and provide clearer analysis – yet more realistic and revealing than originally expected. Examples for this are features such as Rendez-vous-Point, Global Validation, Result Filter and many cross references, which back our effort to implement valid test runs with everything needed and to quick-start even complex scenarios. Here, Neotys has often thought further than the user. Additional NeoLoad options being especially useful for us:

• Unlimited combination and management of license-free agents for different platforms by using regular internet connections.
• Flexible handling of parameters extracted from server response.
• Well-arranged test runs to validate new click paths.
• Troubleshoot option to transfer test data directly to the NeoLoad support.”

Outlook

Meanwhile, RBT has successfully accomplished numerous other web tests with NeoLoad, e.g. performance assessments when developing internet blogs, web portals or applications with many dynamic parameters and validation rules. Also, when monitoring RTMP video streams, RBT utilizes NeoLoad for pilot measurements. At present, RBT relies on NeoLoad in order to advance CMS modernization projects of its associates.