Running Extreme Load Tests on Applications

By searching the internet, it is easy to find lots of materials on how to start load testing, how to run load tests on internal applications, etc. But when it comes to running extreme load tests, it’s suddenly more difficult to find recommendations.

But is there a difference between testing applications designed to handle more than 100K users and testing normal web applications having only 5k users?

That’s exactly the purpose of this article. There are of course differences linked to the complexity of the environment. The designed architecture will impact the way you are delivering your load testing project.

Of course the load testing methodology would be exactly the same:

  • First build a realistic load policy on the application. In fact, this is the most important part of the project because it will point out which user journey needs to be involved in performance testing, how many users per user journey, the realistic think time, the network constraints, the browsers, etc.
  • Design your scenario on your load testing solution. In extreme load tests, you need to make sure that your scenarios are stable with all your datasets. We don’t want to generate more overhead on the load testing solution.
  • Setup the service level agreement (SLA) for each important business action
  • Include user experience in your load testing. Make sure that the user experience is still good with the load.
  • Monitor your architecture. Remember running load tests without monitoring is like watching a horror movie on the radio… you will just hear people screaming without knowing why. Setting up the monitoring will allow you to be an actor during your load tests. Monitoring means have relevant counters on each layer of your architecture: cache, web server, application server, database, and of course your operating system.
    Some projects only look at the CPU. Your application has been designed with complex layers. Would your doctor only check your temperature? Of course not! He is making sure that each of the all-important KPIs is fine. So why only look at CPU?
  • Run the test
  • Analyze the results

Extreme load testing will likely involve architecture with:

  • A CDN having many nodes and a dedicated algorithm to guarantee the stability of the “caching” solution
  • Elastic architecture that will spin up servers depending on the load
  • Cloud providers

Lots of customers are sure that elastic architecture will guarantee the availability of their platform in any conditions. Of course it has been designed to guarantee availability, but one important point to understand with elastic architecture is: The system will spin up a server depending on specific threshold. A server doesn’t spin up in 1 second. In fact, it will take between 5-10 seconds (or more) depending on your solution. So if you don’t define the exact threshold then your users will experience bad response times until the new nodes become available.

The approach for this kind of architecture is to:

  • Disable the “elastic” solution to work only on your template server. You need to fine tune your template server because it would be your reference.
  • Once the template node tuned. Enable the elastic solution to find the right threshold to spin up server. The idea is to define the right moment to start a new node by taking into account the fact that your new node will be available 10 minutes later.
    Remember 10 minutes on a popular application could mean 100K new users.
    In fact, depending on your business, you could also handle your peak traffic by starting a large number of nodes during specific hours. Netflix is starting a large number of nodes on the evening to handle the massive audience of their members.

What about the CDN?

Running tests with more than 100K virtual users requires Cloud based load generators to avoid extra workload: your test will involve 400 or more load generators. Don’t spend time and money on setting up your load testing architecture – focus on your main objectives: TESTING. Make sure before running your test that you won’t saturate your load testing architecture. Your objective is to test your application, not your load testing solution. So take the time to properly size your load testing architecture in terms of bandwidth, server’s configuration, etc.

Running tests from the Cloud means that your will use maybe 20-30 geos for your load test. In real life, users are spread over thousands of locations. CDN providers are usually using lots of datacenters to deliver the content on the nearest location of the end-user. The nearest datacenter is defined with the help of the DNS resolution. Running tests on the CDN from the Cloud will then involve only the nodes near by the Cloud provider. In that case, you won’t be able to qualify properly the efficiency of the cache because you will only involve a few number of CDN nodes.

Another point is DNS resolution needs to be done frequently to avoid any saturation of the CDN. The CDN is frequently updating the IP address on the DNS resolution to avoid saturating their own servers.

Testing with a CDN during a load test means you won’t be able to retrieve realistic KPIs on the offload ratio of the CDN. The only way to qualify properly the CDN would be to involve hundreds of thousands of locations (this will then involve most of the CDN’s nodes).

Our recommendation would be to:

  • Qualify the architecture without the CDN to make sure that your application is able to handle most of the load.
  • Monitor the traffic of your website on beta production with RUM or BPM during a few weeks to make sure that the caching policy is efficient.

Cloud Provider

If your application is hosted on a Cloud provider, you will need to generate load from a different Cloud provider to avoid generating local traffic within the Cloud provider’s datacenter.

Conclusion

Basically there are no major differences from normal load testing, but there are few requirements to understand and include in your methodology. The complexity of the architecture will include more tasks to generate representative traffic on your application.

Leave a Reply

Your email address will not be published. Required fields are marked *