As you may already know, SPDY (pronounced “SPeeDY”) is a new networking protocol introduced by Google and designed to reduce the web latency associated with HTTP. With SPDY, web pages load up to 64 percent faster than HTTP alone, according to Google. It accomplishes this by adding a session layer between HTTP and SSL that supports concurrent, interleaved streams over a single TCP connection. The initial draft of HTTP/2.0, the future of the web, is based on SPDY, which is a welcome step forward considering that HTTP/1.0 was released in 1996. SPDY holds great potential for mobile devices, for which latency is more of an issue and the market is catching on. As evidence of this, Microsoft recently announced the next version of Internet Explorer will support the new protocol.

Today, SPDY can be deployed on Apache by installing a single module (mod-spdy, the Apache SPDY module). No changes to website content are required. SPDY is now enabled by default on many of the latest modern browsers, including Chrome, Firefox, Opera, and Android Browser. (You can check your own browser’s support by visiting isspdyenabled.com).

So, simply by deploying the SPDY module on your Apache server, you can provide a large percentage of your users with better response times! More importantly, the response times of a website are correlated with the site’s conversion rate and bounce rate—so SPDY can have a significant positive impact on your business!

With all the advantages of SPDY, you may be wondering if there is a price to pay on the server side. Moving from HTTP to SPDY adds encryption and compression overhead, which clearly requires more resources. It does, however, use fewer TCP connections. It wasn’t immediately clear to me if all these changes in aggregate would cause noticeable drawbacks on the server.

To get to the bottom of this question, I thought it would be interesting to get some real-world performance metrics on SPDY, in particular its effect on the server side. To obtain this information, I used the new SPDY support in NeoLoad to run load tests of SPDY and compare its performance against HTTP and HTTPS.

Test Configuration Details

My approach was to test a server with a fairly typical configuration—not one that is finely tuned for a specific purpose. This setup provides a basis for comparing HTTP, HTTPS, and SPDY. It’s important to note here that changes to the application or the server configuration will produce different results—perhaps vastly different.

Test setup:

  • Server:
    • OS: Linux CentOS 6.3
    • Hardware:
      • Intel Core i3 CPU 540 @ 3.07GHz
      • 4GB RAM
    • Apache
      • Version 2.2.15
      • worker MPM: <IfModuleworker.c>
        StartServers 5
        MaxClients 600
        MinSpareThreads 50
        MaxSpareThreads 150
        ThreadsPerChild 50
        MaxRequestsPerChild 0
        </IfModule>
      • SPDY module (mod-spdy-beta-0.9.3.3-386.x86_64). Default SPDY optimization flags were used (uncommented suggested values):
        • SpdyMaxThreadsPerProcess 30
        • SpdyMaxStreamsPerConnection 100
  • Application:
    • 1 web page: 1 HTML 9.5kB + 50 PNG icons (32×32).
    • The web page and all resources were served by the same server.
  • Load testing tool: NeoLoad 4.1.2
  • Tests:
    • Ramp-up from 1 to 1000 users for 20 minutes.
    • Scenario: 1 page call, wait for 5s, restart.
    • Chrome browser simulated, with 6 parallel connections to the server.
    • Emulated latency: 40ms in download, 45ms in upload.
    • Test HTTP, HTTPS, and SPDY under the same conditions.
    • Aggregation period of 30s for graphs. This smoothes variations to simplify comparison between several tests.

Results and Analysis

First I wanted to examine the hit rate (hits per second), average response time (duration), and number of errors as the number of concurrent users increased.

Here are the results for the HTTP test:

spdy-http

The plot illustrates the following:

  • At 3 minutes, the maximum number of page per second is reached.  This maximum rate (16 pages/s or 827 request/s) is reached at 120 virtual users (VU), and the rate decreases as more concurrent users are added.
  • At the same time (the 3-minute mark), response time starts to increase significantly.
  • The first errors (which investigation revealed to be timeouts) appear at 405 users.

The table below compares these metrics for HTTP, HTTPS, and SPDY:

HTTP HTTPS SPDY
Maximum pages/s 16.3 pages/safter 3 minutes at 120 users 15.9 pages/safter 3 minutes at 120 users 98 pages/safter 14 minutes at 777 users
Page response time at 100 users 1.1s 1.3s 1.1s
Page response time at 120 users 1.4 s 1.5s 1.1s
Page response time at 200 users 7.1s 7.8s 1.1s
Page response time at 777 users 70.2s 72s 2.7s
First error 405 Users 225 Users 884 Users
Service-level agreement violation (< 3s page load time) 133 Users 133 Users 794 Users

Based on a service-level agreement (SLA) of less than 3s for page load time, my test server handled almost six times as many users with SPDY than HTTP!

More Users With Fewer Workers

To handle incoming requests in parallel, Apache web servers use worker threads to process each request. Because each thread consumes resources and memory, Apache lets the system administrator limit thread use, which has an effect on how incoming requests are handled.

My HTTP and HTTPS results are typical of a scenario known as thread starvation.

This is confirmed by the following graph, which shows busy workers in orange, page rate in green, and user load in blue:

spdy-worker

The number of busy Apache workers (threads) hits its maximum at the same time the page rate reaches its maximum.

In contrast to HTTP/S, SPDY uses a single connection for all requests. The principal benefit of this is the decreased latency seen by the client, resulting in reduced response times for users. There is another great benefit on the server side: SPDY clients consume one worker instead of six. As a result, it takes much longer (that is, many more users) for all workers to become busy:

spdy-http-workers

SPDY enables the server to handle more users with the same number of workers.

Server CPU and Memory Consumption

So far, I showed how SPDY affects Apache worker usage with my test configuration. Next, I wanted to see how CPU usage and memory were affected by SPDY.

Here is a plot showing system idle time (low when the CPU is used intensively) for HTTP/HTTPS/SPDY:

spdy-linux-cpu-idle

No surprise here. As most HTTP/HTTPS requests are waiting to be served after the thread limit is reached, they don’t consume much CPU. For the SPDY tests, Apache is below the worker limit for much longer, resulting in a higher page rate and consequently more CPU used. The curve plateau is reached when all workers are busy (around 11 minutes).

To better understand the behavior before the worker limit is reached, I focused on the numbers at 2 minutes, 88 virtual users, and 12.5 pages/s. The CPU idle time measurements are an average of six values taken over 30 seconds:

CPU HTTP HTTPS SPDY
CPU Idle time at 2 minutes 93.33% 91.5% 92%

The values are comparable. SPDY consumes slightly less CPU than HTTPS and slightly more than HTTP for this load.

On the memory side, here’s the global overview:

spdy-memory

Again, it’s interesting to examine the memory used before the worker limit is reached. The numbers are an average of six measurements over 30 seconds.

System Memory 01:00 02:00 Difference = Consumed Memory
HTTP 3,357 MB 3,416 MB 59 MB
HTTPS 3,500 MB 3,579 MB 79 MB
SPDY 3,607 MB 3,631 MB 24 MB

The server consumed less memory when handling SPDY requests than when handling HTTP and HTTPS.

During this test, SPDY consumed just 41% of the memory consumed by HTTP, and just 30% of the memory consumed by HTTPS.

Conclusion

It’s no surprise that SPDY improves response times on the client side. That’s what it was designed to do. It turns out that SPDY also has advantages on the server side:

  • Compared to HTTPS, SPDY requests consume less resources (CPU and memory) on the server.
  • Compared to HTTP, SPDY requests consume less memory but a bit more CPU. This may be good, bad, or irrelevant depending on which resource (if either) is currently limiting your server.
  • Compared to HTTP/S, SPDY requires fewer Apache worker threads, which increases server capacity. As a result, the server may attract more SPDY traffic.

From these results, it is clear that Apache tuning performed for HTTP/S may not be appropriate after SPDY is enabled on the server. It’s important to take the time to re-evaluate server sizing and retune the server before you start handling SPDY requests from your users.

As I mentioned earlier, the test results I’ve shared here are specific to my server configuration and the web page I used for the tests. It’s important to understand how your server will perform under a realistic user load before you make SPDY available to the users on your site. I encourage you to test your own server and website using a load testing tool such as NeoLoad to see how SPDY will affect your users’ response times and your server’s performance.

, ,

14 Responses to “Evaluating the Performance of SPDY-enabled Web Servers”

  1. Ty says:

    SPDY uses less resources than HTTP(S)? RLY? Have a look at the graphs – the green SPDY line is worse than the blue and orange for majority of time!

    • Hervé Servy says:

      Hi Ty! It may look strange at first sight, the point is that the major part of the curve is while the workers limit is reached. Looking at that part is biased because SPDY uses more resources because it serves more requests (less Worker required). The comparison between HTTP/S and SPDY is done at the very beginning, when there’s free Workers for all.

      • PG says:

        I guess then what’s missing is a separate measurement of CPU consumption for HTTP and SPDY under the same load.

        Then the knowledge on the resource utilization doesn’t have to be derived – it will be measured.

        • Hervé Servy says:

          Hi, Not sure I fully understand your point. The SPDY engine we chose was strongly linked to Apache, and it was hard to separate the Apache CPU usage from the SPDY process one. Hervé

  2. dan says:

    Hi, is your SPDY test with or without TLS?

  3. Murali says:

    Hi, can you provide source of your web app which you used to test it.

    • Hervé Servy says:

      Hi, We used a sample page made of 50 to 100 small images to download. That was a delibarate choice and the start point of an interesting discussion about the typology of pages that could take the highest benefit of SPDY. Hervé

  4. [...] Evaluating the Performance of SPDY-enabled Web Servers [...]

  5. Excellent introduction to SPDY and valuable insight to performance, thank you for providing this!

    I have two questions:

    1) Shouldn’t the article title use the singular word server instead of servers?

    I ask because you tested a single Apache server and a single implementation of SPDY (mod-spdy).

    2) Did you rerun your tests to see if the results were consistent? Is rerunning tests and comparing results a feature of your NeoLoad product?

    I ask because, while I do believe your observations are correct, A sample size of one run may not be adequate to insure your conclusions will hold up.

    Thanks again.

    • Hervé Servy says:

      Thank you for your careful reading! You are correct: we made the test on a single server. About the methodology, we ran several tests on this server, with and without SPDY. We also considered the cache of the server and made our tests as reproductible as possible. Hervé

  6. [...] But I have to admit that this year, I begin to see the first projects using technologies that are drastically changing the principles of the network dialog between a browser and a server. I see our first SPDY projects. Does it change the server capacity? It just changes everything! Have a look at the measures I performed and documented in this article. [...]

  7. […] SPDY requires more power from the server to support compression and encryption but needs fewer TCP connections that partially offset the additional overhead, according to an analysis earlier this month by Herve Servy, writing on the blog site of Web-app testing-tool vendor Neotys USA. […]

  8. […] machen wollte. Ob es sicht rentiert, SPDY für den eigenen Webserver zu aktivieren, kann man hier […]

Leave a Reply