This is the first in a two part series where I’ll be describing how to load test a web server. In this post I’ll go over some basic load testing concepts and get everything up and running. In the next post I’ll compare the performance of two popular web servers (Apache and lighttpd) running a popular content management system (Wordpress) and explore how performance can be improved.

First, some caveats. An artificial load test is a pretty poor substitute for real world traffic: nothing’s as good as the real thing. But if you’re wondering whether changing your apache configuration improved your server’s performance, whether APC sped up your PHP scripts, or whether upgrading to MySQL 5 decreased your site’s response time, a good load testing package is just what you need.

There are a whole host of HTTP load test tools available, and most of them have pretty similar feature sets. What I look for is something simple and easy to setup and run. I don’t do load testing very often, and I don’t want to be relearning some complicated application suite every few months. And, like I said, we’re not trying to duplicate the real world here. So, while features like log file replay and browser session capturing are nice, we don’t need them in this case.

There are a number of popular load testing tools that meet these criteria including http_load, ab, and flood. I’ll be using a simple command line utility developed by HP Labs called httperf for my load tests because it’s well documented and easy to use.

The Setup

Ideally you’ll want at least two dedicated machines for a proper load test. One machine will be the test server, the other the client. It’s important that there isn’t a whole lot of extraneous activity on either machine because it could affect the results by either using precious resources on the server, or interfering with the clients ability to accurately generate the desired workload.

It’s also important that the client and server have a low-latency network connection linking them. Dropped packets and other network activity will almost certainly skew your results. Again, the ideal setup is for the client and server machines to be connected to the same network appliance (switch, hub, router, etc.) over a high speed connection (at least 10Mbps, perhaps more).

That said, don’t fret if your setup is less than ideal. If you can get over the fact that your results aren’t perfect, you can still gain valuable insight into your site’s performance.

For my tests I’ll be using my Sun Ultra 20 workstation (1.8GHz AMD Model 144 Processor / 2.5GB RAM) as the server, and my MacBook Pro as the client (2.33GHz Intel Core 2 Duo Processor / 2GB RAM).

Installation

Installing httperf couldn’t be simpler. Grab the tarball, extract, configure, make. Simple enough. The binary ends up in the src directory, so if you want to run the program in place look there. Otherwise finish with sudo make install to install the application on your system.

$ wget -q ftp://ftp.hpl.hp.com/pub/httperf/httperf-0.9.0.tar.gz
$ tar -zxvf httperf-0.9.0.tar.gz > /dev/null
$ cd httperf-0.9.0
$ ./configure > /dev/null
$ make
make  all-recursive
Making all in src
Making all in gen
if cc -DHAVE_CONFIG_H -I. -I. -I../..    -I./.. -g -O2 -DHAVE_SSL -MT call_seq.o -MD -MP -MF ".deps/call_seq.Tpo" -c -o call_seq.o call_seq.c; \
then mv -f ".deps/call_seq.Tpo" ".deps/call_seq.Po"; else rm -f ".deps/call_seq.Tpo"; exit 1; fi
if cc -DHAVE_CONFIG_H -I. -I. -I../..    -I./.. -g -O2 -DHAVE_SSL -MT conn_rate.o -MD -MP -MF ".deps/conn_rate.Tpo" -c -o conn_rate.o conn_rate.c; \
...

Running Tests

As a demonstration, I’ve checked out a fresh copy of Wordpress from the Wordpress subversion repository. I’ll be load testing a default installation, served by Apache2 (mpm-prefork). In order to provide a sufficient test dataset, I’ve exported the posts from this blog into the new installation.

Web server software is not at its peak performance immediately after it’s launched because various caches the software utilizes have not been primed. Unless you are specifically testing the performance after a dry-start you should run a quick set of requests through the server prior to your load test.

How you go about running tests depends on what you want to accomplish. If you just want to know that your application can sustain a certain level of traffic then you can run a single test with a few parameters and make sure the response times are acceptable. I’m interested in seeing what the maximum sustainable traffic rate is for my new Wordpress installation, so I’ll need to run a series of tests, increasing the load each time, until the server is saturated.

Once the server is overloaded (we are sending more requests than the server is replying to) we can look at the response rate to determine the maximum number of requests per second the server can continuously serve. The test must also run long enough for the server to reach a steady state. Two or three minutes is usually sufficient.

My first test will send five requests per second for three minutes (a total of 900 requests).

$ ./httperf --server 192.168.0.107 --port 80 --uri /b/?p=28 \\
> --rate 5 --num-con 900 --num-call 1 --timeout 5
Maximum connect burst length: 1

Total: connections 900 requests 900 replies 900 test-duration 179.956 s

Connection rate: 5.0 conn/s (200.0 ms/conn, <=2 concurrent connections)
Connection time [ms]: min 153.4 avg 157.6 max 383.2 median 155.5 stddev 14.8
Connection time [ms]: connect 0.2
Connection length [replies/conn]: 1.000

Request rate: 5.0 req/s (200.0 ms/req)
Request size [B]: 73.0

Reply rate [replies/s]: min 5.0 avg 5.0 max 5.0 stddev 0.0 (35 samples)
Reply time [ms]: response 142.6 transfer 14.7
Reply size [B]: header 283.0 content 11594.0 footer 2.0 (total 11879.0)
Reply status: 1xx=0 2xx=900 3xx=0 4xx=0 5xx=0

CPU time [s]: user 39.29 system 128.22 (user 21.8% system 71.3% total 93.1%)
Net I/O: 58.4 KB/s (0.5*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

The results show that the server handled this traffic load readily. The reply rate was 5.0/sec with no measurable variation, and all 900 requests received a response within our timeout period. The average response time was 142.6ms, which is certainly acceptable for a web application (it’s below the rate of human perception).

Let’s bump the traffic rate up to 6 requests / second.

$ ./httperf --server 192.168.0.107 --port 80 --uri /b/?p=28 \\
> --rate 6 --num-con 1080 --num-call 1 --timeout 5
Maximum connect burst length: 1

Total: connections 1080 requests 1080 replies 1080 test-duration 179.988 s

Connection rate: 6.0 conn/s (166.7 ms/conn, <=3 concurrent connections)
Connection time [ms]: min 151.2 avg 161.0 max 1834.3 median 153.5 stddev 57.6
Connection time [ms]: connect 0.2
Connection length [replies/conn]: 1.000

Request rate: 6.0 req/s (166.7 ms/req)
Request size [B]: 73.0

Reply rate [replies/s]: min 5.8 avg 6.0 max 6.2 stddev 0.1 (36 samples)
Reply time [ms]: response 145.2 transfer 15.6
Reply size [B]: header 283.0 content 11594.0 footer 2.0 (total 11879.0)
Reply status: 1xx=0 2xx=1080 3xx=0 4xx=0 5xx=0

CPU time [s]: user 37.05 system 135.37 (user 20.6% system 75.2% total 95.8%)
Net I/O: 70.0 KB/s (0.6*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

We’re beginning to see signs of a struggle. The server managed the request load, but barely. The response rate slowed to a minimum of 5.8 per second during one period (samples are taken every 5 seconds), and the average response time was 160.0ms. If we increase the load again we’ll likely see the server begin to fail.

$ ./httperf --server 192.168.0.107 --port 80 --uri /b/?p=28 \\
> --rate 7 --num-con 210 --num-call 1 --timeout 5
Maximum connect burst length: 1

Total: connections 210 requests 210 replies 201 test-duration 32.100 s

Connection rate: 6.5 conn/s (152.9 ms/conn, <=27 concurrent connections)
Connection time [ms]: min 159.8 avg 2241.5 max 4229.0 median 2478.5 stddev 921.9
Connection time [ms]: connect 0.2
Connection length [replies/conn]: 1.000

Request rate: 6.5 req/s (152.9 ms/req)
Request size [B]: 73.0

Reply rate [replies/s]: min 5.6 avg 6.2 max 6.8 stddev 0.6 (6 samples)
Reply time [ms]: response 2116.4 transfer 132.5
Reply size [B]: header 283.0 content 11594.0 footer 2.0 (total 11879.0)
Reply status: 1xx=0 2xx=201 3xx=0 4xx=0 5xx=0

CPU time [s]: user 1.81 system 29.51 (user 5.6% system 91.9% total 97.5%)
Net I/O: 73.1 KB/s (0.6*10^6 bps)

Errors: total 9 client-timo 9 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

At seven requests per second we start to see timeouts as the server’s reply rate drops below the rate at which incoming requests are received. If this request rate was sustained for an extended period of time the server would begin to backlog requests and eventually stop accepting new connections. So we’ve found the maximum request rate our application can handle.

Summary

So our fresh Wordpress installation, running under Apache, can sustain a maximum of just over 6 requests per second. So, what does that tell us? By itself, not a whole lot. But it’s a good baseline measurement that we can use to compare our current setup to future configurations, after installing plugins or reconfiguring the web server.

In the second part of this series I’ll be doing just that. I’ll be load testing Wordpress again, under a variety of configurations (and two different web servers), to see if we can improve it’s performance.