Linux HTTP Load Testing with httperf

Linux logohttperf is an easy-to-use but powerful GPL2 command line (CLI) stress and load testing tool for linux.

Installing httperf

CentOS 6:

yum install httperf

CentOS 7:

wget http://ftp.tu-chemnitz.de/pub/linux/dag/redhat/el7/en/x86_64/rpmforge/RPMS/rpmforge-release-0.5.3-1.el7.rf.x86_64.rpm
rpm -Uvh rpmforge-release-0.5.3-1.el7.rf.x86_64.rpm
yum install httperf

Running httperf

  1. Always get permission from the site owner first before doing load testing
  2. It’s important to start by calibrating your tool first. Send one request and check the response:
$ httperf --server www.example.com --uri /index.php --print-request --print-reply -d10

If you see non-200 HTTP responses, like this 301 example response below, then you need to ensure you have the correct –uri parameter:

httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
httperf: maximum number of open descriptors = 1024
SH0:GET /index.php HTTP/1.1
SH0:User-Agent: httperf/0.9.0
SH0:Host: www.example.com
SH0:
SS0: header 83 content 0
RH0:HTTP/1.1 301 Moved Permanently

You can ignore the open files warning – it’s a bug in httperf. Just keep the load under 200 connections, or compile your own version from source.

Now we’re ready to do concurrent testing:

$ httperf --server www.example.com --uri /index.php --num-conn 20 --num-cal 10 --rate 2 --timeout 5
httperf --timeout=5 --client=0/1 --server=www.example.com --port=80 --uri=/blog --rate=2 --send-buffer=4096 --recv-buffer=16384 --num-conns=20 --num-calls=10
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 1

Total: connections 20 requests 200 replies 200 test-duration 10.675 s

Connection rate: 1.9 conn/s (533.8 ms/conn, <=4 concurrent connections)
Connection time [ms]: min 1175.2 avg 1266.2 max 1728.3 median 1179.5 stddev 179.3
Connection time [ms]: connect 63.4
Connection length [replies/conn]: 10.000

Request rate: 18.7 req/s (53.4 ms/req)
Request size [B]: 73.0

Reply rate [replies/s]: min 18.2 avg 19.1 max 20.0 stddev 1.3 (2 samples)
Reply time [ms]: response 120.3 transfer 0.0
Reply size [B]: header 238.0 content 0.0 footer 0.0 (total 238.0)
Reply status: 1xx=0 2xx=0 3xx=200 4xx=0 5xx=0

CPU time [s]: user 2.36 system 8.30 (user 22.1% system 77.7% total 99.9%)
Net I/O: 5.7 KB/s (0.0*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

Always check for non-zero error counts.

Going Pro

After you're comfortable using httperf, here's how to take it to the next level:

  1. use a dedicated physical machine separate from your subject under test to reduce intrusive latencies, and tail the server logs in separate terminal windows. Graph CPU and RAM consumption of the subject.
  2. build your own version of httperf with your preferred options. On CentOS 7:
    git clone https://github.com/httperf/httperf.git
    cd httperf
    # read Readme.md
    sudo yum install automake openssl-devel libtool
    libtoolize --force
    autoreconf -i
    automake
    ./configure
    make
    sudo make install
    
    read the links below and configure open files, port range and TCP timeout
  3. do runs 3 times at different times of the day and/or seasons
  4. again, always check for non-zero error counts
  5. add load and stress testing to your server and application deployment checklists. There's always some kind of surprise just waiting to be discovered. :)
Advanced Notes
  1. test tools are one of those things where you really need the source code to get what you want
  2. Runnning strace httperf ..., we see that httperf does polling with the select() system call. Hmm ...
    [..]
    select(4, [3], [], NULL, {0, 0})        = 0 (Timeout)
    select(4, [3], [], NULL, {0, 0})        = 0 (Timeout)
    select(4, [3], [], NULL, {0, 0})        = 0 (Timeout)
    [..]
    

akamaras.com: stress test your web server with httperf
SO: Changing the file descriptor size in httperf
easyengine.io: Increase "Open Files Limit"
brendangregg.com: The USE Method

Posted in API Programming, Cloud, Linux, Open Source, Tech | Leave a comment

The first ever photograph of light as both a particle and wave



Magnified image of electrons interacting with a standing photonic wave along a thin wire. The standing wave shows the wave nature of light, and the coloration measures the change in velocity as photons interact with electrons (particles)

The first ever photograph of light as both a particle and wave [2015]

Posted in Tech | Leave a comment

Basic JMeter Load Testing of Web Sites and Rest APIs

JMeter LogoThis is an intro to load testing with Apache JMeter, an Open Source load testing tool.

As a developer, QA or Operations engineer, it’s important to be familiar with what load testing tools can do, and to know how to configure a few actual tools.

I usually reach for a load testing tool in the following scenarios, which appear similar, but really are very different. You can divide stress and QA testing into 4 categories:

  1. I want to know what will happen when 100 requests are sent to a single or handful of endpoints in a brief time interval, usually one second (connection and configuration testing)
  2. I want to know what will happen under sustained load of 50 requests/second to a single or handful of endpoints, for typically 5 minutes (performance testing)
  3. I want to know that all of an application’s pages respond successfully, typically using 1 thread (application testing)
  4. I want to know how many synthetic users that application’s pages respond successfully under load, typically 20 – 100 threads. (application load testing)

Note that I don’t consider a simulated load to be meaningful for predicting human loads.

For example, on one intranet project, 70,000 users were happy with a phone book web app that only load tested to 20 simultaneous users. The test was useful in indicating that nothing was misconfigured, but not useful in predicting how many people could actually use it.

Load tests just tell me:

  1. if something is misconfigured or broken. If I get less than once response per second, or the server stops listening for a period of seconds, then we know there is a problem to investigate.
  2. numbers that I can compare to other runs over time.

Why JMeter?

JMeter is:

  • convenient (after the first time you learn it)
  • popular in the Java community, so worth being familiar with
  • Open Source (free)
  • extensible

The disadvantages of JMeter are that:

  • it has a complex UI
  • Java GC pauses can affect results on longer test runs. You can mitigate that by setting up your tests using the UI, then run the tests from the command line as recommended.

JMeter Installation

  1. check your Java version for 1.7 or 1.8 with java -version
  2. download and install JMeter
  3. read the JMeter Getting Started guide
  4. read the first 7 pages of the Basic Scripting with JMeter tutorial by Simon Knight
  5. setup an initial test using the JMeter UI. You must include a Response Assertion for a credible test. Then save to a jmx file as bin/Mysite.com.jmx (it’s an XML file with your settings.)

Automating

Make 4 copies of your jmx file with:

cp -p Mysite.com.jmx Mysite1.jmx
cp -p Mysite.com.jmx Mysite2.jmx
cp -p Mysite.com.jmx Mysite2.jmx
cp -p Mysite.com.jmx Mysite3.jmx

Edit each jmx file to customize the properties according to the 4 strategies I listed above:

LoopController.loops
ThreadGroup.num_threads

(If you want to invest some time, you can parameterize those as documented in the JMeter FAQ.)

Create the following bash script make_mysite1.sh so that you can run your test from the command line:

#!/bin/bash

# Program: test_mysite1.sh

rm -f mysite1.log

./jmeter -n -t Mysite1.jmx -l mysite1.samples.log -j mysite1.log

grep "Thread Group" mysite1.samples.log | grep -v [O]K

Running Tests

  1. Ensure you’re authorized before running any load test against a server you don’t own.
  2. bash test_mysite1.sh
  3. Analyze the response codes and timings. Test samples will be in mysite1.samples.log, and reports in mysite1.log.

When to Run

Every time a change is made to your environment, you should re-run the load tests. So include it as part of your release process checklist.

Distributed Testing

After you’re familiar with load testing using a single client with JMeter, you can learn about using multiple load test clients.

Bonus – “Soak Testing”

In the telco industry, historically new systems have undergone “soak testing.” This is operating test systems under a realistic load for one month or more to “provide a measure of a system’s stability over an extended period of time.”

JMeter Too Difficult?

There’s a couple options for people who want results without fussing with JMeter:

  1. Command Line Interface (CLI) – httperf
  2. Graphical User Interface (GUI) – Microsoft’s discontinued Web Application Stress Tool (WAST) aka “Homer” is an incredibly easy-to-use, distributed and powerful Windows graphical tool – “its ease of use means it actually gets used.” If you want to do load testing from Windows client machines, you can download it from here.

    WAST is so good that it has fans, which can’t be said for any other load test tool. It was replaced by Visual Studio Team System’s (VSTS) Test Manager.

JMeter: FAQ, Best Practices
SO: Load test with varying number of threads in JMeter

Posted in API Programming, GC Pauses, Java, Open Source, REST API Programming, Tech | Leave a comment

How to Build Linux rkt Container Manager on CentOS 6.7

Linux logoInstalling the rkt container manager on CentOS 6.x with yum will give you this error:

# yum -y install go rkt
# rkt run
rkt: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by rkt)

glibc is not something you can easily upgrade yourself, but you can build rkt from source. On CentOS 6.6 and 6.7, this works:

#!/bin/bash

sudo yum install -y go squashfs-tools libacl-devel glibc-static trousers-devel

wget https://github.com/coreos/rkt/archive/v1.15.0.tar.gz &&
tar zxvf - < v1.15.0.tar.gz &&
cd rkt-1.15.0 &&
./autogen.sh

echo "insert 'echo' at line 5667 to workaround old autogen bug"
vi configure

./configure --disable-sdjournal --with-stage1-flavors=fly --disable-tpm &&
echo 'readlink -f "$@"' > realpath &&
chmod +x realpath &&
export PATH=$PATH:. &&
make
# now test by downloading a Docker Ubuntu image and running it (requires about 256 MB RAM)
sudo ./build-rkt-1.15.0/target/bin/rkt run --interactive docker://ubuntu --insecure-options=image

CoreOS Issue #1063: build with old glibc so rkt runs on CentOS 6?

Posted in API Programming, Cloud, Open Source, Tech | Leave a comment

Linux rkt on CentOS7 is Just Too Easy

Linux logoThe rkt (pronounced “rocket”) container manager is just too easy to run on CentOS7!

Here’s me running a Docker Ubuntu 16.04.1 LTS image on CentOS7 (Dell 1950 III with 8 GB RAM on 100 Mbps Internet connection) for the first time in under a minute. The Ubuntu Docker image actually starts in 3 seconds once downloaded.

Download the rkt RPM then …

# rpm -Uvh rkt-1.18.0-1.x86_64.rpm

# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 
# uptime
06:16:04 up 141 days, 51 min, 1 user,load average: 0.18, 0.26, 0.22

# rkt run --interactive docker://ubuntu --insecure-options=image
Downloading sha256:6bbedd9b76a [================] 49.9 MB / 49.9 MB
Downloading sha256:fc19d60a83f [================]     824 B / 824 B
Downloading sha256:668604fde02 [================]     160 B / 160 B
Downloading sha256:de413bb911f [================]     444 B / 444 B
Downloading sha256:2879a7ad314 [================]     678 B / 678 B

root@rkt-2ee79be0-a70b-44be-90fd-1a1a54c17216:/# cat /etc/os-release 

NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial

root@rkt-2ee79be0-a70b-44be-90fd-1a1a54c17216:/# uptime
06:16:27 up 141 days, 52 min, 0 users,load average: 0.25, 0.27, 0.22

root@rkt-2ee79be0-a70b-44be-90fd-1a1a54c17216:/# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 06:16 ?        00:00:00 /usr/lib/systemd/systemd --default-standard-output=tty --log-target=null --show-status=0
root         3     1  0 06:16 ?        00:00:00 /usr/lib/systemd/systemd-journald
root         5     1  0 06:16 console  00:00:00 /bin/bash
root        13     5  0 06:16 console  00:00:00 ps -ef

root@rkt-2ee79be0-a70b-44be-90fd-1a1a54c17216:/# exit

# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 
# uptime
06:16:58 up 141 days, 52 min, 1 user,load average: 0.15, 0.24, 0.22

Is 0.0% memory usage light-weight enough? :)

# ps aux | egrep -e "[U]SER|[r]kt"

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      6711  0.4  0.0  41772  2244 pts/0    S+   07:22   0:01 stage1/rootfs/usr/lib/ld-linux-x86-64.so.2 stage1/rootfs/usr/bin/systemd-nspawn --boot --notify-ready=yes --register=true --link-journal=try-guest --quiet --uuid=40660e8b-09c0-46c2-893e-53de6d4068ff --machine=rkt-40660e8b-09c0-46c2-893e-53de6d4068ff --directory=stage1/rootfs --capability=CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FSETID,CAP_FOWNER,CAP_KILL,CAP_MKNOD,CAP_NET_RAW,CAP_NET_BIND_SERVICE,CAP_SETUID,CAP_SETGID,CAP_SETPCAP,CAP_SETFCAP,CAP_SYS_CHROOT -- --default-standard-output=tty --log-target=null --show-status=0

https://coreos.com/blog/getting-started-with-rkt-1.0.html
rkt prepare
Get Started with rkt Containers in Three Minutes
build with old glibc so rkt runs on CentOS 6?
https://github.com/JCMais/node-libcurl/issues/45: ./build-rkt-1.15.0/target/bin/rkt run –interactive docker://ubuntu –insecure-options=image

Posted in Cloud, Linux, Open Source, Tech | Leave a comment