When doing server upgrades with multiple servers, the ideal way is to:
1. take one instance out of the pool
2. drain connections on it
3. upgrade it
4. put it back into the pool
5. back to #1.
The various techniques can be categorized as:
1. application-level
2. load-balancer-level
3. OS-level
The most graceful method is to use an application-level feature, since the application knows what its worker status is.
For example, with httpd on CentOS or Redhat, either use the apachectl command, or add the graceful-stop option to /etc/init.d/httpd:
#!/bin/bash
set -e
echo "info: Draining connections ..."
apachectl graceful-stop
echo "info: You have 5 minutes to start and finish your upgrade."
sleep 300
apachectl start
echo "info: httpd restarted!"
exit 0
If we didn’t have an application-specific way to do that, we could use iptables:
#!/bin/bash
set -e
iptables -I INPUT -j DROP -p tcp --syn --destination-port 80
echo "info: Draining connections ..."
sleep 60
echo "info: You have 5 minutes to start and finish your upgrade"
sleep 300
iptables -D INPUT -j DROP -p tcp --syn --destination-port 80
echo "info: iptables allowing new incoming connections!"
exit 0
With HAProxy we can do this on the HAProxy host (do yum -y install socat first):
#!/bin/bash
set -e
echo "set server application-backend/www0 state drain" | socat unix-connect:/var/run/haproxy.sock stdio
echo "info: Draining connections ..."
sleep 60
echo "set server application-backend/www0 state maint" | socat unix-connect:/var/run/haproxy.sock stdio
echo "info: You have 5 minutes to start and finish your upgrade"
sleep 300
echo "set server application-backend/www0 state ready" | socat unix-connect:/var/run/haproxy.sock stdio
echo "info: haproxy allowing new incoming connections!"
exit 0
Sample HAProxy “show stat” output while www0 is draining (notice the “DRAIN” status):
[root@gw ~]# echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio
Name: HAProxy
Version: 1.5.10
[..]
[root@gw ~]# echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
http-in,FRONTEND,,,0,1,2000,6,998,19219,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,0,5,1,0,0,0,,0,1,6,,,0,0,0,0,,,,,,,,
https-in,FRONTEND,,,0,20,2000,12,294,2556,0,0,11,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,2,,,,0,0,1,11,0,0,,0,2,12,,,0,0,0,0,,,,,,,,
application-backend,www0,0,0,0,1,5000,3,583,1078,,0,,0,0,0,0,DRAIN,1,1,0,0,0,385,0,,1,4,1,,3,,2,0,,1,L7OK,301,0,0,2,1,0,0,0,0,,,,0,0,,,,,442,Moved Permanently,,0,0,0,1,
application-backend,www1,0,0,0,1,5000,4,709,18640,,0,,0,0,0,0,UP,1,1,0,0,0,755,0,,1,4,2,,4,,2,0,,1,L7OK,301,0,0,3,1,0,0,0,0,,,,0,0,,,,,141,Moved Permanently,,0,1,8,8,
application-backend,BACKEND,0,0,0,1,400,7,1292,19718,0,0,,0,0,0,0,UP,1,1,0,,0,755,0,,1,4,0,,7,,1,0,,1,,,,0,5,2,0,0,0,,,,,0,0,0,0,0,0,141,,,0,1,8,8,
For nginx:
#!/bin/bash
set -e
echo "info: Draining connections ..."
nginx -s quit
echo "info: You have 5 minutes to start and finish your upgrade."
sleep 300
nginx -s start
echo "info: nginx restarted!"
exit 0
If you’re using a configuration management system, like puppet or Chef, you can remove the service from your load balancer pool. This works well in practice with only 2 or 3 servers, though draining is usually not considered.
Note that when using the popular “reverse HAProxy” setup with application servers running HAProxy on localhost, and HAProxy forwarding localhost requests to the real servers (like httpd), then you want to stop or block the httpd services on the real server end. Otherwise you would have to make changes on multiple application servers.
In a future post, I’ll discuss zero-downtime deploys.
Drain connections on restart of NGINX process? (with iptables)
Tomcat’s Graceful Shutdown with Daemons and Shutdown Hooks
Get haproxy stats/informations via socat
Go net/http: add built-in graceful shutdown support to Server #4674
haproxy.tech-notes.net: HAProxy Socket Commands
I gave a talk at the 
In the 90’s and 00’s, programmers used to scorn front-end designers who called writing HTML “programming.” That changed to fear when CSS arrived and pixel-perfect results raised the bar.
I went to 
From the 
Current consumer ecommerce weather report for Manila in Dec., 2016 …
Wow. Who knew the little 
This is an intro to load testing with 
I went to the complimentary 
At 
Below is my patch that implements the CREATE [[NO] FORCE] VIEW options against MySQL/
I recently tuned an existing application that used dozens of views and hundreds of stored procedures using MySQL 5.6.
I recently did some work with 
