Linux Graceful Service Shutdown Techniques

Linux logoWhen doing server upgrades with multiple servers, the ideal way is to:

1. take one instance out of the pool
2. drain connections on it
3. upgrade it
4. put it back into the pool
5. back to #1.

The various techniques can be categorized as:

1. application-level
2. load-balancer-level
3. OS-level

The most graceful method is to use an application-level feature, since the application knows what its worker status is.

For example, with httpd on CentOS or Redhat, either use the apachectl command, or add the graceful-stop option to /etc/init.d/httpd:

#!/bin/bash

set -e

echo "info: Draining connections ..."
apachectl graceful-stop
echo "info: You have 5 minutes to start and finish your upgrade."
sleep 300
apachectl start
echo "info: httpd restarted!"
exit 0

If we didn’t have an application-specific way to do that, we could use iptables:

#!/bin/bash

set -e

iptables -I INPUT -j DROP -p tcp --syn --destination-port 80
echo "info: Draining connections ..."
sleep 60
echo "info: You have 5 minutes to start and finish your upgrade"
sleep 300
iptables -D INPUT -j DROP -p tcp --syn --destination-port 80
echo "info: iptables allowing new incoming connections!"
exit 0

With HAProxy we can do this on the HAProxy host (do yum -y install socat first):

#!/bin/bash

set -e

echo "set server application-backend/www0 state drain" | socat unix-connect:/var/run/haproxy.sock stdio
echo "info: Draining connections ..."
sleep 60
echo "set server application-backend/www0 state maint" | socat unix-connect:/var/run/haproxy.sock stdio
echo "info: You have 5 minutes to start and finish your upgrade"
sleep 300
echo "set server application-backend/www0 state ready" | socat unix-connect:/var/run/haproxy.sock stdio
echo "info: haproxy allowing new incoming connections!"
exit 0

Sample HAProxy “show stat” output while www0 is draining (notice the “DRAIN” status):

[root@gw ~]# echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio
Name: HAProxy
Version: 1.5.10
[..]

[root@gw ~]# echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
http-in,FRONTEND,,,0,1,2000,6,998,19219,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,0,5,1,0,0,0,,0,1,6,,,0,0,0,0,,,,,,,,
https-in,FRONTEND,,,0,20,2000,12,294,2556,0,0,11,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,2,,,,0,0,1,11,0,0,,0,2,12,,,0,0,0,0,,,,,,,,
application-backend,www0,0,0,0,1,5000,3,583,1078,,0,,0,0,0,0,DRAIN,1,1,0,0,0,385,0,,1,4,1,,3,,2,0,,1,L7OK,301,0,0,2,1,0,0,0,0,,,,0,0,,,,,442,Moved Permanently,,0,0,0,1,
application-backend,www1,0,0,0,1,5000,4,709,18640,,0,,0,0,0,0,UP,1,1,0,0,0,755,0,,1,4,2,,4,,2,0,,1,L7OK,301,0,0,3,1,0,0,0,0,,,,0,0,,,,,141,Moved Permanently,,0,1,8,8,
application-backend,BACKEND,0,0,0,1,400,7,1292,19718,0,0,,0,0,0,0,UP,1,1,0,,0,755,0,,1,4,0,,7,,1,0,,1,,,,0,5,2,0,0,0,,,,,0,0,0,0,0,0,141,,,0,1,8,8,

For nginx:

#!/bin/bash

set -e

echo "info: Draining connections ..."
nginx -s quit
echo "info: You have 5 minutes to start and finish your upgrade."
sleep 300
nginx -s start
echo "info: nginx restarted!"
exit 0

If you’re using a configuration management system, like puppet or Chef, you can remove the service from your load balancer pool. This works well in practice with only 2 or 3 servers, though draining is usually not considered.

Note that when using the popular “reverse HAProxy” setup with application servers running HAProxy on localhost, and HAProxy forwarding localhost requests to the real servers (like httpd), then you want to stop or block the httpd services on the real server end. Otherwise you would have to make changes on multiple application servers.

In a future post, I’ll discuss zero-downtime deploys.

Drain connections on restart of NGINX process? (with iptables)
Tomcat’s Graceful Shutdown with Daemons and Shutdown Hooks
Get haproxy stats/informations via socat
Go net/http: add built-in graceful shutdown support to Server #4674
haproxy.tech-notes.net: HAProxy Socket Commands

This entry was posted in API Programming, Business, Cloud, Java, Linux, Microservices, Open Source, Tech. Bookmark the permalink.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.