Since IO accounting was added to the linux kernel in 2.6.20, it’s been possible to examine IO per task.
Guillaume Chazarain’s iotop.py takes advantage of that to show disk IO in a format similar to the venerable top program.
It’s unlikely yum install iotop will work on your older linux distro yet though …
Prerequisites for running iotop.py are Python 2.4.3 with the python-ctypes module or Python 2.5+ (to preserve your existing python install, do make altinstall) and linux kernel 2.6.20+ with IO accounting enabled (TASKSTATS and TASK_IO_ACCOUNTING.)
If you’re using python 2.4.3, then install the python ctypes module like this as root:
yum install python-ctypes
Note that Redhat has backported IO accounting to 2.6.18-144.el5 kernel, which is available in 5.4 (also CentOS 5.4.)
If you need to build a newer kernel, here’s the required options:
make menuconfig General setup ---> [*] Export task/process statistics through netlink (EXPERIMENTAL) [*] Enable per-task delay accounting (EXPERIMENTAL) [*] Enable extended accounting over taskstats (EXPERIMENTAL) [*] Enable per-task storage I/O accounting (EXPERIMENTAL)
More info: the 4 kernel config settings needed are:
CONFIG_TASKSTATS=y CONFIG_TASK_DELAY_ACCT=y CONFIG_TASK_XACCT=y CONFIG_TASK_IO_ACCOUNTING=y
iotop.py accepts several command line options for filtering, including PID, user and process/thread-view. Once the program is loaded you can use the keyboard arrow keys to change which column gets sorted.
blktrace (btrace) is another tool for examining IO.
blktrace /dev/sda -a issue -a complete -w 3600 -o - | blkiomon -I 10 -h - btrace -s /dev/sda
Also, there is a linux parameter that acts as a watchdog for long-running IO and syslogs events that exceed the timeout:
echo 120 > /proc/sys/kernel/hung_task_timeout_secs
It produces syslog entries like this:
Feb 7 13:35:43 www kernel: INFO: task kjournald:1202 blocked for more than 120 seconds. Feb 7 13:35:43 www kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message
Guillaume Chazarain’s blog
Linux::Taskstats::Read Perl module
Tricks to diagnose processes blocked on strong I/O in linux
DTrace iotop
DTrace iotop samples
Running DTrace from Solaris Against Linux in Brandz
Fedora Daily Package: iotop – Display I/O Activity by Process
top-disk-users
Tracing block I/O operations on Linux hosts with blktrace
blktrace(8) – Linux man page
Red Hat backported I/O accounting to RHEL5
CentOS 5 iotop packages
Getting More out of top
Spying on Apache and PHP with Strace
