Snooping on I/O using iosnoop

A while ago I blogged about Brendan Gregg's excellent book for tracking down performance issues titled "Systems Performance, Enterprise and the Cloud".   Brendan has also produced a useful I/O diagnostic bash script iosnoop that uses ftrace to gather block device I/O events in real time.

The following example snoops on I/O for 1 second:

$ sudo iosnoop 1
Tracing block I/O for 1 seconds (buffered)...
COMM             PID    TYPE DEV      BLOCK        BYTES     LATms
kworker/u16:2    650    W    8,0      441077032    28672      1.46
kworker/u16:2    650    W    8,0      441077024    4096       1.45
kworker/u16:2    650    W    8,0      364810624    462848     1.35
kworker/u16:2    650    W    8,0      364810240    69632      1.34

And the next example snoops and shows start and end time stamps:

$ sudo iosnoop -ts  
Tracing block I/O. Ctrl-C to end.  
STARTs        ENDs          COMM           PID  TYPE DEV   BLOCK    BYTES   LATms  
35253.062020  35253.063148  jbd2/sda1-211  211  WS   8,0   29737200   53248   1.13  
35253.063210  35253.063261  jbd2/sda1-211  211  FWS  8,0   18446744073709551615 0     0.05  
35253.063282  35253.063616  <idle>         0    WS   8,0   29737304   4096    0.33  
35253.063650  35253.063688  gawk           551  FWS  8,0   18446744073709551615 0     0.04  
35253.766711  35253.767158  kworker/u16:0  305  W    8,0   433580264  4096    0.45  
35253.766778  35253.767258  kworker/0:1H   321  FWS  8,0   18446744073709551615 0     0.48  
35253.767289  35253.767635  <idle>         0    WS   8,0   273358464  4096    0.35  
35253.767309  35253.767654  <idle>         0    W    8,0   118371312  4096    0.35  
35253.767648  35253.767741  <idle>         0    FWS  8,0   18446744073709551615 0     0.09  
^C  
Ending tracing...  

One needs to run the tool as root as it uses ftrace. There are a selection of filtering options, such as showing I/O from a specific device, I/O issues of a specific I/O type, selecting I/O on a specific PID or a specific name. iosnoop also can display the I/O completion times, start times and Queue insertion I/O start time. On Ubuntu, iosnoop can be installed using:

sudo apt-get install perf-tools-unstable

A useful I/O analysis tool indeed. For more details, install the tool and read the iosnoop man page.