Friday, 21 March 2014

forkstat: a new tool to trace process activity

One of my on-going projects is to try to reduce system activity where possible to try to shave off wasted power consumption.   One of the more interesting problems is when very short lived processes are spawned off and die and traditional tools such as ps and top sometimes don't catch that activity.   Over last weekend I wrote the bulk of the forkstat tool to track down these processes.

Forkstat uses the kernel proc connector interface to detect process activity.  Proc connector allows forkstat to receive notifications of process events such as fork, exec, exit, core dump and changing the process name in the comm field over a socket connection.

By default, forkstat will just log fork, exec and exit events, but the -e option allows one to specify one or more of the fork, exec, exit, core dump or comm events.  When a fork event occurs, forkstat will log the PID and process name of the parent and child, allowing one to easily identify where processes are originating.    Where possible, forkstat attempts to track the life time of a process and will log the duration of a processes when it exits (note: this is not an estimate of the CPU used).

The -S option to forkstat will dump out a statistical summary of activity.  This is useful to identify the frequency of processes activity and hence identifying the top offenders.

Forkstat is now available in Ubuntu 14.04 Trusty Tahr LTS.  To install forkstat use:

 sudo apt-get install forkstat  

For more information on the tool and examples of the forkstat output, visit the forkstat quick start page.

9 comments:

  1. This looks awesome.

    ReplyDelete
  2. This looks awesome.

    (The commenting system, however, does not)

    ReplyDelete
  3. How is this different from strace?

    It looks like strace with a few parameters, maybe grepped does pretty much the same but it's already there and well-tested and stuff.

    ReplyDelete
    Replies
    1. strace works on a per process basis, whereas forkstat will catch all fork/exec/exit/etc activity in the system.

      Delete
    2. This is also much lower overhead. strace will drastically slow down your process.

      Delete
  4. DTrace or systap can do this and more.

    ReplyDelete
    Replies
    1. I guess one could say the same for useful tools such as vmstat, iostat etc..

      Delete
  5. Just curious - what are the security implications of that kernel connector facility? Doesn't it allow unprivileged users to snoop on syscalls of other users, perhaps in a way that monitoring /proc/* doesn't? (What of USER_NS etc.?)

    ReplyDelete
    Replies
    1. One needs to run the tool as root, so unprivileged users can't use this tool for snooping

      Delete