forkstat: a new tool to trace process activity

One of my on-going projects is to try to reduce system activity where possible to try to shave off wasted power consumption.   One of the more interesting problems is when very short lived processes are spawned off and die and traditional tools such as ps and top sometimes don't catch that activity.   Over last weekend I wrote the bulk of the forkstat tool to track down these processes.

Forkstat uses the kernel proc connector interface to detect process activity.  Proc connector allows forkstat to receive notifications of process events such as fork, exec, exit, core dump and changing the process name in the comm field over a socket connection.

By default, forkstat will just log fork, exec and exit events, but the -e option allows one to specify one or more of the fork, exec, exit, core dump or comm events.  When a fork event occurs, forkstat will log the PID and process name of the parent and child, allowing one to easily identify where processes are originating.    Where possible, forkstat attempts to track the life time of a process and will log the duration of a processes when it exits (note: this is not an estimate of the CPU used).

The -S option to forkstat will dump out a statistical summary of activity.  This is useful to identify the frequency of processes activity and hence identifying the top offenders.

Forkstat is now available in Ubuntu 14.04 Trusty Tahr LTS.  To install forkstat use:

 sudo apt-get install forkstat  

For more information on the tool and examples of the forkstat output, visit the forkstat quick start page.

atop - AT Computing's top

Last month I blogged about htop, a process monitoring tool. Today, while I was searching for a way of monitoring disk activity, I stumbled across atop (AT Computing's top). While it shares a lot of similarity with top, it is distinct enough to be useful in it's own rights. Because it is an interactive text based tool, it is driven by simple key presses, for example, pressing 'h' brings up a help page showing how to drive the tool.

Some useful key commands are as follows:

g - generic info (default)
m - memory details
d - disk details
n - network details
s - scheduling and thread-group info
v - various info (ppid, user/group, date/time, status, exitcode)
c - full command-line per process

..to name but a few.

One feature I especially like is atop highlights in red colour the processes that may be misbehaving, such as CPU, memory or disk I/O hoggers; very helpful when a rogue program is killing your machine or sucking away power or I/O.

To install atop, simply use:

sudo apt-get install atop

Below are some screen shots:

atop highlighting CPU hogging

atop highlighting I/O hogging

So, while I may always turn to ps,top or vmstat as my first port of call when I want to check my systems activity, atop is another very useful utility in my toolkit that comes in useful in seeking and resolving misbehaving systems. Check it out and let me know what you think!

htop - an alternative to top?

At times it is useful to see which process is burning up all those CPU cycles. Normally I use the top command, which does the trick. I've recently stumbled upon the htop command, which is a little like top, but has some neat interactive features which makes it a little more powerful than top in my opinion.

To install it, simply do:

sudo apt-get install htop

htop allows one to see CPU usage per processor, and allows one to scroll up and down the process list. If you've not used it yet, I recommend installing it and giving it a try.