light-weight process stats with cpustat

A while ago I was working on identifying busy processes on small Ubuntu devices and required a tool that could look at per process stats (from /proc/$pid/stat) in a fast and efficient way with minimal overhead.   There are plenty of tools such as "top" and "atop" that can show per-process CPU utilisation stats, but most of these aren't useful on really slow low-power devices as they consume several tens of megacycles collecting and displaying the results.

I developed cpustat to be compact and efficient, as well as provide enough stats to allow me to easily identify CPU sucking processes.   To optimise the code, I used tools such as perf to identify code hotspots as well as valgrind's cachegrind to identify poorly designed cache inefficient data structures.

The majority of the savings were in the parsing of data from /proc - originally I used simple fscanf() style parsing; over several optimisation rounds I ended up with hand-crafted numeric and string scanning parsing that saved several hundred thousand cycles per iteration.

I also made some optimisations by tweaking the hash table sizes to match the input data more appropriately.  Also, by careful re-use of heap allocations, I was able to reduce malloc()/free() calls and save some heap management overhead.

Some very frequent string look-ups were replaced with hash lookups and frequently accessed data was duplicated rather than referenced indirectly to keep data local to reduce cache stalls and hence speed up data comparison lookup time.

The source has been statically checked by CoverityScan, cppcheck and also clang's scan-build to check for bugs introduced in the optimisation steps.

Example of cpustat

cpustat is now available in Ubuntu 15.10 Wily Werewolf.   Visit the cpustat project page for more details.