LatencyTop - Measuring Latency in Linux

I'm spending some of my free time fiddling around some of the features with the 2.6.31 kernel in Ubuntu Karmic. While on my quest to check out system latencies I installed LatencyTop and ran a few quick tests.

To install latencytop, use:

sudo apt-get install latencytop

And to run use:

sudo latencytop

LatencyTop typically comes in useful when problems where the kernel does not respond quickly enough to real time requirements, for example, skipping audio, or when a server is fully loaded but one still as plenty of free CPU cycles left over. I'm very interested in the latter case as my Quad Core Xeon server is seeing a CPU load of 85% on all 4 cores when building a kernel and I want to figure out why I'm not able to fully max the system out even when the build is just going to a memory based filesystem (reducing the HDD I/O bottleneck to zero).

Basically, LatencyTop allows one to look at cases where applications want to run but are blocked because a resource is not available so the kernel blocks that process. The beauty of LatencyTop is that it shows what is happening in the kernel and with user space processes, allowing one to get an idea of what's wrong without too much difficulty. Once the reason for the latency is identified then it's a case of fixing code, which is where the fun begins :-)

For more information on LatencyTop, checkout the project home page and the announcement page too.

Below is an example of the tool running on my Lenovo laptop:


..as one can see process hddtemp has suffered from a 115.4 millisecond latency doing a SCSI disk ioctl(). While this is quite a large delay, it's not a mission critical application - it's just data being gathered for my Hardware Sensors applet.

Thanks to the Intel Open Source Technology Center for this neat tool. Keep them coming Intel!