Slipping in a few more stress-ng features before 2015

During idle moments in the final few weeks of 2014 I have been adding some more stressors and features to stress-ng as well as tidying up the code and fixing some small bugs that have crept in during the last development spin.   Stress-ng aims to stress a machine with various simple torture tests to trip overheating and kernel race conditions.

The mmap stressor now has the '--mmap-file' to use synchronous file backed memory mapping instead of the default anonymous mapping, and the '--mmap-async' option enables asynchronous file mapping if desired.

For socket stressing, the '--sock-domain unix' option now allows AF_UNIX (aka AF_LOCAL) sockets to be used. This compliments the existing AF_INET and AF_INET6 IPv4 and IPv6 protocols available with this stress test.

The CPU stressor now includes mixed integer and floating point stressors, covering 32 and 64 bit integer mixes with the float, double and long double floating point types. The generated object code contains a nice mix of operations which should exercise various functional units in the CPU.  For example, when running on a hyper-threaded CPU one notices a performance hit because these cpu stressor methods heavily contend on the CPU math functional blocks.

File based locking has been extended with the new lockf stressor, this stresses multiple locking and unlocking on portions of a small file and the default blocking mode can be turned into a CPU consuming rapid polling retry with the '--lockf-nonblock' option.

The dup(2) system call is also now stressed with the new dup stressor. This just repeatedly dup's a file opened on /dev/zero until all the free file slots are full, and then closes these. It is very much like the open stressors.

The fcntl(2) F_SETLEASE command is stress tested with the new lease stressor. This has a parent process that rapidly locks and unlocks a file based lease and 1 or more child processes try to open this file and cause lease breaking signal notifications to the parent.

For x86 CPUs, the cache stressor includes two new cache specific options. The '--cache-fence' option forces write serialization on each store operation, while the '--cache-flush' option forces flush cache on each store operation. The code has been specifically written to not incur any overhead if these options are not enabled or are not available on non-x86 CPUs.

This release also includes the stress-ng project mascot too; a humble CPU being tortured and stressed by a rather angry flame.

For more information, visit the stress-ng project page, or check out the stress-ng manual.

Controlling data flow using sluice

Earlier this year I was instrumenting wifi power consumption and needed a way to produce and also consume data at a specific rate to pipe over netcat for my measurements.  I had some older crufty code around to do this but it needed some polishing up so I eventually got around to turning this into a more usable tool called "sluice".  (A sluice gate controls flow of water,  the sluice tool controls rate of data though a pipe).

The sluice package is currently available in PPA:colin-king/white and is built for Ubuntu Trusty, Utopic and Vivid, but there the git repository and tarballs are available too if one wants to build this from source.

The following starts a netcat 'server' to send largefile at a rate of 1MB a second using 1K buffers and reports the transfer stats to stderr with the -v verbose mode enabled:

cat largefile | sluice -r 1MB -i 1K -v | nc -l 127.0.0.1 1234 

Sluice also allows one to adjust the read/write buffer sizes dynamically to try to avoid buffer underflow or overflows while trying to match the specified transfer rate.

Sluice can be used as a data sink on the receiving side and also has an option to throw the data away if one just wants to stream data and test the data link rates, e.g., get data from somehost.com on port 1234 and throw it away at 2MB a second:

nc somehost.com 1234 | sluice -d -r 2MB -i 8K

And finally, sluice as a "tee" mode, where data is copied to stdout and to a specified output file using the -t option.

For more details, refer to the sluice project page.

It is approaching the Christmas Holiday season, so it's that time again to write some slightly obfuscated C in a seasonal way.  This year I thought I would try some coloured ASCII art for the output for a little variety.

#define r(s)  s[e%(sizeof s-1)]
#include  /*        */
#define S "%s"/*        Have */
#define u printf(/*        a */
#define c )J/*         Merry */
#define W "H"/*    Christmas */
#define e rand()/*       and */
#define U(i) v[i]/*        a */
#define C(q) q[]=/*    Happy */
#define J ;;/*      New Year */
#define O [v]/* Colin.I.King */


                       typedef a
        ;              a m, v[6]        ,
         H;a main(
                       ){char C(
                   o){033,91,0},C(D)
                      "*Oo", C(t
                     )"^~#",Q[ ]=
                    "13747516",C(s
                    )".x+*";u S"2"
                     "J%s0;0H%s0"
                       ";37;40"
                      "m",o,o, o
                    c while(U(!!m)
                  <22)u S"%dm%39s\n"        ,
                 o,0 O++>19?42:',',""
                c while(0 O++<'~') u S
               "%d;%dH%s44m%c",o,e%21,e
               %39,o,r(s)c for(J){1 O=1
              -U(1),srand(v),u S"0;0"W S
              "0;2;%dm",o,o,' 'c for(m=0
              ;m>>4<1;++m){u S"%d;%d"W,o
    ,          m+2,20-m c;for(H=0;H<1+(m
               <<1);H++){4 O=!H|H==m<<1        ,
                2 O=!(e&05),U(3)=H>m*5/
                 3,5 O=r(D)J if(4 O|U(
                  2)){u S"%d;%d;3%cm"
                    "%c",o,U(4)?','
:'*',3 O?2:1+(U(1)^(1&e)),r(Q),U(5)c}else u S"42;32\
;%dm%c",o,1+3 O,r(t)c u S"0m",o c} }while(m<19)u S"\
%d;19"W S"33;2;7m  #\n",o,1+ ++m,o c sleep(m>=-H c}}

The source can be downloaded from here and compiled and run as follows:

gcc snowman.c -o snowman
./snowman

and press control-C to exit when you have seen enough.

Measuring stalled instructions with perf stat

Recently I was playing around with CPU loading and was trying to estimate the number of compute operations being executed on my machine.  In particular, I was interested to see how many instructions per cycle and stall cycles I was hitting on the more demanding instructions.   Fortunately, perf stat allows one to get detailed processor statistics to measure this.

In my first test, I wanted to see how the Intel rdrand instruction performed with 2 CPUs loaded (each with a hyper-thread):

$ perf stat stress-ng --rdrand 4 -t 60 --times
stress-ng: info: [7762] dispatching hogs: 4 rdrand
stress-ng: info: [7762] successful run completed in 60.00s
stress-ng: info: [7762] for a 60.00s run time:
stress-ng: info: [7762]     240.01s available CPU time
stress-ng: info: [7762]     231.05s user time   ( 96.27%)
stress-ng: info: [7762]       0.11s system time (  0.05%)
stress-ng: info: [7762]     231.16s total time  ( 96.31%)

 Performance counter stats for 'stress-ng --rdrand 4 -t 60 --times':

     231161.945062      task-clock (msec)         #    3.852 CPUs utilized          
            18,450      context-switches          #    0.080 K/sec                  
                92      cpu-migrations            #    0.000 K/sec                  
               821      page-faults               #    0.004 K/sec                  
   667,745,260,420      cycles                    #    2.889 GHz                    
   646,960,295,083      stalled-cycles-frontend   #   96.89% frontend cycles idle   
         stalled-cycles-backend   
    13,702,533,103      instructions              #    0.02  insns per cycle        
                                                  #   47.21  stalled cycles per insn
     6,549,840,185      branches                  #   28.334 M/sec                  
         2,352,175      branch-misses             #    0.04% of all branches        

      60.006455711 seconds time elapsed

stress-ng's rdrand test just performs a 64 bit rdrand read and loops on this until the data is ready, and performs this 32 times in an unrolled loop.  Perf stat shows that each rdrand + loop sequence on average consumes about 47 stall cycles showing that rdrand is probably just waiting for the PRNG block to produce random data.

My next experiment was to run the stress-ng ackermann stressor; this performs a lot of recursion, hence one should see a predominantly large amount of branching.

$ perf stat stress-ng --cpu 4 --cpu-method ackermann -t 60 --times
stress-ng: info: [7796] dispatching hogs: 4 cpu
stress-ng: info: [7796] successful run completed in 60.03s
stress-ng: info: [7796] for a 60.03s run time:
stress-ng: info: [7796]     240.12s available CPU time
stress-ng: info: [7796]     226.69s user time   ( 94.41%)
stress-ng: info: [7796]       0.26s system time (  0.11%)
stress-ng: info: [7796]     226.95s total time  ( 94.52%)

 Performance counter stats for 'stress-ng --cpu 4 --cpu-method ackermann -t 60 --times':

     226928.278602      task-clock (msec)         #    3.780 CPUs utilized          
            21,752      context-switches          #    0.096 K/sec                  
               127      cpu-migrations            #    0.001 K/sec                  
               927      page-faults               #    0.004 K/sec                  
   594,117,596,619      cycles                    #    2.618 GHz                    
   298,809,437,018      stalled-cycles-frontend   #   50.29% frontend cycles idle   
         stalled-cycles-backend   
   845,746,011,976      instructions              #    1.42  insns per cycle        
                                                  #    0.35  stalled cycles per insn
   298,414,546,095      branches                  # 1315.017 M/sec                  
        95,739,331      branch-misses             #    0.03% of all branches        

      60.032115099 seconds time elapsed

..so about 35% of the time is used in branching and we're getting  about 1.42 instructions per cycle and no many stall cycles, so the code is most probably executing inside the instruction cache, which isn't surprising because the test is rather small.

My final experiment was to measure the stall cycles when performing complex long double floating point math operations, again with stress-ng.

$ perf stat stress-ng --cpu 4 --cpu-method clongdouble -t 60 --times
stress-ng: info: [7854] dispatching hogs: 4 cpu
stress-ng: info: [7854] successful run completed in 60.00s
stress-ng: info: [7854] for a 60.00s run time:
stress-ng: info: [7854]     240.00s available CPU time
stress-ng: info: [7854]     225.15s user time   ( 93.81%)
stress-ng: info: [7854]       0.44s system time (  0.18%)
stress-ng: info: [7854]     225.59s total time  ( 93.99%)

 Performance counter stats for 'stress-ng --cpu 4 --cpu-method clongdouble -t 60 --times':

     225578.329426      task-clock (msec)         #    3.757 CPUs utilized          
            38,443      context-switches          #    0.170 K/sec                  
                96      cpu-migrations            #    0.000 K/sec                  
               845      page-faults               #    0.004 K/sec                  
   651,620,307,394      cycles                    #    2.889 GHz                    
   521,346,311,902      stalled-cycles-frontend   #   80.01% frontend cycles idle   
         stalled-cycles-backend   
    17,079,721,567      instructions              #    0.03  insns per cycle        
                                                  #   30.52  stalled cycles per insn
     2,903,757,437      branches                  #   12.873 M/sec                  
        52,844,177      branch-misses             #    1.82% of all branches        

      60.048819970 seconds time elapsed

The complex math operations take some time to complete, stalling on average over 35 cycles per op.  Instead of using 4 concurrent processes, I re-ran this using just the two CPUs and eliminating 2 of the hyperthreads.  This resulted in 25.4 stall cycles per instruction showing that hyperthreaded processes are stalling because of contention on the floating point units.

Perf stat is an incredibly useful tool for examining performance issues at a very low level.   It is simple to use and yet provides excellent stats to allow one to identify issues and fine tune performance critical code.  Well worth using.

Systems Performance: Enterprise and the Cloud

Before I started some analysis on benchmarking various popular file systems on Linux I was recommended to read "Systems Performance: Enterprise  and the Cloud" by Brendan Gregg.

In today's modern server and cloud based systems the multi-layered complexity can make it hard to pin point performance issues and bottlenecks. This book is packed full useful analysis techniques covering tracing, kernel internals, tools and benchmarking.

Critical to getting a well balanced and tuned system are all the different components, and the book has chapters covering CPU optimisation (cores, threading, caching and internconnects),  memory optimisation (virtual memory, paging, swapping, allocators, busses),  file system I/O, storage, networking (protcols, sockets, physical connections) and typical issues facing cloud computing.

The book is full of very useful examples and practical instructions on how to drill down and discover performance issues in a system and also includes some real-world case studies too.

It has helped me become even more focused on how to analyse performance issues and consider how to do deep system instrumentation to be able to understand where any why performance regressions occur.

All-in-all, a most systematic and well written book that I'd recommend to anyone running large complex servers and cloud computing environments.

even more stress in stress-ng

Over the past few weeks in spare moments I've been adding more stress methods to stress-ng  ready for Ubuntu 15.04 Vivid Vervet.   My intention is to produce a rich set of stress methods that can stress and exercise many facets of a system to force out bugs, catch thermal over-runs and generally torture a kernel in a controlled repeatable manner.

I've also re-structured the tool in several ways to enhance the features and make it easier to maintain.  The cpu stress method has been re-worked to include nearly 40 different ways to stress a processor, covering:

• Bit manipulation: bitops, crc16, hamming
• Integer operations: int8, int16, int32, int64, rand
• Floating point:  long double, double,  float, ln2, hyperbolic, trig
• Recursion: ackermann, hanoi
• Computation: correlate, euler, explog, fibonacci, gcd, gray, idct, matrixprod, nsqrt, omega, phi, prime, psi, rgb, sieve, sqrt, zeta
• Hashing: jenkin, pjw
• Control flow: jmp, loop

..the intention was to have a wide enough eclectic mix of CPU exercising tests that cover a wide range of typical operations found in computationally intense software.   Use the new --cpu-method option to select the specific CPU stressor, or --cpu-method all to exercise all of them sequentially.

I've also added more generic system stress methods too:

• bigheap - re-allocs to force OOM killing
• rename - rename files rapidly
• utime - update file modification times to create lots of dirty file metadata
• fstat - rapid fstat'ing of large quantities of files
• qsort - sorting of large quantities of random data
• msg - System V message sending/receiving
• nice - rapid re-nicing processes
• sigfpe - catch rapid division by zero errors using SIGFPE
• rdrand - rapid reading of Intel random number generator using the rdrand instruction (Ivybridge and later CPUs only)

Other new options:

• metrics-brief - this dumps out only the bogo-op metrics that are relevant for just the tests that were run.
• verify - this will sanity check the stress results per iteration to ensure memory operations and CPU computations are working as expected. Hopefully this will catch any errors on a hot machine that has errors in the hardware. 
• sequential - this will run all the stress methods one by one (for a default of 60 seconds each) rather than all in parallel.   Use this with the --timeout option to run all the stress methods sequentially each for a specified amount of time. 
• Specifying 0 instances of any stress method will run an instance of the stress method on all online CPUs. 

The tool also builds and runs on Debian kFreeBSD and GNU HURD kernels although some stress methods or stress options are not included due to lack of support on these other kernels.
The stress-ng man page gives far more explanation of each stress method and more detailed examples of how to use the tool.

For more details, visit here or read the manual.

a final few more features in stress-ng

While hoping to get a feature complete stress-ng sooner than later, I found a few more ways to fiendishly stress a system.

Stress-ng 0.01.22 will be landing soon in Ubuntu 14.10 with three more stress mechanisms:

• CPU affinity stressing; this rapidly changes CPU affinity of the stress processes just to keep the scheduling busy wasting effort.
• Timer stressing using the real-time clock; this allows one to generate a large amount of timer interrupts, so it is a useful interrupt saturation test.
• Directory entry thrashing; this creates and deletes a selectable number of zero length files and hence populates and destroys directory entries.

I have also removed the need to use rand() for random number generation for some of the stress tests and re-used a the faster MWC "random" number generator to add in some well known and very simple math operations for CPU stressing.

Stress-ng now has 15 different simple stress mechanisms that exercise CPU, cache, memory, file system, I/O and CPU schedulers.  I could add more tests, but I think this is a large enough set to allow one to thrash a machine and see how well it performs under pressure.

more stress with stress-ng

Since my last article about stress-ng I have been adding a few more stress mechanisms to stress-ng:

• file locking - exercise file locking with one or more processes (the more processes the better).
• fallocate - this allocates a 4MB file, sync's, truncates to zero size and syncs repeatedly
• yield - this loops on sched_yield() to repeatedly relinquish the CPU forcing a high context switch rate when run with multiple yielding processes.

Also, I have added some new features to tweak scheduling, I/O characteristics and memory allocations of the running stress processes:

• --sched and --sched-prio options to specify the scheduler type and priority
• --ionice-class and --ionice-level options to tweak I/O niceness
• --vm-populate option to populate (pre-fault) page tables for a mapping for the --vm stress test.

If I think of other mechanisms to stress the kernel I will add them, but for now, stress-ng is becoming almost feature complete.

idlestat: a tool to measure times in idle and operational states

Linaro's idlestat is another useful tool in the arsenal of CPU monitoring utilities.  Idlestat monitors and captures CPU C-state and P-state transitions using the kernel Ftrace tracer and outputs statistics based on entering/exiting each state for each CPU.  Idlestat  also captures IRQ activity as well which ones caused a CPU to exit an idle state -  knowing why a processor came out of a deep C state is always very useful way to help diagnose power consumption issues.

Using idlestat is easy, to capture 20 seconds of activity into a log file called example.log, run:

 sudo idlestat --trace -f example.log -t 20    

..and this will display the per CPU C-state and P-state and IRQ statistics for that run.

One can also take the saved log file and parse it again to calculate the statistics again using:

 idlestat --import -f example.log  

One can get the source from here and I've packaged version 0.3 (plus a bunch of minor fixes that will land in 0.4) for Ubuntu 14.10 Utopic Unicorn.

stress-ng: an updated system stress test tool

Recently added to Ubuntu 14.10 is stress-ng, a simple tool designed to stress various components of a Linux system.   stress-ng is a re-implementation of the original stress tool written by Amos  Waterland and adds various new ways to exercise a computer as well as a very simple "bogo-operation" set of metrics for each stress method.

stress-ng current contains the following methods to exercise the machine:

• CPU compute - just lots of sqrt() operations on pseudo-random values. One can also specify the % loading of the CPUs
• Cache thrashing, a naive cache read/write exerciser
• Drive stress by writing and removing many temporary files
• Process creation and termination, just lots of fork() + exit() calls
• I/O syncs, just forcing lots of sync() calls
• VM stress via mmap(), memory write and munmap()
• Pipe I/O, large pipe writes and reads that exercise pipe, copying and context switching
• Socket stressing, much like the pipe I/O test but using sockets
• Context switching between a pair of producer and consumer processes

Many of the above stress methods have additional configuration options.  Each stress method can be run by one or more child processes.

The --metrics option dumps the number of operations performed by each stress method, aka "bogo ops", bogos because they are a rough and unscientific metric.  One can specify how long to run a test either by test duration in sections or by bogo ops.

I've tried to make stress-ng compatible with the older stress tool, but note that it is not guaranteed to produce identical results as the common test methods between the two tools have been implemented differently.

Stress-ng has been a useful for helping me measure different power consuming loads.  It is also useful with various thermald optimisation tweaks on one of my older machines.

For more information, consult the stress-ng manual page.  Be warned, this tool can make your system get seriously busy and warm!

smemstat: a new tool to report shared memory usage

Back in April I wrote smemstat in to dump out the per process shared memory usage based on the memory mapping data from /proc/$pid/smaps.   I tried to make smemstat as compact as possible with minimised changes in heap size to reduce the impact on the total system shared memory statistics.

smemstat reports the memory utilised per process taking in consideration pages that are shared with other processes. So, if two processes share 64K of memory, each process will report that memory as 32K each.

smemstat has to modes of operation: "dump all current memory stats" and a "dump periodic change in memory stats".  The former is useful for a single snapshot view of memory utilisation, where as the latter is useful to observe memory size changes over time. Exiting or new processes that share memory with other processes will cause a change the reported amounted of memory used by the processes because this memory is shared amongst a changed number of processes.

There are various options to smemstat, so consult the man page for more details.  One useful option is -o that collects the smemstat statistics into a JSON formatted output file. This can be useful for memory based tests as the structured JSON data can be easily parsed and analysed.

smemstat  has landed in Ubuntu Utopic 14.10 and is also available in Debian. Examples of smemstat can be found here. I hope it proves to be a useful utility.

Firmware Test Suite new features for the Ubuntu 14.04 LTS release

In the last 6 months the Firmware Test Suite (fwts) has seen a lot of development and bug fixing activity in preparation for the Ubuntu 14.04 LTS Trusty release.  It is timely to give a brief overview of new features and improvements that have landed in this busy development cycle:

UEFI uefidump, additional support for:

• KEK, KEKDefault, PK, PKDefault global variables scan
• db, dbx, dbt variables scan
• messaging device path: add Fibre Channel Ex subtype-21, ATA subtype-18, Fibre Channel Ex subtype-21, USB WWID subtype-16, VLAN subtype-20, Device Logical Unit subtype-17, SAS Ex subtype-22, iSCSI subtype-19, NVM Express namespace subtype-23, Media Protocol subtype-5, PIWG Firmware File subtype-6, PIWG Firmware Volume subtype-7 and extend the Messaging Device Path type Vendor subtype-10.

ACPI related:

• Update to ACPICA version 20140325 
• Add S3 hybrid suspend / resume support with new --s3-hybrid option 
• Improved reporting of errors on ACPI evaluation errors
• New _PLD, _CRS and _PRS dump utilities
• New General Purpose Events (GPE) dump utility
• --disassemble-aml option accepts an output directory argument
• Add DBG2, DBGP, SPCR and MCHI tables to acpidump utility 
• Add -R, --rsdp option to specify the RSDP address
• _IFT, _SRV, _PIC, _UDP, _UPP, _PMM, _MSG, _GAI, _CID, _CDM and _CBA checks added to method test

SMBIOS:

• dmicheck: add more checks for invalid DMI fields

Architecture related:

• Support for i386 amd64 armel armhf aarch64 ppc64 ppc64e.  fwts support for aarch64 was a notable achievement

Kernel Log Scanning:

• Sync klog scanning with 3.13 kernel error messages

Miscellaneous:

• Remove unused LaunchPad bug tagging
• Add Ivybridge and Haswell MSRs to msr test
• Check CPU maximum frequencies

The fwts regression tests have been incorporated into the fwts repository and can be run with "make check". These tests are automatically run at build time to catch regressions.  fwts is now being regularly checked with static code analysis tools smatch, cppcheck and Coverity Scan and this has helped find memory leaks and numerous corner case bugs.  We also exercise fwts with a database of ACPI tables from real hardware and synthetically generated broken tables to check for regressions. 

Contributors to fwts in the current release cycle are (in alphabetical order):  Alex Hung, Colin King, Ivan Hu, Jeffrey Bastian, Keng-Yu Lin, Matt Fleming.  Also, thanks to Naresh Bhat for testing and feedback for the aarch64 port and to Robert Moore for the on-going work with ACPICA.

As ever, all contributions are welcome, including bug reports and feature requests.  Visit the fwts wiki page for more details.

forkstat: a new tool to trace process activity

One of my on-going projects is to try to reduce system activity where possible to try to shave off wasted power consumption.   One of the more interesting problems is when very short lived processes are spawned off and die and traditional tools such as ps and top sometimes don't catch that activity.   Over last weekend I wrote the bulk of the forkstat tool to track down these processes.

Forkstat uses the kernel proc connector interface to detect process activity.  Proc connector allows forkstat to receive notifications of process events such as fork, exec, exit, core dump and changing the process name in the comm field over a socket connection.

By default, forkstat will just log fork, exec and exit events, but the -e option allows one to specify one or more of the fork, exec, exit, core dump or comm events.  When a fork event occurs, forkstat will log the PID and process name of the parent and child, allowing one to easily identify where processes are originating.    Where possible, forkstat attempts to track the life time of a process and will log the duration of a processes when it exits (note: this is not an estimate of the CPU used).

The -S option to forkstat will dump out a statistical summary of activity.  This is useful to identify the frequency of processes activity and hence identifying the top offenders.

Forkstat is now available in Ubuntu 14.04 Trusty Tahr LTS.  To install forkstat use:

 sudo apt-get install forkstat  

For more information on the tool and examples of the forkstat output, visit the forkstat quick start page.

Keeping cool with thermald

The push for higher performance desktops and laptops has inevitably lead to higher power dissipation.  Laptops have also shrunk in size leading to increasing problems with removing excess heat and thermal overrun on heavily loaded high end machines.

Intel's thermald prevents machines from overheating and has been recently introduced in the Ubuntu Trusty 14.04 LTS release.  Thermald actively monitors thermal sensors and will attempt to keep the hardware cool by modifying a variety of cooling controls:
 

* Active or passive cooling devices as presented in sysfs

* The Running Average Power Limit (RAPL) driver (Sandybridge upwards)

* The Intel P-state CPU frequency driver (Sandybridge upwards)

* The CPU freq driver

* The Intel PowerClamp driver

Thermald has been found to be especially useful when using the Intel P-state CPU frequency scaling driver since this can push the CPU harder than other CPU frequency scaling drivers.

Over the past several weeks I've been working with Intel to shake out some final bugs and get thermald included into Ubuntu 14.04 LTS, so kudos to Srinivas Pandruvada for handling my patches and also providing a lot of timely fixes too.

By default, thermald works without any need for configuration, however, if one has incorrect thermal trip settings or other firmware related thermal zone bugs one can write one's own thermald configuration. 

For further details, consult the Ubuntu thermald wiki page.

Finding small bugs

Over the past few months I've been using static code analysis tools such as cppcheck, Coverity Scan and also smatch on various open source projects.   I've generally found that most open source code is fairly well written, however, most suffer a common pattern of bugs on the error handling paths.  Typically, these are not free'ing up memory or freeing up memory incorrectly.  Other frequent bugs are not initialising variables and overly complex code paths that introduce subtle bugs when certain rare conditions are occur.  Most of these bugs are small and very rarely hit; some of these just silently do things wrong while others can potentially trigger segmentation faults.

The --force option in cppcheck to force the checking of every build configuration has been very useful in finding code paths that are rarely built, executed or tested and hence are likely to contain bugs.

I'm coming to the conclusion that whenever I have to look at some new code I should take 5 minutes or so throwing it at various static code analysis tools to see what pops out and being a good citizen and fixing these and sending these upstream. It's not too much effort and helps reduce some of those more obscure bugs that rarely bite but do linger around in code.

cppcheck - another very useful static code analysis tool

Over the past months I have been using static code analysis tools such as smatch and Coverity Scan on various open source projects that I am involved with.  These, combined with using gcc's -Wall -Wextra have proved useful in tracking down and eliminating various bugs.

Recently I stumbled on cppcheck and gave it a spin on several larger projects.  One of the cppcheck project aims is to find errors that the compiler won't spot and also try to keep the number of false positives found to a minimum.

cppcheck is very easy to use, the default settings just work out of the box. However, for extra checking I enabled the --force option to check of all configurations and the --enable=all to report on checks to be totally thorough and pedantic.

The --enable option is especially useful. It allows one to select different types of checking, for example, coding style, execution performance, portability, unused functions and missing include files.

Even though my code has been through smatch and Coverity Scan, cppcheck still managed to find a few issues using --enable=all

1. unused functions
2. a potential memory leak with realloc(), for example:

buf = realloc(buf, new_size);
if (!buf)
     return NULL;

if realloc() fails, buf can be leaked.  A potential fix is:

tmp = realloc(buf, new_size);
if (!tmp) {
     free(buf);
     return NULL;
} else
     buf = tmp;

3. some potential sscanf buffer overflows
4. some coding style improvements, for example, local auto variables could be moved to a deeper scope

So cppcheck worked well for me.  I recommend referring to the cppcheck project wiki to check out the features and then subjecting your code to it and seeing if it can find any bugs.