Slipping in a few more stress-ng features before 2015

During idle moments in the final few weeks of 2014 I have been adding some more stressors and features to stress-ng as well as tidying up the code and fixing some small bugs that have crept in during the last development spin.   Stress-ng aims to stress a machine with various simple torture tests to trip overheating and kernel race conditions.

The mmap stressor now has the '--mmap-file' to use synchronous file backed memory mapping instead of the default anonymous mapping, and the '--mmap-async' option enables asynchronous file mapping if desired.

For socket stressing, the '--sock-domain unix' option now allows AF_UNIX (aka AF_LOCAL) sockets to be used. This compliments the existing AF_INET and AF_INET6 IPv4 and IPv6 protocols available with this stress test.

The CPU stressor now includes mixed integer and floating point stressors, covering 32 and 64 bit integer mixes with the float, double and long double floating point types. The generated object code contains a nice mix of operations which should exercise various functional units in the CPU.  For example, when running on a hyper-threaded CPU one notices a performance hit because these cpu stressor methods heavily contend on the CPU math functional blocks.

File based locking has been extended with the new lockf stressor, this stresses multiple locking and unlocking on portions of a small file and the default blocking mode can be turned into a CPU consuming rapid polling retry with the '--lockf-nonblock' option.

The dup(2) system call is also now stressed with the new dup stressor. This just repeatedly dup's a file opened on /dev/zero until all the free file slots are full, and then closes these. It is very much like the open stressors.

The fcntl(2) F_SETLEASE command is stress tested with the new lease stressor. This has a parent process that rapidly locks and unlocks a file based lease and 1 or more child processes try to open this file and cause lease breaking signal notifications to the parent.

For x86 CPUs, the cache stressor includes two new cache specific options. The '--cache-fence' option forces write serialization on each store operation, while the '--cache-flush' option forces flush cache on each store operation. The code has been specifically written to not incur any overhead if these options are not enabled or are not available on non-x86 CPUs.

This release also includes the stress-ng project mascot too; a humble CPU being tortured and stressed by a rather angry flame.

For more information, visit the stress-ng project page, or check out the stress-ng manual.

Controlling data flow using sluice

Earlier this year I was instrumenting wifi power consumption and needed a way to produce and also consume data at a specific rate to pipe over netcat for my measurements.  I had some older crufty code around to do this but it needed some polishing up so I eventually got around to turning this into a more usable tool called "sluice".  (A sluice gate controls flow of water,  the sluice tool controls rate of data though a pipe).

The sluice package is currently available in PPA:colin-king/white and is built for Ubuntu Trusty, Utopic and Vivid, but there the git repository and tarballs are available too if one wants to build this from source.

The following starts a netcat 'server' to send largefile at a rate of 1MB a second using 1K buffers and reports the transfer stats to stderr with the -v verbose mode enabled:

cat largefile | sluice -r 1MB -i 1K -v | nc -l 127.0.0.1 1234 

Sluice also allows one to adjust the read/write buffer sizes dynamically to try to avoid buffer underflow or overflows while trying to match the specified transfer rate.

Sluice can be used as a data sink on the receiving side and also has an option to throw the data away if one just wants to stream data and test the data link rates, e.g., get data from somehost.com on port 1234 and throw it away at 2MB a second:

nc somehost.com 1234 | sluice -d -r 2MB -i 8K

And finally, sluice as a "tee" mode, where data is copied to stdout and to a specified output file using the -t option.

For more details, refer to the sluice project page.

It is approaching the Christmas Holiday season, so it's that time again to write some slightly obfuscated C in a seasonal way.  This year I thought I would try some coloured ASCII art for the output for a little variety.

#define r(s)  s[e%(sizeof s-1)]
#include  /*        */
#define S "%s"/*        Have */
#define u printf(/*        a */
#define c )J/*         Merry */
#define W "H"/*    Christmas */
#define e rand()/*       and */
#define U(i) v[i]/*        a */
#define C(q) q[]=/*    Happy */
#define J ;;/*      New Year */
#define O [v]/* Colin.I.King */


                       typedef a
        ;              a m, v[6]        ,
         H;a main(
                       ){char C(
                   o){033,91,0},C(D)
                      "*Oo", C(t
                     )"^~#",Q[ ]=
                    "13747516",C(s
                    )".x+*";u S"2"
                     "J%s0;0H%s0"
                       ";37;40"
                      "m",o,o, o
                    c while(U(!!m)
                  <22)u S"%dm%39s\n"        ,
                 o,0 O++>19?42:',',""
                c while(0 O++<'~') u S
               "%d;%dH%s44m%c",o,e%21,e
               %39,o,r(s)c for(J){1 O=1
              -U(1),srand(v),u S"0;0"W S
              "0;2;%dm",o,o,' 'c for(m=0
              ;m>>4<1;++m){u S"%d;%d"W,o
    ,          m+2,20-m c;for(H=0;H<1+(m
               <<1);H++){4 O=!H|H==m<<1        ,
                2 O=!(e&05),U(3)=H>m*5/
                 3,5 O=r(D)J if(4 O|U(
                  2)){u S"%d;%d;3%cm"
                    "%c",o,U(4)?','
:'*',3 O?2:1+(U(1)^(1&e)),r(Q),U(5)c}else u S"42;32\
;%dm%c",o,1+3 O,r(t)c u S"0m",o c} }while(m<19)u S"\
%d;19"W S"33;2;7m  #\n",o,1+ ++m,o c sleep(m>=-H c}}

The source can be downloaded from here and compiled and run as follows:

gcc snowman.c -o snowman
./snowman

and press control-C to exit when you have seen enough.

Measuring stalled instructions with perf stat

Recently I was playing around with CPU loading and was trying to estimate the number of compute operations being executed on my machine.  In particular, I was interested to see how many instructions per cycle and stall cycles I was hitting on the more demanding instructions.   Fortunately, perf stat allows one to get detailed processor statistics to measure this.

In my first test, I wanted to see how the Intel rdrand instruction performed with 2 CPUs loaded (each with a hyper-thread):

$ perf stat stress-ng --rdrand 4 -t 60 --times
stress-ng: info: [7762] dispatching hogs: 4 rdrand
stress-ng: info: [7762] successful run completed in 60.00s
stress-ng: info: [7762] for a 60.00s run time:
stress-ng: info: [7762]     240.01s available CPU time
stress-ng: info: [7762]     231.05s user time   ( 96.27%)
stress-ng: info: [7762]       0.11s system time (  0.05%)
stress-ng: info: [7762]     231.16s total time  ( 96.31%)

 Performance counter stats for 'stress-ng --rdrand 4 -t 60 --times':

     231161.945062      task-clock (msec)         #    3.852 CPUs utilized          
            18,450      context-switches          #    0.080 K/sec                  
                92      cpu-migrations            #    0.000 K/sec                  
               821      page-faults               #    0.004 K/sec                  
   667,745,260,420      cycles                    #    2.889 GHz                    
   646,960,295,083      stalled-cycles-frontend   #   96.89% frontend cycles idle   
         stalled-cycles-backend   
    13,702,533,103      instructions              #    0.02  insns per cycle        
                                                  #   47.21  stalled cycles per insn
     6,549,840,185      branches                  #   28.334 M/sec                  
         2,352,175      branch-misses             #    0.04% of all branches        

      60.006455711 seconds time elapsed

stress-ng's rdrand test just performs a 64 bit rdrand read and loops on this until the data is ready, and performs this 32 times in an unrolled loop.  Perf stat shows that each rdrand + loop sequence on average consumes about 47 stall cycles showing that rdrand is probably just waiting for the PRNG block to produce random data.

My next experiment was to run the stress-ng ackermann stressor; this performs a lot of recursion, hence one should see a predominantly large amount of branching.

$ perf stat stress-ng --cpu 4 --cpu-method ackermann -t 60 --times
stress-ng: info: [7796] dispatching hogs: 4 cpu
stress-ng: info: [7796] successful run completed in 60.03s
stress-ng: info: [7796] for a 60.03s run time:
stress-ng: info: [7796]     240.12s available CPU time
stress-ng: info: [7796]     226.69s user time   ( 94.41%)
stress-ng: info: [7796]       0.26s system time (  0.11%)
stress-ng: info: [7796]     226.95s total time  ( 94.52%)

 Performance counter stats for 'stress-ng --cpu 4 --cpu-method ackermann -t 60 --times':

     226928.278602      task-clock (msec)         #    3.780 CPUs utilized          
            21,752      context-switches          #    0.096 K/sec                  
               127      cpu-migrations            #    0.001 K/sec                  
               927      page-faults               #    0.004 K/sec                  
   594,117,596,619      cycles                    #    2.618 GHz                    
   298,809,437,018      stalled-cycles-frontend   #   50.29% frontend cycles idle   
         stalled-cycles-backend   
   845,746,011,976      instructions              #    1.42  insns per cycle        
                                                  #    0.35  stalled cycles per insn
   298,414,546,095      branches                  # 1315.017 M/sec                  
        95,739,331      branch-misses             #    0.03% of all branches        

      60.032115099 seconds time elapsed

..so about 35% of the time is used in branching and we're getting  about 1.42 instructions per cycle and no many stall cycles, so the code is most probably executing inside the instruction cache, which isn't surprising because the test is rather small.

My final experiment was to measure the stall cycles when performing complex long double floating point math operations, again with stress-ng.

$ perf stat stress-ng --cpu 4 --cpu-method clongdouble -t 60 --times
stress-ng: info: [7854] dispatching hogs: 4 cpu
stress-ng: info: [7854] successful run completed in 60.00s
stress-ng: info: [7854] for a 60.00s run time:
stress-ng: info: [7854]     240.00s available CPU time
stress-ng: info: [7854]     225.15s user time   ( 93.81%)
stress-ng: info: [7854]       0.44s system time (  0.18%)
stress-ng: info: [7854]     225.59s total time  ( 93.99%)

 Performance counter stats for 'stress-ng --cpu 4 --cpu-method clongdouble -t 60 --times':

     225578.329426      task-clock (msec)         #    3.757 CPUs utilized          
            38,443      context-switches          #    0.170 K/sec                  
                96      cpu-migrations            #    0.000 K/sec                  
               845      page-faults               #    0.004 K/sec                  
   651,620,307,394      cycles                    #    2.889 GHz                    
   521,346,311,902      stalled-cycles-frontend   #   80.01% frontend cycles idle   
         stalled-cycles-backend   
    17,079,721,567      instructions              #    0.03  insns per cycle        
                                                  #   30.52  stalled cycles per insn
     2,903,757,437      branches                  #   12.873 M/sec                  
        52,844,177      branch-misses             #    1.82% of all branches        

      60.048819970 seconds time elapsed

The complex math operations take some time to complete, stalling on average over 35 cycles per op.  Instead of using 4 concurrent processes, I re-ran this using just the two CPUs and eliminating 2 of the hyperthreads.  This resulted in 25.4 stall cycles per instruction showing that hyperthreaded processes are stalling because of contention on the floating point units.

Perf stat is an incredibly useful tool for examining performance issues at a very low level.   It is simple to use and yet provides excellent stats to allow one to identify issues and fine tune performance critical code.  Well worth using.