tag:blogger.com,1999:blog-10167091390533965352024-03-18T17:34:35.884+00:00A Smackerel of OpinionNotes and jottings from an ex-Ubuntu Kernel Team Engineer Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.comBlogger402125tag:blogger.com,1999:blog-1016709139053396535.post-61532517375554762332023-02-10T11:27:00.000+00:002023-02-10T11:27:06.883+00:00Integer shift gotcha<p> Left shifting values seems simple, but the following code contains a bug:<br /></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGglPcqwMpCFsDuwqLh6W728FNEnHVvuEVKuAOKsTXrj1mkest4AVAFhZZ3Dj5rp4FFICi83hY0pNJ-PV_rwQhuK5K_YwrJAZIsRKYn3HK9aK67-WrKr__m9gWAwL0R6RhRfnyNDdS3TTSLIkMnWpExopcLBo2Mn7ostrdIs2a0-ztI9dF-soq-DQ8-A/s342/shift-gotcha.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="227" data-original-width="342" height="212" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGglPcqwMpCFsDuwqLh6W728FNEnHVvuEVKuAOKsTXrj1mkest4AVAFhZZ3Dj5rp4FFICi83hY0pNJ-PV_rwQhuK5K_YwrJAZIsRKYn3HK9aK67-WrKr__m9gWAwL0R6RhRfnyNDdS3TTSLIkMnWpExopcLBo2Mn7ostrdIs2a0-ztI9dF-soq-DQ8-A/s320/shift-gotcha.png" width="320" /> </a></div><div class="separator" style="clear: both; text-align: center;"> </div><div class="separator" style="clear: both; text-align: left;">The literal value 1 is actually a signed int, so the promotion to a uint64_t type will sign extend the value before assigning to x, causing x to be 0xffffffff80000000.</div><div class="separator" style="clear: both; text-align: left;"> </div><div class="separator" style="clear: both; text-align: left;">The fix is simple, just make the literal value 1 an unsigned int using 1U instead of 1.</div><div class="separator" style="clear: both; text-align: left;"> <br /></div><div class="separator" style="clear: both; text-align: center;"> </div><div class="separator" style="clear: both; text-align: center;"> </div><br /><p></p>Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-82170297141537639932021-07-24T23:57:00.003+01:002021-07-24T23:59:14.849+01:00Intel Hardware P-State (HWP) / Intel Speed Shift<p>Intel Hardware P-State (aka Harware Controlled Performance or "Speed Shift") (HWP) is a feature found in more modern x86 Intel CPUs (Skylake onwards). It attempts to select the best CPU frequency and voltage to match the optimal power efficiency for the desired CPU performance. HWP is more responsive than the older operating system controlled methods and should therefore be more effective.</p><p>To test this theory, I exercised my Lenovo T480 i5-8350U 8 thread CPU laptop with the <a href="https://kernel.ubuntu.com/~cking/stress-ng/">stress-ng</a> cpu stressor using the "double" precision math stress method, exercising 1 to 8 of the CPU threads over a 60 second test run. The average CPU temperature and average CPU frequency were measured using <a href="https://kernel.ubuntu.com/~cking/powerstat/">powerstat</a> and the CPU compute throughput was measured using the stress-ng bogo-ops count.</p><p>The HWP mode was set using the x86_energy_perf_policy tool (as found in the Linux source in <a href="https://github.com/torvalds/linux/tree/master/tools/power/x86/x86_energy_perf_policy">tools/power/x86/x86_energy_perf_policy</a>). This allows one to select one of 5 policies: "normal", "performance", "balance-performance", "balance-power" and "power" as well as enabling or disabling turbo frequencies. For the tests, turbo mode was also enabled to allow the CPU to run at higher CPU turbo frequencies.<br /></p><p>The "performance" policy is the least efficient option as the CPU is clocked at a high frequency even when the system is idle and is not idea for a laptop. The "power" policy will optimize for low power; on my system it set the CPU to a maximum of 400MHz which is not ideal for typical uses.</p><p>The more useful "balance-performance" option optimizes for good throughput at the cost of power consumption where as the "balance-power" option optimizes for good power consumption in preference to performance, so I tested these two options.</p><p><b>Comparison #1, CPU throughput (bogo-ops) vs CPU frequency.</b></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQySfovzuNY1dL3DgQ0NhYyloXc_g21tXIWYisDksugFZYCBPUQQQQ8CMuHp4bv3hpMnReWMFN00qk4qGDAVnRoJ3XL3TjJ9UrWViI5aXmZfE6nwr4pFncOIfX8buMYPn_asR3wof2erKP/s605/bogo-ops-vs-ghz.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQySfovzuNY1dL3DgQ0NhYyloXc_g21tXIWYisDksugFZYCBPUQQQQ8CMuHp4bv3hpMnReWMFN00qk4qGDAVnRoJ3XL3TjJ9UrWViI5aXmZfE6nwr4pFncOIfX8buMYPn_asR3wof2erKP/w640-h360/bogo-ops-vs-ghz.png" width="640" /></a></div><p>The two HWP policies are almost identical in CPU bogo-ops throughput vs CPU frequency. This is hardly surprising - the compute throughput for math intensive operations should scale with CPU clock frequency. Note that 5 or more CPU threads sees a reduction in compute throughput because the CPU <a href="https://en.wikipedia.org/wiki/Hyper-threading">hyper-threads</a> are being used.</p><p><b>Comparison #2, CPU package temperature vs CPU threads used.</b></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwwKfcI1If3KeOYgMuBlEKGXMKxF9usFw3IIaUrzjnZK57rLvotNg5taeVQEKdOA58k07jIVGSpcttjQHkzFirX6PNxOTulUqQnqDNUfqj_SqMV3gIA1fMcnPPMD3vohgsm9nvryWann3L/s605/cpu-temp-vs-threads.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwwKfcI1If3KeOYgMuBlEKGXMKxF9usFw3IIaUrzjnZK57rLvotNg5taeVQEKdOA58k07jIVGSpcttjQHkzFirX6PNxOTulUqQnqDNUfqj_SqMV3gIA1fMcnPPMD3vohgsm9nvryWann3L/w640-h360/cpu-temp-vs-threads.png" width="640" /></a></div><p>Not a big surprise, the more CPU threads being exercised the hotter the CPU package will get. The balance-power policy shows a cooler running CPU than the balance-performance policy. The balance-performance policy is running hot even when one or a few threads are being used.</p><p><b>Comparison #3, Power consumed vs CPU threads used.</b></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7T5cOrOGZWXpb8VBvvd8HmDbPB5grxRmScHw3PnPloQRh8Ho4BID6SfG_oDX4UZwmWT9KVPpFlXIgUGoazK5QWc9AnB5gngRaX-NPdoGIRhhCdxvOTSx_rUMHk6n1ZHac5l1NWXzip_oQ/s605/power-vs-threads.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7T5cOrOGZWXpb8VBvvd8HmDbPB5grxRmScHw3PnPloQRh8Ho4BID6SfG_oDX4UZwmWT9KVPpFlXIgUGoazK5QWc9AnB5gngRaX-NPdoGIRhhCdxvOTSx_rUMHk6n1ZHac5l1NWXzip_oQ/w640-h360/power-vs-threads.png" width="640" /></a></div><p>Clearly the balance-performance option is consuming more power than balance-power, this matches the CPU temperature measurements too. More power, higher temperature.<br /></p><p>Comparison #4, Maximum CPU frequency vs CPU threads used.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi45liFxQqDMB08jhKa-aFNIZDUrvOk2MzfQQI8iykkVsxSlKx9ZlNZWUR5YzBt_BCqL5Y-RyUORJL6s-49wPmd8l6g8KiKNKHTvA0sJBtusbXdDx_kffk1L4VQrfD9VoBXTz5i1vDxBZNI/s605/cpu-freq-vs-threads.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi45liFxQqDMB08jhKa-aFNIZDUrvOk2MzfQQI8iykkVsxSlKx9ZlNZWUR5YzBt_BCqL5Y-RyUORJL6s-49wPmd8l6g8KiKNKHTvA0sJBtusbXdDx_kffk1L4VQrfD9VoBXTz5i1vDxBZNI/w640-h360/cpu-freq-vs-threads.png" width="640" /></a></div><p>With the balance-performance option, the average maximum CPU frequency drops as more CPU threads are used. Intel turbo boost allows one to clock a few CPUs to higher frequencies, exercising more CPUs leads to more power and hence more heat. To keep the CPU package from hitting thermal overrun, CPU frequency and voltage has to be scaled down when using more CPUs. </p><p>This also is true (but far less pronounced) for the balance-power option. As once can see, balance-performance runs the CPU at a much higher frequency, which is great for compute at the expense of power consumption and heat.</p><p><b>Comparison #5, Compute throughput vs power consumed</b>.</p><p>So running with the balance-performance runs the CPU at higher speed and hence one gets more compute throughput per unit of time compared to the balance-power mode. That's great if your laptop is plugged into the mains and you want to get some compute intensive tasks performed quickly. However, is this more efficient? </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrgSydvt6Y4-nuPDBdUxiKYBUGztqmTgPKo-iQ-rnrK2MPWewtWaA9YwPlpGe34_l4Z1ZhOVWbitiUjCUvfxCPeHD6ryhaOI4pVtszjrByfGt7IAAFiQdQUQYDlTt5Tu7slFRJozHHxg_A/s605/compute-vs-power.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrgSydvt6Y4-nuPDBdUxiKYBUGztqmTgPKo-iQ-rnrK2MPWewtWaA9YwPlpGe34_l4Z1ZhOVWbitiUjCUvfxCPeHD6ryhaOI4pVtszjrByfGt7IAAFiQdQUQYDlTt5Tu7slFRJozHHxg_A/w640-h360/compute-vs-power.png" width="640" /></a></div><p>Comparing the amount of compute performance with the power consumed shows that the balance-power option is more efficient than balance-performance. Basically with balance-power more compute is possible with the same amount of energy compared to balance-performance, but it will take longer to complete.</p><p><b>CPU frequency scaling over time </b><br /></p><p>The 60 second duration tests were long enough for the CPU to warm up enough reach thermal limits causing HWP to throttle back the voltage and CPU frequencies. The following graphs illustrate how running with the balance-performance option allows the CPU to run for several seconds at a high turbo frequency before it hits a thermal limit and then the CPU frequency and power is adjusted to avoid thermal overrun:<br /></p><p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ_vaOjYIRU3ewvEsJbPGF6uw7BDREWWyzOPCRskdREEgQOt44Ii_LMWTNJ1CuuGi7tqTK77Nl3-26Hcja5OAXxdV1UwFe7xvNC0RI_9GjaUntuWB28u5DEVLWBlBMMz1mfYvnE5QxffP8/s605/cpu-temp-over-time.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ_vaOjYIRU3ewvEsJbPGF6uw7BDREWWyzOPCRskdREEgQOt44Ii_LMWTNJ1CuuGi7tqTK77Nl3-26Hcja5OAXxdV1UwFe7xvNC0RI_9GjaUntuWB28u5DEVLWBlBMMz1mfYvnE5QxffP8/w640-h360/cpu-temp-over-time.png" width="640" /></a></div>After 8 seconds the CPU package reached 92 degrees C and then CPU frequency scaling kicks in:<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3WUIoCsuB7SwSGcGRQKAawMpAdgA38GOAJ2GrtcWWBJDcxgjgdB4LTffkDoIQvZEaxmx_5b4M_U8YzRiV45J3QRDqHjvRTDi6LAt-9XCpvB2Pm6rL2idmmZk8wNwWBd6QQJABYzLbxRjV/s605/cpu-freq-over-time.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3WUIoCsuB7SwSGcGRQKAawMpAdgA38GOAJ2GrtcWWBJDcxgjgdB4LTffkDoIQvZEaxmx_5b4M_U8YzRiV45J3QRDqHjvRTDi6LAt-9XCpvB2Pm6rL2idmmZk8wNwWBd6QQJABYzLbxRjV/w640-h360/cpu-freq-over-time.png" width="640" /></a></div><p>..and power consumption drops too:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidvIBwGUF8QL9_vaZcXaKLPnZcC0Nnobrk8Pi-rP9d1Jc3dnXglvInfk5UdZaNCqwUPJHaE0ak2K09wv1bak8Nzg7sc29chBGNTHfDhJdS0tdpFt62zlBnBZqv2ZsTOjDZeZfAxY-E_9r-/s605/cpu-power-over-time.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidvIBwGUF8QL9_vaZcXaKLPnZcC0Nnobrk8Pi-rP9d1Jc3dnXglvInfk5UdZaNCqwUPJHaE0ak2K09wv1bak8Nzg7sc29chBGNTHfDhJdS0tdpFt62zlBnBZqv2ZsTOjDZeZfAxY-E_9r-/w640-h360/cpu-power-over-time.png" width="640" /></a></div>..it is interesting to note that we can only run for ~9 seconds before the CPU is scaled back to around the same CPU frequency that the balance-power option allows. <p><b>Conclusion </b><br /></p><p>Running with HWP balance-power option is a good default choice for maximizing compute while minimizing power consumption for a modern Intel based laptop. If one wants to crank up the performance at the expense of battery life, then the balance-performance option is most useful. <br /></p><p>The balance-performance option when a laptop is plugged into the mains (e.g. via a base-station) may seem like a good idea to get peak compute performance. Note that this may not be useful in the long term as the CPU frequency
may drop back to reduce thermal overrun. However, for bursty infrequent
demanding CPU uses this may be a good choice. I personally refrain from using this as it makes my CPU rather run hot and it's less efficient so it's not ideal for reducing my carbon footprint.</p><p>Laptop manufacturers normally set the default HWP option as "balance-power", but this may be changed in the BIOS settings (look for power balance, HWP or Speed Shift options) or changed with x86_energy_perf_policy tool (found in the linux-tools-generic package in Ubuntu).<br /></p>Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-1735277059437133932021-07-09T11:15:00.003+01:002021-07-09T11:15:29.042+01:00New features in stress-ng 0.12.12<p>The release of <a href="https://github.com/ColinIanKing/stress-ng">stress-ng 0.12.12</a> incorporates some useful features and a handful of new stressors.</p><p>Media devices such as HDDs and SSDs normally support <a href="https://en.wikipedia.org/wiki/S.M.A.R.T.">Self-Monitoring, Analysis and Reporting Technology</a> (S.M.A.R.T.) to detect and report various measurements of drive reliability. To complement the various file system and I/O stressors, stress-ng now has a --smart option that checks for any changes in the S.M.A.R.T. measurements and will report these at the end of a stress run, for example:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjk5kEX-BL6kALmmGEPYPC2DcIa5c585nq-0ix4Y0X2zO7y_YNL6ZY6YCGE5yocoIEbY6mdPdbJbVxywJU469YoEzVZHi1bc31XKhMy_0QslKRkPfABcJAb3Ue-GK1_kwadtKR7h9AJfk6I/s965/smart.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="194" data-original-width="965" height="129" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjk5kEX-BL6kALmmGEPYPC2DcIa5c585nq-0ix4Y0X2zO7y_YNL6ZY6YCGE5yocoIEbY6mdPdbJbVxywJU469YoEzVZHi1bc31XKhMy_0QslKRkPfABcJAb3Ue-GK1_kwadtKR7h9AJfk6I/w640-h129/smart.png" width="640" /></a></div><p>..as one can see, there are errors on /dev/sdc and this explains why the ZFS pool was having performance issues.</p><p>For x86 CPUs I have added a new stressor to trigger System Management Interrupts via writes to port 0xb2 to force the CPU into <a href="https://en.wikipedia.org/wiki/System_Management_Mode">System Management Mode</a> in ring -2. The --smi stressor option will also measure the time taken to service the SMI. To run this stressor, one needs the --pathological option since this may hang the computer and they behave like non-maskable interrupts:<br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgesElD65sFH_lE_1o2JyVqWpRfrnUc5doeywBZmEkwPqzR0xaTIRQgf93bmZ-oggoVorXq-aIpX4MLg0qXk_J5tZHDtoJ6K_YzFY_1Wk0ejA1mLIG72-ZmWg9jsCC4d3B7YBkoDI6dYI8V/s940/smi.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="100" data-original-width="940" height="68" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgesElD65sFH_lE_1o2JyVqWpRfrnUc5doeywBZmEkwPqzR0xaTIRQgf93bmZ-oggoVorXq-aIpX4MLg0qXk_J5tZHDtoJ6K_YzFY_1Wk0ejA1mLIG72-ZmWg9jsCC4d3B7YBkoDI6dYI8V/w640-h68/smi.png" width="640" /></a></div><p></p><p>To exercise the munmap(2) system call a new munmap stressor has been added. This creates child processes that walk through their memory mappings from /proc/$pid/maps and unmap pages on libraries that are not being used. The unapping is performed by striding across the mapping in page multiples of prime size to create many mapping holes to exercise the VM mapping structures. These unmappings can create SIGSEGV segmentation faults that silently get handled and respawn a new child stressor. Example:</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEig3aiLG5q0jQDYNRYs-cZWccrCbPLqCRRf1fq7b5s-bHYD47Doyqjta4I3OKM8j9GNNmdog-XJcYXsW7jlYvjwiI7TFWQvN7Vvt6PlVoA4crlWg5kjYJeAARP9ZisZtDuW-UyFcaRN8ocZ/s798/munmap.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="193" data-original-width="798" height="154" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEig3aiLG5q0jQDYNRYs-cZWccrCbPLqCRRf1fq7b5s-bHYD47Doyqjta4I3OKM8j9GNNmdog-XJcYXsW7jlYvjwiI7TFWQvN7Vvt6PlVoA4crlWg5kjYJeAARP9ZisZtDuW-UyFcaRN8ocZ/w640-h154/munmap.png" width="640" /></a></div> <p></p><p>There some new options for the fork, vfork and vforkmany stressors, a new vm mode has been added to try and exercise virtual memory mappings. This enables detrimental performance virtual memory advice using madvise on all pages of the new child process. Where possible this will try to set every page in the new process with using madvise MADV_MERGEABLE, MADV_WILLNEED, MADV_HUGEPAGE and MADV_RANDOM flags. The following shows how to enable the vm options for the fork and vfork stressors:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjem9Ow95SbtOciH57T6PkcqppX_w7lzmiyB5j5K7uZnjxUmYUFHsIFDPf1PuUbRAdYe7W1qu2byoefOnWTNkxIrKQo7I9YdIXIY-c0j76oP5O3Ci_SpSE7sgX3TK7mhBb2zXAkxPQmgw42/s790/vm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="199" data-original-width="790" height="162" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjem9Ow95SbtOciH57T6PkcqppX_w7lzmiyB5j5K7uZnjxUmYUFHsIFDPf1PuUbRAdYe7W1qu2byoefOnWTNkxIrKQo7I9YdIXIY-c0j76oP5O3Ci_SpSE7sgX3TK7mhBb2zXAkxPQmgw42/w640-h162/vm.png" width="640" /></a></div><p></p><p>One final new feature is the --skip-silent option. This will disable printing of messages when a stressor is skipped, for example, if the stressor is not supported by the kernel, the hardware or a support library is not available.</p><p>As usual for each release, stress-ng incorporates bug fixes and has been tested on a wide variety of Linux/*BSD/UNIX/POSIX systems and across a range of processor architectures (arm32, arm64, amd64, i386, ppc64el, RISC-V s390x, sparc64, m68k. It has also been statically analysed with <a href="https://www.synopsys.com/software-integrity/security-testing/static-analysis-sast.html">Coverity</a> and <a href="http://cppcheck.sourceforge.net/">cppcheck</a> and built cleanly with pedantic build flags on gcc and clang.</p><p><br /></p><p> </p><p><br /></p><p><br /></p><p><br /></p><p> <br /></p><p><br /></p><p><br /></p><p><br /></p><p><br /></p><p><br /></p><br /><br />Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-10826611146683442582021-05-21T10:12:00.002+01:002021-05-21T11:58:04.557+01:00Adjacent C string concatenation gotcha<p>C has the useful feature of adjacent allowing literal strings to be automatically concatenated. This is described in K&R "The C programming language" 2nd edition, page 194, section A2.6 "String Literals":</p><p>"Adjacent string literals are concatenated into a single string."</p><p>Unfortunately over the years I've seen several occasions where this useful feature has led to bugs not being detected, for example, the following code:</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiW1vhC36TS58k857xw6M5BMqH2ST_bKML43LMhMYMHZGBlSJSJDduetcoeYkBPnMTus6uADptLkM1eU9Ysf5S2eBdriZNSFG0IEpaLTueX6k-e6j5Yh0TjIUi0y2XuuTVVBJ7FQW2AWBly/s481/literals.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="195" data-original-width="481" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiW1vhC36TS58k857xw6M5BMqH2ST_bKML43LMhMYMHZGBlSJSJDduetcoeYkBPnMTus6uADptLkM1eU9Ysf5S2eBdriZNSFG0IEpaLTueX6k-e6j5Yh0TjIUi0y2XuuTVVBJ7FQW2AWBly/s16000/literals.png" /></a></div> <p></p><p>A simple typo in the "Does Not Compute" error message ends up with the last two literal strings being "silently" concatenated, causing an array out of bounds read when error_msgs[7] is accessed.</p><p>This can also bite when a new literal string is added to the end of the array. The previous string requires a comma to be added when a new string is added to the array, and sometimes this is overlooked. An example of this is in <a href="https://github.com/acpica/acpica/commit/81eb9c383e6dee0f1b6620e91e5c3dbb48234831">ACPICA commit 81eb9c383e6dee0f1b6620e91e5c3dbb48234831</a> - fortunately static analysis detected this and it has been fixed with <a href="https://github.com/acpica/acpica/commit/3f8c79fc22fd64b739f51268654a6783a874520e">commit 3f8c79fc22fd64b739f51268654a6783a874520e</a><br /></p><p>The concerning issue is that this useful string concatenation feature can produce hazardous outcomes with coding mistakes that are simple to make but hard to notice.<br /></p><p> </p><p> </p>Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-59318469371770811392021-04-23T12:55:00.004+01:002021-04-23T12:55:58.146+01:00C Ternary operator gotcha (type conversions)<p>The C ternary operator expr1 ? expr2 : expr3 has a subtle side note described in K&R 2nd edition, page 52, section 2.11: <br /></p><p>"If expr2 and expr3 are of different types, the type of the result is determined by the conversion rules discussed earlier in the chapter".</p><p>This refers to page 44, section 2.7 "Type Conversions". It's worth reading a few times along with section A6.5 "Arithmetic Conversions".<br /></p><p>Here is an example of a type conversion gotcha:<br /></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBjJJKqB-wliDsP46bEGZJjnm9h8Jpzt0Eb4CnkavcSpTJ87X8jA-7rQ1g4gRrVDXawDpJjixmm5NsmDOtnpngPA_qhNEKuraSFImTNKDXxuufVhQ1DpQgYoEE7scAbFCZId49iVjFbT9W/s436/ternary-1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="282" data-original-width="436" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBjJJKqB-wliDsP46bEGZJjnm9h8Jpzt0Eb4CnkavcSpTJ87X8jA-7rQ1g4gRrVDXawDpJjixmm5NsmDOtnpngPA_qhNEKuraSFImTNKDXxuufVhQ1DpQgYoEE7scAbFCZId49iVjFbT9W/s16000/ternary-1.png" /></a></div> <p></p><p>At a glance one would think the program would print out -1 as the output. Note that the expr2 is actually type converted to unsigned int so the result is 4294967295 if int types are 32 bits wide.</p><p>One solution is to type convert expr2 and/or expr3 to a long int and because that is wider than the unsigned int x this takes precedence. Alternatively just use:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKVyxyAzLK9MmJ9cWDp_7UowdEEKPbDMSlUqgc2iwNGvi0X9JacuzPvSnKdt_uU4GvRHghKrNx1IN-giwpAbs6GCVrJENNX_djXtvpcDfIYIbX5YbFpVjeRhwq5YZlzLuEan9_60ao7u9-/s432/ternary-2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="317" data-original-width="432" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKVyxyAzLK9MmJ9cWDp_7UowdEEKPbDMSlUqgc2iwNGvi0X9JacuzPvSnKdt_uU4GvRHghKrNx1IN-giwpAbs6GCVrJENNX_djXtvpcDfIYIbX5YbFpVjeRhwq5YZlzLuEan9_60ao7u9-/s16000/ternary-2.png" /></a></div><p>References: <a href="https://www.spinics.net/lists/kernel/msg3914809.html">soc: aspeed: fix a ternary sign expansion bug</a><br /></p>Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-87547040644919456382021-03-30T12:37:00.004+01:002021-03-30T12:37:32.923+01:00A C for-loop Gotcha<p>The C infinite for-loop gotcha is one of the less frequent issues I find with static analysis, but I feel it is worth documenting because it is obscure but easy to do.</p><p>Consider the following C example:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8M1-bKdo8C6wUwNCoSUl1ILBc-qGU3TxzPTcjViExA9elHvppiksccZf8jmP2ePAJOli_ok0MYAzSnW0r9nA1Zi_CdN7Xqsr50FD8E1_FbGvN7KS0H7dIRzlGYeWA08A4ywuNgmzcU0_w/s360/for-loop.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="160" data-original-width="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8M1-bKdo8C6wUwNCoSUl1ILBc-qGU3TxzPTcjViExA9elHvppiksccZf8jmP2ePAJOli_ok0MYAzSnW0r9nA1Zi_CdN7Xqsr50FD8E1_FbGvN7KS0H7dIRzlGYeWA08A4ywuNgmzcU0_w/s16000/for-loop.png" /></a></div><p></p><p>Since i is a 8 bit integer, it will wrap around to zero when it reaches the maximum 8 bit value of 255 and so we end up with an infinite loop if the upper limit of the loop n is 256 or more. </p><p>The fix is simple, always ensure the loop counter is at least as wide as the type of the maximum limit of the loop. This example, variable i should be a uint32_t type.</p><p>I've seen this occur in the Linux kernel a few times. Sometimes it is because the loop counter is being passed into a function call that expects a specific type such as a u8, u16. In other occasions I've seen a u16 (or short) integer being used presumably because it was expected to produce faster code, however, most commonly 32 bit integers just as fast (or sometimes faster) than 16 bit integers for this kind of operation.<br /></p>Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com1tag:blogger.com,1999:blog-1016709139053396535.post-46837175632867671712021-03-28T23:33:00.003+01:002021-03-28T23:33:46.672+01:00A Common C Integer Multiplication Mistake<p>Multiplying integers in C is easy. It is also easy to get it
wrong. A common issue found using static analysis on the Linux kernel
is the integer overflow before widening gotcha.</p><p>Consider the following code that takes the 2 unsigned 32 bit integers, multiplies them together and returns the unsigned 64 bit result:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMlEx1PLLtVQ6ljaMGGoNjbpvP5jumilajPtenoTU1MOE8_GcMrk3Tt0Lo2flRIpTWjwJ7qJA0PcW8dphC_lC6rLOBBf_v9_HvIfLm77Sh8MkLlgnaK-wTqaOJcK-FczSSEJo5LQIaH1AU/s403/32x32.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="118" data-original-width="403" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMlEx1PLLtVQ6ljaMGGoNjbpvP5jumilajPtenoTU1MOE8_GcMrk3Tt0Lo2flRIpTWjwJ7qJA0PcW8dphC_lC6rLOBBf_v9_HvIfLm77Sh8MkLlgnaK-wTqaOJcK-FczSSEJo5LQIaH1AU/s16000/32x32.png" /></a></div><p>The multiplication is performed using unsigned 32 bit arithmetic and the unsigned 32 bit results is widened to an unsigned 64 bit when assigned to ret. A
way to fix this is to explicitly cast a to a uint64_t before the multiplication to ensure an unsigned 64 bit multiplication is performed: <br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhimUzEkZlCLflAS9MPRTYivqEuhCnurrc-kvmLdxpyLp8BMDy-OMSC0bYmCLny-9v8l4mlSyOCKm3Cs8jej_CyjO3ZB9WU2JVipe-pPxeTpQIkfM2KxxBQcj6C1k-Yo8r537JZnyUMDh0l/s405/32x32ok.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="126" data-original-width="405" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhimUzEkZlCLflAS9MPRTYivqEuhCnurrc-kvmLdxpyLp8BMDy-OMSC0bYmCLny-9v8l4mlSyOCKm3Cs8jej_CyjO3ZB9WU2JVipe-pPxeTpQIkfM2KxxBQcj6C1k-Yo8r537JZnyUMDh0l/s16000/32x32ok.png" /></a></div>Fortunately static analysis finds these issues. Unfortunately it is a bug that keeps on occurring in new code.Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com1tag:blogger.com,1999:blog-1016709139053396535.post-37680078139409260312021-03-18T17:20:00.003+00:002021-03-18T17:30:35.751+00:00A common C integer shifting mistake<p>Shifting integers in C is easy. Alas it is also easy to get it wrong. A common issue found using static analysis on the Linux kernel is the unintentional sign extension gotcha.</p><p>Consider the following code that takes the 4 unsigned 8 bit integers in array data and returns an unsigned 64 bit integer: <br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj84SwprN05nV0JJ9LX2IQn2S0OSubbt2Z4Vn4qo_Ye39WbVMnHJWwHpXnK-exSRtXOkb91-n6Dk_-4MxWLd7KRBK9RPjUyIER-Zvhwa_cjRDmZUcY2hyfj7AHTOPauEQ7YUhlg3bK9R3po/s341/Cshift.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="212" data-original-width="341" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj84SwprN05nV0JJ9LX2IQn2S0OSubbt2Z4Vn4qo_Ye39WbVMnHJWwHpXnK-exSRtXOkb91-n6Dk_-4MxWLd7KRBK9RPjUyIER-Zvhwa_cjRDmZUcY2hyfj7AHTOPauEQ7YUhlg3bK9R3po/s16000/Cshift.png" /></a></div><p>C promotes the uint8_t integers into signed ints on the right shift. If data[3] has the upper bit set, for example with the value 0x80 and data[2]..data[0] are zero, the shifted 32 bit signed result is sign extended to a 64 bit long and the final result is 0xffffffff80000000. A way to fix this is to explicitly cast data[3] to a uint64_t before shifting it. <br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKODqSg1IT8m4u6iKGe0g6nQhrr5bdV4pFj4WJGVYCZaaQ7YLNSjgJps4Nj0_bPUKZAqEvV2bK7NkCWy1fgA03Oi7v5b-L-Df8Gp271OlIBz_4jVd7mWd2IuUjuDVfepH3aqQSTtPzp-SJ/s402/CshiftOK.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="212" data-original-width="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKODqSg1IT8m4u6iKGe0g6nQhrr5bdV4pFj4WJGVYCZaaQ7YLNSjgJps4Nj0_bPUKZAqEvV2bK7NkCWy1fgA03Oi7v5b-L-Df8Gp271OlIBz_4jVd7mWd2IuUjuDVfepH3aqQSTtPzp-SJ/s16000/CshiftOK.png" /></a></div><br /><p></p><p>Fortunately static analysis finds these issues. Unfortunately it is a bug that keeps on occurring in new code.<br /></p>Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com2tag:blogger.com,1999:blog-1016709139053396535.post-92019986356414044752021-01-04T16:44:00.000+00:002021-01-04T16:44:20.953+00:00Improving kernel test coverage with stress-ng<p>Over the past year there has been focused work on improving the test coverage of the Linux Kernel with stress-ng. Increased test coverage exercises more kernel code and hence improves the breadth of testing, allowing us to be more confident that more corner cases are being handled correctly.<br /></p><p>The test coverage has been improved in several ways:</p><ol style="text-align: left;"><li>testing more system calls; most system calls are being now exercised</li><li>adding more ioctl() command tests<br /></li><li>exercising system call error handling paths</li><li>exercise more system call options and flags</li><li>keeping track of new features added to recent kernels and adding stress test cases for these</li><li>adding support for new architectures (RISC-V for example) <br /></li></ol><p>Each stress-ng release is run with various stressor options against the latest kernel (built with <a href="https://www.kernel.org/doc/html/v4.14/dev-tools/gcov.html">gcov</a> enabled). The gcov data is processed with <a href="http://ltp.sourceforge.net/coverage/lcov.php">lcov</a> to produce human readable kernel source code containing <a href="https://kernel.ubuntu.com/~cking/kernel-coverage/stress-ng/">coverage annotations</a> to help inform where to add more test coverage for the next release cycle of stress-ng. <br /></p><p>Linux Foundation sponsored Piyush Goyal for 3 months to add test cases that exercise system call test failure paths and I appreciate this help in improving stress-ng. I finally completed this tedious task at the end of 2020 with the release of stress-ng 0.12.00.<br /></p><p>Below is a chart showing how the kernel coverage generated by stress-ng has been increasing since 2015. The amber line shows lines of code exercised and the green line shows kernel functions exercised.<br /></p><p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSzhdMghaV11m2nu1z8XT6vNZC2_aX5_aPhk_FZY3TA58jb3OYJQKz36LFFk5PyF2LFkEne4EIGPF2_rQH5hHoA3_8GfaGu2wBbcM04WElTq39w2vMIblMKhmWHgydo1JTAVGZXctpvoE9/s664/Screenshot_2021-01-04+Kernel+Test+-+Grafana%25281%2529.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="261" data-original-width="664" height="203" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSzhdMghaV11m2nu1z8XT6vNZC2_aX5_aPhk_FZY3TA58jb3OYJQKz36LFFk5PyF2LFkEne4EIGPF2_rQH5hHoA3_8GfaGu2wBbcM04WElTq39w2vMIblMKhmWHgydo1JTAVGZXctpvoE9/w515-h203/Screenshot_2021-01-04+Kernel+Test+-+Grafana%25281%2529.png" width="515" /></a></div><br /><p>..one can see that there was a large increase of kernel test coverage in the latter half of 2020 with stress-ng. In all, 2020 saw ~20% increase on kernel coverage, most of this was driven using the gcov analysis, however, there is more to do.</p><p>What next? Apart from continuing to add support for new kernel system calls and features I hope to improve the kernel coverage test script to exercise more file systems; it will be interesting to see what kind of bugs get found. I'll also be keeping the <a href="https://kernel.ubuntu.com/~cking/stress-ng/">stress-ng project</a> page refreshed as this tracks bugs that stress-ng has found in the Linux kernel.</p><p>As it stands, release 0.12.00 was a major milestone for stress-ng as it marks the completion of the major work items to improve kernel test coverage.<br /></p>Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-30926480069189583452020-09-25T13:07:00.003+01:002020-09-25T13:11:43.427+01:00Kernel janitor work: fixing spelling mistakes in kernel messages<p>The Linux 5.9-rc6 kernel source contains over 300,000 literal strings used in kernel messages of various sorts (errors, warnings, etc) and it is no surprise that typos and spelling mistakes slip into these messages from time to time. <br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTEwZTdSGmov04tc92X-AJ5la4NqqBluzFZF0IiDDY6AJMYcmopuk-C_h7SjpU1S0j-FvuLdu5Nwy8NQb4RpUSA9LLg5pPmX2vIMKfMfJJhQH6IKGNiNyM0gPyJU_agCRt37Rqsxe2eKO6/s150/oed.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="150" data-original-width="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTEwZTdSGmov04tc92X-AJ5la4NqqBluzFZF0IiDDY6AJMYcmopuk-C_h7SjpU1S0j-FvuLdu5Nwy8NQb4RpUSA9LLg5pPmX2vIMKfMfJJhQH6IKGNiNyM0gPyJU_agCRt37Rqsxe2eKO6/s16000/oed.png" /></a></div><p>To catch spelling mistakes I run a daily automated job that fetches the tip from <a href="https://www.kernel.org/doc/man-pages/linux-next.html">linux-next</a> and runs a fast spelling checker tool that finds all spelling mistakes and then diff's these against the results from the previous day. The diff is emailed to me and I put my kernel janitor hat on, fix these up and send these to the upstream developers and maintainers.</p><p>The spelling checker tool is a fast-and-dirty C parser that finds literal strings and also variable names and checks these against a US English dictionary containing over 100,000 words. As fun weekend side project I hand optimized the checker to be able to parse and spell check several millions lines of kernel C code per second.<br /></p><p>Every 3 or so months I collate all the fixes I've made and where appropriate I add new spelling mistake patterns to the kernel checkpatch spelling dictionary. Kernel developers should in practice run checkpatch.pl on their patches before submitting them upstream and hopefully the dictionary will catch a lot of the regular spelling mistakes.</p><p>Over the past couple of years I've seen less spelling mistakes creep into the kernel, either because folk are running checkpatch more nowadays and/or that the dictionary is now able to catch more spelling mistakes. As it stands, this is good as it means less work to fix these up.</p><p>Spelling mistakes may be trivial fixes, but cleaning these up helps make the kernel errors appear more professional and can also help clear up some ambiguous messages. <br /></p>Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-40129646888148931282020-04-30T13:01:00.003+01:002020-04-30T13:01:46.849+01:00easy capturing of kernel stack traces with virsh<span style="font-family: inherit;">Today I needed to capture a rather large kernel stack dump, this is rather trivial using virsh. Using virt-manager I created a VM named vm-focal and in the guest ran:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"><span class="prompt1">sudo systemctl <span class="nb">enable</span> serial-getty@ttyS0.service</span></span><span class="prompt1"> </span></span><br />
<br />
<span style="font-family: inherit;"><span class="prompt1"><span style="font-family: inherit;">Then on the host running the VM I ran<span style="font-family: inherit;">:</span></span></span></span><br />
<br />
<span style="font-family: "Courier New", Courier, monospace;">virsh console vm-focal</span><br />
<br />
<span class="prompt1"><span style="font-family: inherit;"><span class="prompt1"><span style="font-family: inherit;">Then all I needed to do was produce the stack dump and the console output was successfully dumped by virsh. Easy.</span></span></span></span><br />
<h2>
<span class="prompt1"><span style="font-family: inherit;"></span></span></h2>
Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-55183569021371400052019-11-04T09:33:00.001+00:002019-11-04T09:33:57.661+00:00stress-ng Embedded Linux Conference Europe 2019 presentationLast week I attended the <a href="https://events19.linuxfoundation.org/events/embedded-linux-conference-europe-2019/">Embedded Linux Conference </a>(Europe 2019) and presented a talk on <a href="https://kernel.ubuntu.com/~cking/stress-ng/">stress-ng</a>. The <a href="https://static.sched.com/hosted_files/osseu19/29/Lyon-stress-ng-presentation-oct-2019.pdf">slide deck for this presentation</a> is now available.Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-34666138731133012422019-10-17T11:24:00.002+01:002019-10-17T11:24:32.195+01:00Stress testing CPU temperaturesStress testing CPU temperatures is not exactly straight forward. CPU designs vary from CPU to CPU and each have their own strengths and weaknesses in cache design, integer maths, floating point math, bit-wise logical operations and branch prediction to name but a few. I've been asked several times about the "best" CPU stressor method in <a href="https://kernel.ubuntu.com/~cking/stress-ng/">stress-ng</a> to use to make a CPU run hot.<br />
<br />
As an experiment I ran all the CPU stressor methods in stress-ng for 60 seconds across a range of devices, from small ARM based Raspberry Pi 2 and 3 to much larger Xeon desktop servers just to see how hot CPUs get. The thermal measurements were based on the most relevant thermal zones, for example, on x86 this is the CPU package thermal zone. In between each stress run 60 seconds of idle time was added to allow the CPU to cool.<br />
<br />
Below are the results:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnnR56szAD3N2ehfFOpV9TIBXsh2y29_YG60tn7RykArmzaqzBixb7ew3Dw00qrrjHcSzl5MctdwO4x9E7D6TPTJNwpCRvIqSMlz_Lpx599FN5xwoM8FHdP4Wzt5eBVQRhp4RhTjNFo0WM/s1600/stress-ng-cpu-temperatures.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1468" data-original-width="661" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnnR56szAD3N2ehfFOpV9TIBXsh2y29_YG60tn7RykArmzaqzBixb7ew3Dw00qrrjHcSzl5MctdwO4x9E7D6TPTJNwpCRvIqSMlz_Lpx599FN5xwoM8FHdP4Wzt5eBVQRhp4RhTjNFo0WM/s1600/stress-ng-cpu-temperatures.png" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
As one can see, quite a mixed set of results and it is hard to recommend any specific CPU stressor method as the "best" across a range of CPUs. It does appear that the mix of 64 bit integer and floating point cpu stress methods do seem to be generally rather good for making most CPUs run hot.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
With this is mind, I think we can conclude there is no such thing as a perfect way to make a CPU run hot as it is very architecture dependant. Fortunately the stress-ng CPU stressor has a large suite of methods to exercise the CPU in different ways, so there should be a good stressor somewhere in that collection to max out your CPU. Knowing which one is the tricky part(!)</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com1tag:blogger.com,1999:blog-1016709139053396535.post-91083832121491578442019-09-10T10:47:00.002+01:002019-09-10T10:49:26.522+01:00Boot speed improvements for Ubuntu 19.10 Eoan ErmineThe early boot requires loading and decompressing the kernel and initramfs from the boot storage device. This speed is dependent on several factors, speed of loading an image from the boot device, the CPU and memory/cache speed for decompression and the compression type.<br />
<br />
Generally speaking, the smallest (best) compression takes longer to decompress due to the extra complexity in the compression algorithm. Thus we have a trade-off between load time vs decompression time.<br />
<br />
For slow rotational media (such as a 5400 RPM HDD) with a slow CPU the loading time can be the dominant factor. For faster devices (such as a SSD) with a slow CPU, decompression time may be the dominate factor. For devices with fast 7200-10000 RPM HDDs with fast CPUs, the time to seek to the data starts to dominate the load time, so load times for different compressed kernel sizes is only slightly different in load time.<br />
<br />
The Ubuntu kernel team ran several experiments benchmarking several x86 configurations using the x86 TSC (Time Stamp Counter) to measure kernel load and decompression time for 6 different compression types: BZIP2, GZIP, LZ4, LZMA, LZMO and XZ. BZIP2, LZMA and XZ are slow to decompress so they got ruled out very quickly from further tests.<br />
<br />
In compression size, GZIP produces the smallest compressed kernel size, followed by LZO (~16% larger) and LZ4 (~25% larger). With decompression time, LZ4 is over 7 times faster than GZIP, and LZO being ~1.25 times faster then GZIP on x86. <br />
<br />
In absolute wall-clock times, the following kernel load and decompress results were observed:<br />
<br />
Lenovo x220 laptop, 5400 RPM HDD:<br />
LZ4 best, 0.24s faster than the GZIP total time of 1.57s<br />
<br />
Lenovo x220 laptop, SSD:<br />
LZ4 best, 0.29s faster than the GZIP total time of 0.87s<br />
<br />
Xeon 8 thread desktop with 7200 RPM HDD:<br />
LZ4 best, 0.05s faster than the GZIP total time of 0.32s<br />
<br />
VM on a Xeon 8 thread desktop host with SSD RAID ZFD backing store:<br />
LZ4 best, 0.05s faster than the GZIP total time of 0.24s<br />
<br />
Even with slow spinning media
and a slow CPU, the longer load time of the LZ4 kernel is overcome by
the far faster decompression time.
As media gets faster, the load time difference between GZIP, LZ4 and
LZO diminishes and the decompression time becomes the dominant speed
factor with LZ4 the clear winner.<br />
<br />
For Ubuntu 19.10 Eoan Ermine, LZ4 will be the default decompression for x86, ppc64el and s390 kernels and for the initramfs too.<br />
<br />
<b>References:</b><br />
Analysis: <a href="https://kernel.ubuntu.com/~cking/boot-speed-eoan-5.3/kernel-compression-method.txt">https://kernel.ubuntu.com/~cking/boot-speed-eoan-5.3/kernel-compression-method.txt</a><br />
Data: <a href="https://kernel.ubuntu.com/~cking/boot-speed-eoan-5.3/boot-speed-compression-5.3-rc4.ods">https://kernel.ubuntu.com/~cking/boot-speed-eoan-5.3/boot-speed-compression-5.3-rc4.ods </a><br />
<br />Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com2tag:blogger.com,1999:blog-1016709139053396535.post-39133012262651341332019-08-13T12:14:00.001+01:002019-08-13T12:14:44.661+01:00Monitoring page faults with faultstat<span class="inline_editor_value" id="__w2_wurEWHbL4_answer_content"><span class="ui_qtext_rendered_qtext">Whenever a process accesses a virtual address where there isn't currently a physical page mapped into its process space then a <a href="https://en.wikipedia.org/wiki/Page_fault">page fault </a>occurs. This causes an interrupt so that the kernel can handle the page fault. </span></span><br />
<br />
<span class="inline_editor_value" id="__w2_wurEWHbL4_answer_content"><span class="ui_qtext_rendered_qtext">A minor page fault occurs when the kernel can </span></span><span class="inline_editor_value" id="__w2_wurEWHbL4_answer_content"><span class="ui_qtext_rendered_qtext"><span class="inline_editor_value" id="__w2_wurEWHbL4_answer_content"><span class="ui_qtext_rendered_qtext">successfully map a physically resident page for the faulted user-space virtual address (for example, accessing a memory resident page that is already shared by other processes). Major page faults occur when accessing a page that has been swapped out or accessing a file backed memory mapped page that is not resident in memory.</span></span></span></span><br />
<br />
<span class="inline_editor_value" id="__w2_wurEWHbL4_answer_content"><span class="ui_qtext_rendered_qtext"><span class="inline_editor_value" id="__w2_wurEWHbL4_answer_content"><span class="ui_qtext_rendered_qtext">Page faults incur <a href="https://en.wikipedia.org/wiki/Latency_(engineering)">latency</a> in the running of a program, major faults especially so because of the delay of loading pages in from a storage device.</span></span></span></span><br />
<span class="inline_editor_value" id="__w2_wurEWHbL4_answer_content"><span class="ui_qtext_rendered_qtext"><span class="inline_editor_value" id="__w2_wurEWHbL4_answer_content"><span class="ui_qtext_rendered_qtext"><br /></span></span></span></span>
The <a href="https://kernel.ubuntu.com/~cking/faultstat/">faultstat</a> tool allows one to easily monitor page fault activity allowing one to find the most active page faulting processes. Running faultstat with no options will dump the page fault statistics of all processes sorted in major+minor page fault order.<br />
<br />
Faultstat also has a "<a href="https://en.wikipedia.org/wiki/Top_(software)">top</a>" like mode, inoking it with the -T option will display the top page faulting processes again in major+minor page fault order.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhanW6thZ7eHh6YIwgA8eOdzKMgh-0NqnFbL_1Fp8kugd6YwC-m4vFlJ7SyOrJZ-_E9ohxjHPBJgRBFthug2jOv_6XupYDKyas4K2EM35GpHQq5M_Qtm4_BA4SbO0CZ69265MW3A0NYYK-S/s1600/faultstat.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="507" data-original-width="923" height="348" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhanW6thZ7eHh6YIwgA8eOdzKMgh-0NqnFbL_1Fp8kugd6YwC-m4vFlJ7SyOrJZ-_E9ohxjHPBJgRBFthug2jOv_6XupYDKyas4K2EM35GpHQq5M_Qtm4_BA4SbO0CZ69265MW3A0NYYK-S/s640/faultstat.png" width="640" /></a></div>
<br />
The Major and Minor columns show the respective major and minor page faults. The +Major and +Minor columns show the recent increase of page faults. The Swap column shows the swap size of the process in pages. <br />
<br />
Pressing the 's' key will switch through the sort order. Pressing the 'a' key will add an arrow annotation showing page fault growth change. The 't' key will toggle between cumulative major/minor page total to current change in major/minor faults.<br />
<br />
The faultstat tool has just landed in Ubuntu Eoan and can also be installed as a <a href="https://snapcraft.io/faultstat">snap</a>. The source can is available on <a href="https://github.com/ColinIanKing/faultstat">github</a>. <br />
<span class="inline_editor_value" id="__w2_wurEWHbL4_answer_content"><span class="ui_qtext_rendered_qtext"><span class="inline_editor_value" id="__w2_wurEWHbL4_answer_content"><span class="ui_qtext_rendered_qtext"><br /></span></span></span></span>Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-46886838794793155592019-06-08T17:26:00.000+01:002019-06-08T17:26:42.375+01:00Working towards stress-ng 0.10.00<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKjd80To9csuHt1AdIseyf4HFO9fHbgo2RbuNPO1SBQ3S6zQKDMwftdbv_3Y-9XiibXN2cYxMfjLphHphJ7hrEJFNjJju4-YtJll9gLFnvIburaajNH5-5dAus_rINtBFCbfnBJjn-ON_Q/s1600/stress-ng-large.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="460" data-original-width="388" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKjd80To9csuHt1AdIseyf4HFO9fHbgo2RbuNPO1SBQ3S6zQKDMwftdbv_3Y-9XiibXN2cYxMfjLphHphJ7hrEJFNjJju4-YtJll9gLFnvIburaajNH5-5dAus_rINtBFCbfnBJjn-ON_Q/s200/stress-ng-large.png" width="168" /></a>Over the past 9+ months I've been cleaning up <a href="https://wiki.ubuntu.com/Kernel/Reference/stress-ng">stress-ng </a>in preparation for a V0.10.00 release. Stress-ng is a portable Linux/UNIX Swiss army knife of micro-benchmarking kernel stress tests.<br />
<br />
The Ubuntu kernel team uses stress-ng for kernel regression testing in several ways:<br />
<ul>
<li>Checking that the kernel does not crash when being stressed tested</li>
<li>Performance (bogo-op throughput) regression checks</li>
<li>Power consumption regression checks</li>
<li>Core CPU Thermal regression checks </li>
</ul>
The wide range of micro benchmarks in stress-ng allow us to keep track of a range of metrics so that we can catch regressions.<br />
<br />
I've tried to focus on several aspects of stress-ng over the last last development cycle:<br />
<ul>
<li>Improve per-stressor modularization. A lot of code has been moved from the core of stress-ng back into each stress test.</li>
<li>Clean up a lot of corner case bugs found when we've been testing stress-ng in production. We exercise stress-ng on a lot of hardware and in various cloud instances, so we find occasional bugs in stress-ng.</li>
<li>Improve usability, for example, adding bash command completion.</li>
<li>Improve portability (various kernels, compilers and C libraries). It really builds on runs on a *lot* of Linux/UNIX/POSIX systems.</li>
<li>Improve kernel test coverage. Try to exercise more kernel core functionality and reach parts other tests don't yet reach. </li>
</ul>
Over the past several days I've been running various versions of stress-ng on a gcov enabled 5.0 kernel to measure kernel test coverage with stress-ng. As shown below, the tool has been slowly gaining more core kernel coverage over time:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbMMLU2HvCN6cH7TJ2qE1bPecvbo0O-D_5tXmpEohBrId5Drqjfo59Us8sRiPR6PtQIQ0270rb6nSAGKtDt4OuuAUsT8C2zHcb8TOD9P2EF2n7vVFYeunM_0ecI4pH3xUIbx1Gecf3xpnT/s1600/stress-ng-kernel-5.0-coverage.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="303" data-original-width="585" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbMMLU2HvCN6cH7TJ2qE1bPecvbo0O-D_5tXmpEohBrId5Drqjfo59Us8sRiPR6PtQIQ0270rb6nSAGKtDt4OuuAUsT8C2zHcb8TOD9P2EF2n7vVFYeunM_0ecI4pH3xUIbx1Gecf3xpnT/s1600/stress-ng-kernel-5.0-coverage.png" /></a></div>
With the use of gcov + lcov, I can observe where stress-ng is not currently exercising the kernel and this allows me to devise stress tests to touch these un-exercised parts. The tool has a history of tripping kernel bugs, so I'm quite pleased it has helped us to find corners of the kernel that needed improving.<br />
<br />
This week I released <a href="https://github.com/ColinIanKing/stress-ng">V0.09.59 of stress-ng</a>. Apart from the usual sets of clean up changes and bug fixes, this new release now incorporates bash command line completion to make it easier to use. Once the 5.2 Linux kernel has been released and I'm satisfied that stress-ng covers new 5.2 features I will probably be releasing V0.10.00. This will be a major release milestone now that stress-ng has realized most of my original design goals.Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-67032756357593128572019-01-05T19:39:00.000+00:002019-01-05T19:39:16.304+00:00Kernel commits with "Fixes" Tag (revisited)Last year I <a href="http://smackerelofopinion.blogspot.com/2018/03/kernel-commits-with-fixes-tag.html">wrote about kernel commits</a> that are tagged with the "Fixes" tag. Kernel developers use
the "Fixes" tag on a bug fix commit to reference an older commit that originally
introduced the bug. The adoption of the tag has been steadily increasing since v3.12 of the kernel:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzLgHtWDXm5Io7_fOK6tLBW4DL4Yy0aC0fPD2-XU7zTKivRHM9gIdECsnWexmATdtSxdYPgDIOniBvDoDKTxXY2XKDuSOGtVlFU8aucPt7iQWdOJWblbtNUgj7b_1_OZ9zibcheJt6Utyd/s1600/commits-tag.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzLgHtWDXm5Io7_fOK6tLBW4DL4Yy0aC0fPD2-XU7zTKivRHM9gIdECsnWexmATdtSxdYPgDIOniBvDoDKTxXY2XKDuSOGtVlFU8aucPt7iQWdOJWblbtNUgj7b_1_OZ9zibcheJt6Utyd/s1600/commits-tag.png" /></a></div>
The red line shows the number of commits per release of the kernel, and the blue line shows the number of commits that contain a "Fixes" tag.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcnefhIUMfpBtCT92aHTsmfAWCVClZJAZGiDYrDe8q9rTEUP9rtXGZKDcYltY56BVAbaXEDclz9kXbM6vu8QvV0VLmvYVOmjyxBB4gHjHubmwPfPSH76nIKrQVJkkD2qiTetY4ssTBRDWd/s1600/fixes-tag-percent.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcnefhIUMfpBtCT92aHTsmfAWCVClZJAZGiDYrDe8q9rTEUP9rtXGZKDcYltY56BVAbaXEDclz9kXbM6vu8QvV0VLmvYVOmjyxBB4gHjHubmwPfPSH76nIKrQVJkkD2qiTetY4ssTBRDWd/s1600/fixes-tag-percent.png" /></a></div>
In terms of % of commits that contain the "Fixes" tag, one can see it has been steadily increasing since v3.12 and almost 12.5% of kernel commits in v4.20 are tagged this way.<br />
<br />
The fixes tag contains the commit SHA of the commit that was fixed, so one can look up the date of the fix and of the commit being fixed and determine the time taken to fix a bug. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNgxKR-41P1l5aTDq34pFSdn6NdTVNSLa1cCE5PLF8wjaM_TXrDA2pkNqMs1EV66oSdW54gChpQICxuJmKe4ksOTTEfZchfKs9_sZO0_fFIexLAv-405UYUFOf0aMmOUQuBYa5HEGigzlC/s1600/fixes-distribution.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNgxKR-41P1l5aTDq34pFSdn6NdTVNSLa1cCE5PLF8wjaM_TXrDA2pkNqMs1EV66oSdW54gChpQICxuJmKe4ksOTTEfZchfKs9_sZO0_fFIexLAv-405UYUFOf0aMmOUQuBYa5HEGigzlC/s1600/fixes-distribution.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
As one can see, a lot of issues get fixed on the first few hundred days, and some bugs take years to get fixed. Zooming into the first hundred days of fixes the distribution looks like:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtH8W7-OiYmSPUhlKiNKv4qybsx6WNdcsgcUGf7O76UYlQjT3YKs0Gint3I3eBv1_1RKs5tfOZTVBOQ7SEMGMKHn9mstPn7sZoOXGPcEXdplElTqBDuuzdiiEV_hM9IfKPyvjtFvseXZuQ/s1600/fixes-distribution-100.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="340" data-original-width="605" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtH8W7-OiYmSPUhlKiNKv4qybsx6WNdcsgcUGf7O76UYlQjT3YKs0Gint3I3eBv1_1RKs5tfOZTVBOQ7SEMGMKHn9mstPn7sZoOXGPcEXdplElTqBDuuzdiiEV_hM9IfKPyvjtFvseXZuQ/s1600/fixes-distribution-100.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><br /></td></tr>
</tbody></table>
..the modal point is at day 4, I suspect these are issues that get found quickly when commits land in linux-next and are found in early testing, integration builds and static analysis.<br />
<br />
Out of the thousands of "Fixes" tagged commits and the time to fix an issue one can determine how long it takes to fix a specific percentage of the bugs:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGNR_4c9rSFlpysGjnx43jVipICD4uEZiaRxmT58EkjtG4dD12UQa9f9x47zY_4SoF4YRuhT6zlApHOyKYF9v2b-lyBmNEM2IRewPgNwZDG00WLm-8_MrJsBSQU7uYiwGQxkW8Xyv7ssF3/s1600/fixes-total.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGNR_4c9rSFlpysGjnx43jVipICD4uEZiaRxmT58EkjtG4dD12UQa9f9x47zY_4SoF4YRuhT6zlApHOyKYF9v2b-lyBmNEM2IRewPgNwZDG00WLm-8_MrJsBSQU7uYiwGQxkW8Xyv7ssF3/s1600/fixes-total.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
In the graph above, 50% of fixes are made within 151 days of the original commit, ~69% of fixes are made within a year of the original commit and ~82% of fixes are made within 2 years. The long tail indicates that there are some bugs that take a while to be found and fixed, the final 10% of bugs take more than 3.5 years to be found and fixed.<br />
<br />
Comparing the time to fix issues for kernel versions v4.0, v4.10 and v4.20 for bugs that are fixed in less than 50 days we have:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFx9XeuPf9UlR8uL38l9QONnC_IZOyxp0GPiJSxx-vyHUyqXxuoqnm6lfAVRbNQDEz1DKrdx3ik5JOmHtzBTfP4VPwC2di0ksLcX_tD6C_XnCc3i8x0-UZ632NtGbzjoI7lfjlLcT-g8v9/s1600/fixes-vs-kernel-50.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="459" data-original-width="582" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFx9XeuPf9UlR8uL38l9QONnC_IZOyxp0GPiJSxx-vyHUyqXxuoqnm6lfAVRbNQDEz1DKrdx3ik5JOmHtzBTfP4VPwC2di0ksLcX_tD6C_XnCc3i8x0-UZ632NtGbzjoI7lfjlLcT-g8v9/s1600/fixes-vs-kernel-50.png" /></a></div>
<br />
... the trends are similar, however it is worth noting that more bugs are getting found and fixed a little faster in v4.10 and v4.20 than v4.0. It will be interesting to see how these trends develop over the next few years.Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-51183348631498470532018-12-12T09:55:00.002+00:002018-12-12T09:55:28.002+00:00Linux I/O Schedulers<div class="line874">
The Linux kernel I/O schedulers attempt to balance the need to get the best possible I/O performance while also trying to ensure the I/O requests are "fairly" shared among the I/O consumers. There are several I/O schedulers in Linux, each try to solve the I/O scheduling issues using different mechanisms/heuristics and each has their own set of strengths and weaknesses.</div>
<div class="line874">
<br /></div>
<div class="line874">
For traditional spinning media it makes sense to try and order I/O operations so that they are close together to reduce read/write head movement and hence decrease latency. However, this reordering means that some I/O requests may get delayed, and the usual solution is to schedule these delayed requests after a specific time. Faster non-volatile memory devices can generally handle random I/O requests very easily and hence do not require reordering.</div>
<div class="line874">
<br /></div>
<div class="line874">
Balancing the fairness is also an interesting issue. A greedy I/O consumer should not block other I/O consumers and there are various heuristics used to determine the fair sharing of I/O. Generally, the more complex and "fairer" the solution the more compute is required, so selecting a very fair I/O scheduler with a fast I/O device and a slow CPU may not necessarily perform as well as a simpler I/O scheduler.</div>
<div class="line874">
<br /></div>
<div class="line874">
Finally, the types of I/O patterns on the I/O devices influence the I/O scheduler choice, for example, mixed random read/writes vs mainly sequential reads and occasional random writes.</div>
<div class="line874">
<br /></div>
<div class="line874">
Because of the mix of requirements, there is no such thing as a perfect all round I/O scheduler. The defaults being used are chosen to be a good best choice for the general user, however, this may not match everyone's needs. To clarify the choices, the <a href="https://wiki.ubuntu.com/Kernel/">Ubuntu Kernel Team has</a> provided a <a href="https://wiki.ubuntu.com/Kernel/Reference/IOSchedulers">Wiki page </a>describing the choices and how to select and tune the various I/O schedulers. Caveat emptor applies, these are just guidelines and should be used as a starting point to finding the best I/O scheduler for your particular need.</div>
Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com1tag:blogger.com,1999:blog-1016709139053396535.post-79893298437374410272018-12-01T12:43:00.000+00:002018-12-01T12:47:45.040+00:00New features in Forkstat<a href="http://kernel.ubuntu.com/~cking/forkstat/">Forkstat </a>is a simple utility I wrote a while ago that can trace process activity using the rather useful Linux <a href="https://www.kernel.org/doc/Documentation/connector/connector.txt">NETLINK_CONNECTOR</a> API. Recently I have added two extra features that may be of interest:<br />
<br />
1. Improved output using some UTF-8 glyphs. These are used to show process parent/child relationships and various process events, such as termination, core dumping and renaming. Use the new -g (glyph) option to enable this mode. For example:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrKJzdkgo8ZTrfFivqROk5VW0FtM4W84oyzFT0jguSU14Gng19FGAGGyLq3pzsF3xyhsyL3x5E-zNju-SugMfuoZ_OxdJ-9odXxXXUOrHMHTncK7wXYr9E0zM2w_dQvdjrtbjSXDhSwopl/s1600/forkstat-1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="289" data-original-width="392" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrKJzdkgo8ZTrfFivqROk5VW0FtM4W84oyzFT0jguSU14Gng19FGAGGyLq3pzsF3xyhsyL3x5E-zNju-SugMfuoZ_OxdJ-9odXxXXUOrHMHTncK7wXYr9E0zM2w_dQvdjrtbjSXDhSwopl/s1600/forkstat-1.png" /></a></div>
<br />
In the above example, the program "wobble" was started and forks off a child process. The parent then renames itself to wibble (indicated by a turning arrow). The child then segfaults and generates a core dump (indicated by a skull and crossbones), triggering apport to investigate the crash. After this, we observe NetworkManager creating a thread that runs for a very short period of time. This kind of activity is normally impossible to spot while running conventions tools such as ps or top.<br />
<br />
2. By default, forkstat will show the process name using the contents of /proc/$PID/cmdline. The new -c option allows one to instead use the 16 character task "comm" field, and this can be helpful for spotting process name changes on PROC_EVENT_COMM events.<br />
<br />
These are small changes, but I think they make forkstat more useful. The updated forkstat will be available in Ubuntu 19.04 "Disco Dingo".Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-26305601507847693072018-11-22T12:14:00.001+00:002018-11-22T12:37:18.482+00:00 High-level tracing with bpftraceBpftrace is a new high-level tracing language for Linux using the <a href="https://lwn.net/Articles/599755/">extended Berkeley packet filter</a> (eBPF). It is a very powerful and flexible tracing front-end that enables systems to be analyzed much like <a href="https://en.wikipedia.org/wiki/DTrace">DTrace</a>.<br />
<br />
The bpftrace tool is now installable as a <a href="https://snapcraft.io/bpftrace">snap</a>. From the command line one can install it and enable it to use system tracing as follows:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> sudo snap install bpftrace
sudo snap connect bpftrace:system-trace
</code></pre>
<br />
To illustrate the power of bpftrace, here are some simple one-liners:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> # trace openat() system calls
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%d %s %s\n", pid, comm, str(args->filename)); }'
Attaching 1 probe...
1080 irqbalance /proc/interrupts
1080 irqbalance /proc/stat
2255 dmesg /etc/ld.so.cache
2255 dmesg /lib/x86_64-linux-gnu/libtinfo.so.5
2255 dmesg /lib/x86_64-linux-gnu/librt.so.1
2255 dmesg /lib/x86_64-linux-gnu/libc.so.6
2255 dmesg /lib/x86_64-linux-gnu/libpthread.so.0
2255 dmesg /usr/lib/locale/locale-archive
2255 dmesg /lib/terminfo/l/linux
2255 dmesg /home/king/.config/terminal-colors.d
2255 dmesg /etc/terminal-colors.d
2255 dmesg /dev/kmsg
2255 dmesg /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
</code></pre>
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> # count system calls using tracepoints:
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[probe] = count(); }'
@[tracepoint:syscalls:sys_enter_getsockname]: 1
@[tracepoint:syscalls:sys_enter_kill]: 1
@[tracepoint:syscalls:sys_enter_prctl]: 1
@[tracepoint:syscalls:sys_enter_epoll_wait]: 1
@[tracepoint:syscalls:sys_enter_signalfd4]: 2
@[tracepoint:syscalls:sys_enter_utimensat]: 2
@[tracepoint:syscalls:sys_enter_set_robust_list]: 2
@[tracepoint:syscalls:sys_enter_poll]: 2
@[tracepoint:syscalls:sys_enter_socket]: 3
@[tracepoint:syscalls:sys_enter_getrandom]: 3
@[tracepoint:syscalls:sys_enter_setsockopt]: 3
...
</code></pre>
<br />
Note that it is recommended to use bpftrace with Linux 4.9 or higher.<br />
<br />
The bpftrace github project page has an excellent <a href="https://github.com/iovisor/bpftrace/blob/master/README.md">README guide </a>with some worked examples and is a very good place to start. There is also a very useful <a href="https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md">reference guide</a> and <a href="https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md">one-liner tutorial</a> too.<br />
<br />
If you have any useful btftrace one-liners, it would be great to share them. This is an amazingly powerful tool, and it would be interesting to see how it will be used.<br />
<br />
<a href="https://snapcraft.io/bpftrace" title="Get it from the Snap Store">
<img alt="" src="https://snapcraft.io/static/images/badges/en/snap-store-black.svg" />
</a>Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-22596764089788212742018-10-03T10:17:00.000+01:002018-10-03T10:17:13.337+01:00Static Analysis Trends on Linux NextI've been running static analysis using CoverityScan on <a href="https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/?h=next-20181002">linux-next</a> for 2 years with the aim to find bugs (and try to fix some) before they are merged into Linux. I have also been gathering the defect count data and tracking the defect trends:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLcFg50mEfKPmdoAZmXFsaQjgdLrEiTxDgYhMsxV83NXiTVrgHzlokbpBg1YUzMNbJK9sdjWBJ93314gShE6VFTgyNXmvL92PuK4IqobRAZgWqB4cROWGii2zNmjorKq9p4YBncMASS2bc/s1600/linux-next-trend-2018.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="436" data-original-width="625" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLcFg50mEfKPmdoAZmXFsaQjgdLrEiTxDgYhMsxV83NXiTVrgHzlokbpBg1YUzMNbJK9sdjWBJ93314gShE6VFTgyNXmvL92PuK4IqobRAZgWqB4cROWGii2zNmjorKq9p4YBncMASS2bc/s1600/linux-next-trend-2018.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
As one can see from above, CoverityScan has found a considerable amount of defects and these are being steadily fixed by the Linux developer community. The encouraging fact is that the outstanding issues are reducing over time. Some of the spikes in the data are because of changes in the analysis that I'm running (e.g. getting more coverage), but even so, one can see a definite trend downwards in the total defects in the Kernel.<br />
<br />
With static analysis, some of these reported defects are false positives or corner cases that are in fact impossible to occur in real life and I am slowly working through these and annotating them so they don't get reported in the defect count.<br />
<br />
It must be also noted that over these two years the kernel has grown from around 14.6 million to 17.1 million lines of code so the defect count has dropped from 1 defect in every ~2100 lines to 1 defect in every ~3000 lines over the past 2 years. All in all, it is a remarkable improvement for such a large and complex codebase that is growing in size at such rate.Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com1tag:blogger.com,1999:blog-1016709139053396535.post-72825586530919430222018-07-16T12:31:00.002+01:002018-07-16T13:22:31.700+01:00Comparing Latencies and Power consumption with various CPU schedulersThe low-latency kernel offering with Ubuntu provides a kernel tuned for low-latency environments using low-latency kernel configuration options. The x86 kernels by default run with the Intel-Pstate CPU scheduler set to run with the powersave scaling governor biased towards power efficiency.<br />
<br />
While power efficiency is fine for most use-cases, it can introduce latencies due to the fact that the CPU can be running at a low frequency to save power and also switching from a deep C state when idle to a higher C state when servicing an event can also increase on latencies.<br />
<br />
In a somewhat contrived experiment, I rigged up an i7-3770 to collect latency timings of clock_nanosleep() wake-ups with timer event coalescing disabled (timer_slack set to zero) over 60 seconds across a range of CPU scheduler and governor settings on a 4.15 low-latency kernel. This can be achieved using stress-ng, for example:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> sudo stress-ng --cyclic 1 --cyclic-dist 100 –cyclic-sleep=10000 --cpu 1 -l 0 -v \
--cyclic-policy rr --cyclic-method clock_ns --cpu 0 -t 60 --timer-slack 0
</code></pre>
<br />
..the above runs a cyclic measurement collecting latency counts in 100ns buckets with a clock_nanosecond wakeup interval of 10,000 nanoseconds with zero % load CPU stressor and timer slack set to 0 nanoseconds. This dumps latency distribution stats that can be plotted to see where the modal latency points occur and the latency characteristics of the CPU scheduler.<br />
<br />
I also used powerstat to measure the power consumed by the CPU package over a 60 second interval. Measurements for the Intel-Pstate CPU scheduler [performance, powersave] and the ACPI CPU scheduler (intel_pstate=disabled) [performance, powersave, conservative and ondemand] were taken for 1,000,000 down to 10,000 nanosecond timer delays.<br />
<br />
<h3>
<b>1,000,000 nanosecond timer delays (1 millisecond) </b></h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiS73ICHjALqS4CTCCdblASioO9c7_b0rbtTRN00XF8zrPEIxeEsjqpPdt0OT9GzLhCGFT3cya8qOAMqyfAuNiJqvoFBbq6valffr4Jvwcu60SUqg2AMCXtPEcxVAiRcUAVwGXo2wKBlGi1/s1600/1000000power.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiS73ICHjALqS4CTCCdblASioO9c7_b0rbtTRN00XF8zrPEIxeEsjqpPdt0OT9GzLhCGFT3cya8qOAMqyfAuNiJqvoFBbq6valffr4Jvwcu60SUqg2AMCXtPEcxVAiRcUAVwGXo2wKBlGi1/s1600/1000000power.png" /></a></div>
Strangely the powersave Intel-Pstate is using the most power (not what I expected).<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCQTFXVrlm7x7pIM81vfEGhyphenhyphenKTt6t_ccStINUxGNvqHaBDdh3PQBJFfiQChcf1bpXSQifeKDuE4GM_96IUvVi-XI2JBtZNW_nSLPMh2L_lNUsAiOUgYDO6nohWhl83JhQaU4aXT9ceM54b/s1600/latency1000000.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCQTFXVrlm7x7pIM81vfEGhyphenhyphenKTt6t_ccStINUxGNvqHaBDdh3PQBJFfiQChcf1bpXSQifeKDuE4GM_96IUvVi-XI2JBtZNW_nSLPMh2L_lNUsAiOUgYDO6nohWhl83JhQaU4aXT9ceM54b/s1600/latency1000000.png" /></a> The ACPI CPU scheduler in performance mode has the best latency distribution followed by the Intel-Pstate CPU scheduler also in performance mode.<br />
<br />
<h3>
<b>100,000 nanosecond timer delays (100 microseconds)</b></h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgr7jLUaey2CXjH9BIOVFeXfhpO_TRDkUd6jsXJTtELOw33HMzD0DKnHmg4laWTm67T20ez2HwvqSor-KayfiWcvnhpmYYuYmGPAL1Esv173sGRySTRhsweua1g08nHWvDFf52o9i-gqyco/s1600/100000power.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgr7jLUaey2CXjH9BIOVFeXfhpO_TRDkUd6jsXJTtELOw33HMzD0DKnHmg4laWTm67T20ez2HwvqSor-KayfiWcvnhpmYYuYmGPAL1Esv173sGRySTRhsweua1g08nHWvDFf52o9i-gqyco/s1600/100000power.png" /></a></div>
Note that Intel-Pstate performance consumes the most power...<br />
<div class="separator" style="clear: both; text-align: center;">
<b>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXlLyYcIysSiAI2U1YeEDK1iF0jhh6m8muIL7MDCnurEz5OdiFb0wpfhrvYG5rtqxaCjLmBfcH1Ol1VQ3NDBvAEPM_lMp8KWgrWnWVV2HngFbV9UXDJhqWzn2cvIafWm1piMD2tyJzR6oL/s1600/latency100000.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXlLyYcIysSiAI2U1YeEDK1iF0jhh6m8muIL7MDCnurEz5OdiFb0wpfhrvYG5rtqxaCjLmBfcH1Ol1VQ3NDBvAEPM_lMp8KWgrWnWVV2HngFbV9UXDJhqWzn2cvIafWm1piMD2tyJzR6oL/s1600/latency100000.png" /></a></b></div>
...and also has the most responsive low-latency distribution.<br />
<br />
<h3>
<b>10,000 nanosecond timer delays (10 microseconds)</b></h3>
<h3>
<b><div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbTTkXm0xqbQwDlY5N4vC34slq4LNSxXAez1NjPe3bX1U-LGcnUGqgpW0oFf7MbZWjWjCipoFtFtOlbHu22qtwStHGe7VhdzuuLNZl-5-ZaIiMBvlAd6Ea7lqlCOp49Db0YctzMAxNsLCu/s1600/10000power.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbTTkXm0xqbQwDlY5N4vC34slq4LNSxXAez1NjPe3bX1U-LGcnUGqgpW0oFf7MbZWjWjCipoFtFtOlbHu22qtwStHGe7VhdzuuLNZl-5-ZaIiMBvlAd6Ea7lqlCOp49Db0YctzMAxNsLCu/s1600/10000power.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5Xm4BH2ObrIwybTUmv6mHm-SUU9gq9_L52j26ofB22jSDxTnss0Kh9y3wD9FHcvG6xnBISGAWPQSdF7-h8usHwzqoR9pM5MRMucNYlYGX3ESLUFv_exmwVXDuJmZCZHkpN-lX5cSJpkBw/s1600/latency10000.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5Xm4BH2ObrIwybTUmv6mHm-SUU9gq9_L52j26ofB22jSDxTnss0Kh9y3wD9FHcvG6xnBISGAWPQSdF7-h8usHwzqoR9pM5MRMucNYlYGX3ESLUFv_exmwVXDuJmZCZHkpN-lX5cSJpkBw/s1600/latency10000.png" /></a></div>
</b></h3>
In this scenario, the ACPI CPU scheduler in performance mode was consuming the most power and had the best latency distribution.<br />
<br />
It is clear that the best latency responses occur when a CPU scheduler is running in performance mode and this consumes a little more power than other CPU scheduler modes. However, it is not clear which CPU scheduler (Intel-Pstate or ACPI) is best in specific use-cases.<br />
<br />
The conclusion is rather obvious; but needs to be stated. For best low-latency response, set the CPU governor to the <b>performance</b> mode at the cost of <b>higher power consumption</b>. Depending on the use-case, the extra power cost is probably worth the improved latency response.<br />
<br />
As mentioned earlier, this is a somewhat contrived experiment, only one CPU was being exercised with a predictable timer wakeup. A more interesting test would be with data handling, such as incoming packet handling over ethernet at different rates; I will probably experiment with that if and when I get more time. Since this was a synthetic test using stress-ng, it does not represent real world low-latency scenarios, however, it may be worth exploring CPU scheduler settings to tune a low-latency configuration rather than relying on the default CPU scheduler setting.Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-89615594407601475112018-03-23T00:26:00.000+00:002018-03-23T00:26:02.810+00:00Kernel Commits with "Fixes" tagOver the past 5 years there has been a steady increase in the number of kernel bug fix commits that use the "Fixes" tag. Kernel developers use this annotation on a commit to reference an older commit that originally introduced the bug, which is obviously very useful for bug tracking purposes. What is interesting is that there has been a steady take-up of developers using this annotation:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFgQW94Ahcs3yBr3Ues0JAVgL5E9gLKUfYnCpPwooDDFIWnESGo6Ag-7SMyhO4vPnH8pBF2wF3so41IA6nZCEp1vst8uHBNlzFws4NQB66Uj_BEy8y5F-Uh4NbaR-8tVYURmh5XiefUoso/s1600/fixes-tag.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="376" data-original-width="603" height="396" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFgQW94Ahcs3yBr3Ues0JAVgL5E9gLKUfYnCpPwooDDFIWnESGo6Ag-7SMyhO4vPnH8pBF2wF3so41IA6nZCEp1vst8uHBNlzFws4NQB66Uj_BEy8y5F-Uh4NbaR-8tVYURmh5XiefUoso/s640/fixes-tag.png" width="640" /></a></div>
With the 4.15 release, 1859 of the 16223 commits (11.5%) were tagged as "Fixes", so that's a fair amount of work going into bug fixing. I suspect there are more commits that are bug fixes, but aren't using the "Fixes" tag, so it's hard to tell for certain how many commits are fixes without doing deeper analysis. Probably over time this tag will be widely adopted for all bug fixes and the trend line will level out and we will have a better idea of the proportion of commits per release that are just devoted to fixing issues. Let's see how this looks in another 5 years time, I'll keep you posted!Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com0tag:blogger.com,1999:blog-1016709139053396535.post-90376982792199536602018-02-21T16:46:00.005+00:002018-02-21T16:46:43.401+00:00Linux Kernel Module GrowthThe Linux kernel grows at an amazing pace, each kernel release adds more functionality, more drivers and hence more kernel modules. I recently wondered what the trend was for kernel module growth per release, so I performed module builds on kernels v2.6.24 through to v4.16-rc2 for x86-64 to get a better idea of growth rates:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaONn3IfrqTjF8TshFFQFfJYas_fwAFs80356f6k-JGu-ZdQcKHHl9zRSN8-4qAlc-A9At4EbX88lpmIE_nfE2hgpT3LEcsdgpkuTzAjp9pMrisWzYN-5S_dmJfHIGEXqFo-JG3gJvWu5v/s1600/module-growth.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="423" data-original-width="643" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaONn3IfrqTjF8TshFFQFfJYas_fwAFs80356f6k-JGu-ZdQcKHHl9zRSN8-4qAlc-A9At4EbX88lpmIE_nfE2hgpT3LEcsdgpkuTzAjp9pMrisWzYN-5S_dmJfHIGEXqFo-JG3gJvWu5v/s1600/module-growth.png" /></a></div>
<br />
..as one can see, the rate of growth is relatively linear with about 89 modules being added to each kernel release, which is not surprising as the size of the kernel is growing at a fairly linear rate too. It is interesting to see that the number of modules has easily more than tripled in the 10 years between v2.6.24 and v4.16-rc2, with a rate of about 470 new modules per year. At this rate, Linux will see the 10,000th module land in around the year 2025.Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com1tag:blogger.com,1999:blog-1016709139053396535.post-14683363731043289692018-02-03T17:28:00.000+00:002018-02-03T17:28:45.629+00:00stress-ng V0.09.15<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKjd80To9csuHt1AdIseyf4HFO9fHbgo2RbuNPO1SBQ3S6zQKDMwftdbv_3Y-9XiibXN2cYxMfjLphHphJ7hrEJFNjJju4-YtJll9gLFnvIburaajNH5-5dAus_rINtBFCbfnBJjn-ON_Q/s1600/stress-ng-large.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="460" data-original-width="388" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKjd80To9csuHt1AdIseyf4HFO9fHbgo2RbuNPO1SBQ3S6zQKDMwftdbv_3Y-9XiibXN2cYxMfjLphHphJ7hrEJFNjJju4-YtJll9gLFnvIburaajNH5-5dAus_rINtBFCbfnBJjn-ON_Q/s200/stress-ng-large.png" width="168" /></a>It has been a while since my <a href="http://smackerelofopinion.blogspot.co.uk/2017/07/new-features-landing-stress-ng-v00809.html">last post about stress-ng</a> so I thought it would be useful to provide an update on the changes since V0.08.09.<br />
<br />
I have been focusing on making stress-ng more portable so it can build with various versions of clang and gcc as well as run against a wide range of kernels. The portability shims and config detection added to stress-ng allow it to build and run on a wide range of Linux systems, as well as <a href="https://www.gnu.org/software/hurd/hurd.html">GNU/HURD</a>, <a href="http://www.minix3.org/">Minix</a>, <a href="https://www.debian.org/ports/kfreebsd-gnu/">Debian kFreeBSD</a>, various BSD systems, <a href="https://www.openindiana.org/">OpenIndiana</a> and OS X.<br />
<br />
Enabling stress-ng to work on a wide range of architectures and kernels with a range of compiler versions has helped me to find and fix various corner case bugs. Also, static analysis with a various set of tools has helped to drive up the code quality. As ever, I thoroughly recommend using <a href="http://smackerelofopinion.blogspot.co.uk/2017/09/static-analysis-on-linux-kernel.html">static analysis tools</a> on any project to find bugs.<br />
<br />
Since V0.08.09 I've added the following stressors: <br />
<ul>
<li>inode-flags - (using the FS_IOC_GETFLAGS/FS_IOC_SETFLAGS ioctl, see ioctl_iflags(2) for more details.</li>
<li>sockdiag - exercise the Linux sock_diag netlink socket diagnostics</li>
<li>branch - exercise branch prediction</li>
<li>swap - exercise adding and removing variously sized swap partitions</li>
<li>ioport - exercise I/O port read/writes to try and cause CPU I/O bus delays</li>
<li>hrtimers - high resolution timer stressor</li>
<li>physpage - exercise the lookup of a physical page address and page count of a virtual page</li>
<li>mmapaddr - mmap pages to randomly unused VM addresses and exercise mincore and segfault handling</li>
<li>funccall - exercise function calling with a range of function arguments types and sizes, for benchmarking stack/CPU/cache and compiler.</li>
<li>tree - BSD tree (red/black and splay) stressor, good for exercising memory/cache</li>
<li>rawdev - exercise raw block device I/O reads</li>
<li>revio - reverse file offset random writes, causes lots of fragmentation and hence many file extents</li>
<li>mmap-fixed - stress fixed address mmaps, with a wide range of VM addresses</li>
<li>enosys - exercise a wide range of random system call numbers that are not wired up, hence generating ENOSYS errors</li>
<li>sigpipe - stress SIGPIPE signal generation and handling</li>
<li>vm-addr - exercise a wide range of VM addresses for fixed address mmaps with thorough address bit patterns stressing</li>
</ul>
Stress-ng has nearly 200 stressors and many of these have various stress methods than can be selected to perform specific stress testing. These are all <a href="http://kernel.ubuntu.com/~cking/stress-ng/stress-ng.pdf">documented in the manual</a>. I've also updated the <a href="http://kernel.ubuntu.com/~cking/stress-ng/">stress-ng project page</a> with various links to academic papers and presentations that have used stress-ng in various ways to stress computer systems. It is useful to find out how stress-ng is being used so that I can shape this tool in the future.<br />
<br />
As ever, patches for fixes and improvements are always appreciated. Keep on stressing!Colin Ian Kinghttp://www.blogger.com/profile/06458723239721015750noreply@blogger.com5