Digging into TRIM

Enabling the discard mount option to perform auto trimming on a SSD has some downsides.  The non-queued trim command can incur a large performance penalty on deletes; the driver needs to finish pending operations, then peform the trim command before it can resume normal operation again.  Queued TRIM support has been added to the 3.12 kernel which will help improve this issue, however, one requires hardware that supports the Serial ATA revision 3.1 Queued Trim command.

Trimming can trigger internal garbage collection on some firmware and causing some performance impact when one leasts expects it, so enabling TRIM via the discard mount flag may be suboptimal.  It is therefore better to batch up discards and run fstrim periodically when a machine is idle in terms of minimising performance impact (this is also more power efficient too).

For the curious, how much is being discarded when one runs fstrim? Fortunately the kernel has a tracepoint added to ext4_trim_extent and this enables one to observe the block device, allocation group, starting block of the free extent in the allocation group and number of blocks to TRIM.

With debugfs mounted one can easily observe this using:

 echo 1 | sudo tee /sys/kernel/debug/tracing/events/ext4/ext4_trim_extent/enable  
 sudo cat /sys/kernel/debug/tracing/trace_pipe | grep ext4_trim_extent  

then invoke fstrim from another terminal and one will observe:

fstrim-4909  [003] ....  6585.463841: ext4_trim_extent: dev 8,3 group 1378, start 2633, len 829
fstrim-4909  [003] ....  6585.474926: ext4_trim_extent: dev 8,3 group 1378, start 3463, len 187
fstrim-4909  [003] ....  6585.484014: ext4_trim_extent: dev 8,3 group 1378, start 3652, len 474
...

The deeper question is therefore how frequently should run fstrim?  It is hard to say exactly, since it is use-case dependant.  However, anecdotal evidence seems to suggest running fstrim either daily or weekly.

Finding spelling mistakes in source code

There are quite a few useful open source utilities around for finding spelling mistakes in source code.  Most recently I've been using codespell which works well for me.

Codespell is regularly being updated and comes with a dictionary originally derived from Wikipedia. I normally pull the latest updates from the repository before running it against my source code.

Fetching it and using it is relatively simple:

 git clone git://git.profusion.mobi/users/lucas/codespell  
 cd your-project-dir  
 /path-to-codespell/codespell.py  

..and it will find common spelling mistakes in the entire project directory. Easy!

Making software successful

What makes good software into excellent usable software?  In my opinion software that fails to be excellent software generally lacks some key features:

Documentation

A README file in the source is not sufficient for the end-user.  Good quality documentation must explain how to use all the key features in a coherent manner.  For example, a command line tool should be at least shipped with a man page, and this must cover all the options in the tool and explain clearly how each option works.

Documentation must be kept up to date with the features in the software.  If a feature is added to a program and the documentation has not been updated then the developer has been careless and this is a bug. Out of date documentation that provides wrong and useless information make a user lose faith in the tool.

Try to include worked examples whenever possible to provide clarity. A user may look at the man page, see a gazillion options and get lost in how to use a tool for the simple use-case.  So provide an example on how to get a user up and running with the simple use-case and also expand on this for the more complex use cases.  Don't leave the user guessing.

If there are known bugs, document them. At least be honest with the user and let them know that there are issues that are being worked on rather than let them discover it by chance.

Expect quality when developing

Don't ship code that is fragile and easily breaks. That puts users off and they will never come back to it.  There really is no excuse for shipping poor quality code when there are so many useful tools that can be used to improve quality with little or no extra cost.

For example, develop new applications with pedantic options enabled, such as gcc's -Wall -Wextra options.   And check the code for memory leaks with tools such as valgrind and gcc's mudflap build flags.

Use static code analysis with tools such as smatch or Coverity Scan.  Ensure code is reviewed frequently and pedantically.  Humans make mistakes, so the more eyes that can review code the better.   Good reviewers impart their wisdom and make the process a valuable learning process.

Be vigilant with all buffer allocations and inputs.  Use gcc's -fstack-protector to check for buffer overflows and tools such as ElectricFence.  Just set the bar high and don't cut corners.

Write tests.  Test early and often. Fix bugs before adding more functionality.  Fixing bugs can be harder work than adding new shiny features which is why one needs to resist this temptation and focus on putting fixes first over features.

Ensure any input data can't break the application.  In a previous role I analysed an MPEG2 decoder and manually examined every possible input pattern in the bitstream parsing to see if I could break the parser with bad data.  It took time, it required considerable effort, but it ended up being a solid piece of code.  Unsanitised input can break code, so be careful and meticulous.

Sane Defaults

Programs have many options, and the default modus operandi of any tool should be sane and reasonable.   If the user is having to specify lots of extra command line options to make it do the simplest of tasks then they will get frustrated and give up. 

I've used tools where one has to set a whole bunch of obtuse environment variables just to make it do the simplest thing.  Avoid this. If one has to do this, then please document it early on and not buried at the end of the documentation.

Don't break backward compatibility

Adding new features is a great way to continually improve a tool but don't break backward compatibility.  A command line option that changes meaning or disappears between releases is really not helpful.     Sometimes code can no longer be supported and a re-write is required.  However, avoid rolling out replacement code that drops useful and well used features that the user expects.

The list could go on, but I believe that if one strives for quality one will achieve it.  Let's strive to make software not just great, but excellent!

health-check revisited

Earlier this month I wrote about health-check, a tool to sanity check application resource usage.  Since then I've re-worked and simplified the system call tracing and added some new features.

Originally, health-check was designed to attach itself to running processes. To make the tool even easier to use it can now start applications and follow any subsequent new processes spawned off from fork() or clone(). For example:

sudo health-check -u joeuser -f thunderbird

..will start thunderbird (running as user "joeuser") and follow any new processes it creates.

Sometimes applications will force flushing of data or metadata to disk by excessive use of fsync(), fdatasync() and sync().  Health-check will now keep track of these system calls and provide feedback on their use.

Health-check already has some memory checking techniques - however, it now has been extended to check explicitly for heap changes by examining the brk() system call and also keeping track of mmap() and munmap() mappings.  This allows better tracking of potential memory leaks.

Source code can be found at: git://kernel.ubuntu.com/cking/health-check

Packages found in my White PPA in ppa:colin-king/white so to install on Ubuntu systems use:

 sudo add-apt-repository ppa:colin-king/white  
 sudo apt-get update  
 sudo apt-get install health-check