Virgin Media Super Hub and ssh timeouts

After upgrading to a new "shiny" Virgin Media Super Hub I started to get annoying ssh timeouts on idle connections.  After some Googling around I discovered that it suffers from TCP connection timeouts and some users have suggested that there isn't much memory in the device, so it can't save much session data.

Well, how does one workaround this issue?  Setting the keep alive probe down to 50 seconds and then resending it every 10 seconds seems to do the trick for me. I also tweaked the TCP settings so that if no ACK response is received for 5 consecutive times, the connection is marked as broken.  Here's the quick one-liner fix:
 

 sudo sysctl -w net.ipv4.tcp_keepalive_time=50 \
 net.ipv4.tcp_keepalive_intvl=10 \
 net.ipv4.tcp_keepalive_probes=5  

Of course, to make these settings persistent across reboots, add them to /etc/sysctl.conf

I'm not sure if these settings are "optimal", but they do the trick. You're mileage may vary.

Testing eCryptfs

Over the past several months I've been occasionally back-porting a bunch of eCryptfs patches onto older Ubuntu releases.  Each back-ported fix needs to be properly sanity checked and so I've been writing test cases for each one and adding them to the eCryptfs test suite.

To get hold of the test suite, check it out using bzr:

 bzr checkout lp:ecryptfs  

and install the dependencies so one can build the test suite:

 sudo apt-get install debhelper autotools-dev autoconf automake \
 intltool libtool libgcrypt11-dev libglib2.0-dev libkeyutils-dev \
 libnss3-dev libpam0g-dev pkg-config python-dev swig acl \
 ecryptfs-utils libattr1-dev

If you want to test eCrytpfs with xfs and btrfs as the lower file system onto which eCryptfs is mounted, then one needs to also install the tools for these:

 sudo apt-get install xfsprogs btrfs-tools  

And then build the test programs:

 cd ecryptfs  
 autoreconf -ivf  
 intltoolize -c -f  
 ./configure --enable-tests --disable-pywrap  
 make  

To run the tests, one needs to create lower and upper mount points. The tests allow one to create ext2, ext3, ext4, xfs or btrfs loop-back mounted file systems on the lower mount point, and then eCryptfs is mounted on the upper mount point on top.   To create these, use something like:

 sudo mkdir /lower /upper  

The loop-back file system image needs to be placed somewhere too, I generally place mine in a directory /tmp/image, so this needs creating too:

 mkdir /tmp/image  

There are two categories of tests, "safe" and "destructive".  Safe tests should run in such a ways as to not lock up the machine.  Destructive tests try hard to force bugs that can cause kernel oopses or panics. One specifies the test category with the -c option.  Now to run the tests, use:

 sudo ./tests/run_tests.sh -K -c safe -b 1000000 -D /tmp/image -l /lower -u /upper  

The -K option tells the test suite to run the kernel specific tests. These are the ones I am generally interested in since I'm testing kernel patches.

The -b option specifies the size in 1K blocks of the loop-back mounted /lower file system size.  I generally use 1000000 blocks as a minimum.

The -D option specifies the path where the temporary loop-back mounted image is kept and the -l and -u options specified the paths of the lower and upper mount points.

By default the tests will use an ext4 lower filesystem. One can also run specify which file systems to run the tests on using the -f option, this can be a comma separated list of one or more file systems, for example:

 sudo ./tests/run_tests.sh -K -c safe -b 1000000 -D /tmp/image -l /lower -u /upper \
 -f ext2,ext3,ext4,xfs  

And also, instead of running a bunch of tests, one can just a particular test using the -t option:

 sudo ./tests/run_tests.sh -K -c safe -b 1000000 -D /tmp/image -l /lower -u /upper \
 -f ext2,ext3,ext4,xfs -t lp-926292.sh

..which tests the fix for LaunchPad bug 926292
 
We also run these tests regularly on new kernel images to ensure we don't introduce and regressions.   As it stands, I'm currently adding in tests for each bug fix that we back-port and for most new bugs that require a thorough test. I hope to expand the breadth of the tests to ensure we get better general test coverage.

And finally, thanks to Tyler Hicks for writing the test framework and for his valuable help in describing how to construct a bunch of these tests.

Simple performance test of rdrand

My new Lenovo X230 laptop is equipped with a Intel(R) i5-3210M CPU (2.5 GHz, with 3.1 GHz Turbo) which supports the new Digital Random Number Generator (DRNG) - a high performance entropy and random number generator.  The DNRG is read using the new Intel rdrand instruction which can return 64, 32 or 16 bit random numbers.

The DRNG is described in detail in this article and provides very useful code examples in assembler and C which I used to write a simple and naive test to see how well the rdrand performs on my i5-3210M.

For my test, I simply read 100 million 64 bit random numbers on a single thread. The Intel literature states one can get up to about 70 million rdrand invocations per second on 8 threads, so my simple test is rather naive as it only exercises rdrand on one thread.  For a set of 10 iterations on my test, I'm getting around 40-45 nanoseconds per rdrand, or about 22-25 million rdrands per second, which is really impressive.   The test is a mix of assembler and C, and is not totally optimal, so I am sure I can squeeze a little more performance out with some extra work.

The next test I suspect is to see just random the data is and to see how well it compares to other software random number generators... but I will tinker with that after my vacation.

Anyhow, for reference, the test can be found here in my git repository.

Using virt-manager to manage KVM and QEMU instances

I've been using QEMU and KVM for quite a while now for general kernel testing, for example, sanity checking eCryptfs and Ceph.   It can be argued that the best kind of testing is performed on real hardware, however, there are times when it is much more convenient (and faster) to exercise kernel fixes on a virtual machine.

I used to use the command line incantations to run QEMU and KVM, but recently I've moved over to using virt-manager because it so much simpler to use and caters for most of my configuration needs.

Virt-manager provides a very usable GUI and allows one to create, manage, clone and destroy virtual machine instances with ease.

virt-manager view of virtual machines

Each virtual machine can be easy reconfigured in terms of CPU configuration (number and type of CPUs),  memory size, boot options, disk and CD-ROM selection, NIC selection, display server (VNC or Spice), sound device, serial port config, video hardware and USB and IDE controller config.  

One can add and remove additional hardware, such serial port, parallel ports, USB and PCI host devices, watchdog controllers and much more besides.

Configuring a virtual machine

..so reconfiguring a test to run on a single core CPU to multi-core is a simple case of shutting down the virtual machine, bumping up the number of CPUs and booting up again.

By default one can view the virtual machine's console via a VNC viewer in virt-manager and there is provision to scale the screen to the window size, set to full size or resize the virt-manager window to the screen size.  For ease of use, I generally just ssh into the virtual machines and ignore the console unless I can't get the kernel to boot.

virt-manager viewing a 64 bit Natty server (for eCryptfs testing)

Virt-manager is a great tool and well worth giving a spin. For more information on virt-manager visit virt-manager.org