Lost Circuits article on HDD data transfer rates.

Lost Circuits has written an in-depth article on hard disk media, covering aspects of HDD performance, such as transfer rates depending on the zone position (inner to outer diameters), discrepancies between internal transfer rates and benchmarked rates as well as the impact of track slew.

Some of the internal mechanical magic is also described, which helps explain how manufacturers configure their hardware and sector layouts to maximise performance. All very informative.

The website is slow, so be patient!

Hot Laptop

My Lenovo 3000N200 laptop has been playing me up. When I've been fully loading the processor or driving video hard it's been shutting down because of overheating. I suspect periodic SMIs are detecting an overheated CPU and the BIOS just stops the machine to avoid it turning into toast.

Suspecting that the latest 2.6.35 Maverick kernel was the cause I booted with a 2.6.32 Lucid kernel and that didn't help, so it didn't look like an obvious kernel regression.

Well, perhaps it's getting old and cranky - it's nearly 3 years old. Perhaps the thermal paste between the CPU and the heatsink is not working like it should. Since it was most probably a hardware issue I downloaded the service manual and got out the trusty screwdriver and opened it up. Lo and behold 5mm of dust had accumulated over the fan grill which wasn't going to help the poor machine offload all that heat out of the laptop case. I removed the fan, gave it a good clean and removed all the dust from the fan outlet grill.

After reassembly the laptop was good as new. Instead of rebooting at 95+ degrees Celsius the Lenovo now runs happily.

The moral of the story is that I should regularly service the fans on my machines. Cooking the CPU is something I would like to avoid in the future.

Makerbot - an open source 3D printer

This week I'm attending a Ubuntu Kernel Sprint and my colleague Steve Conklin brought along a fantastic gizmo - the Makerbot 3D printer. It is most fascinating watching it print 3D objects by extruding a thin ABS plastic trail of plastic in layers. It can print objects up to 4" x 4" x 6" - the imagination is the limiting factor to what it can print. Steve already demo'd it printing a variety of objects, the most impressive being a working whistle including a moving ball inside the whistle. It's not too slow either - it took about 25 minutes to print the whistle, which isn't bad considering the complexity and size of the object.













It's a cool piece of kit - doubly so because it's completely open sourced.

Hacking the Arduino

I acquired a 16Mhz 8 bit Arduino micro-controller development board last week from my colleague Steve Conklin. The Arduino is a cool little open source electronics development system and allows rapid prototyping - ideal for hacking up small controller projects. I downloaded the C toolkit and GUI development environment from Arduino.cc and configuration was very simple and straight forward.

Basically, one can cruft up small C programs and download these to the board using a USB interface - the code is flashed onto the board so it can run stand-alone - sweet. Just compile, download and go!

Anyhow, today was my first opportunity to get some quality time to get hacking. I rummaged around in my loft and found a bread-board and a bunch of LEDs and resistors that I bought over 20 years ago and rigged up 4 LEDs to be driven from from 4 digital output pins. I quickly hacked up some code to enable the pin outputs and then drive the LEDs, see the video below:


The Arduino has a bunch of digital input/ouput pins and some analogue pins too - although I need to now read up about this to see what I can really do with this kit. Anyhow, thanks to Steve for getting this kit into my hands and walking me through the elementary stuff on how to select the right resistors so that I don't kill the LEDs and the Arduino; I knew I should have remembered all that basic electronics jiggery pokery when I did my Computer Science degree 20+ years ago...

My hope is to hook this up to various bits of hardware to enable me to do the usual debugging and hacking around. I've got various ideas of projects, such as an interface between my PC and my old Commodore 1541 disk drive, but I need to get some reference books out of my loft and do some more research before I get my hands grubby with real code.

What's neat is that I'm able to some relatively fast low-level 8 bit hackery without all that unnecessary OS nonsense making life more complex :-)

Disabling my Synaptics Mouse using xinput

My Lenovo 3000 N200 Laptop has been suffering a bit lately from old age. On returning from a week working with my fellow colleagues in Dublin the 120GB HDD died. Fortunately I had early warning from some SMART monitoring tools telling me I had a potential HDD failure and I was able to backup all my data before the HDD finally died and restore it on a spare HDD. I managed to find a fairly reasonable Western Digital Scorpio 160GB HDD replacement for about £27 (w/o VAT and postage) which should arrive by Monday.

Unfortunately the Synaptics touchpad has been getting really flakey. Typically the touchpad gets random button press events and then the button states get locked and toggling the buttons just returns the mouse into some weird unpredictable state. Try using GNOME when the buttons states are reversed or stuck down and you get to see how frustrating this can be!

This has not been so much of a problem as I normally use a wireless mouse so the touchpad is never normally used. It's just a pain when the touchpad just goes into weird event mode at random times, usually when you least expect it! My long term solution is to strip the machine down and see why the touchpad is misbehaving (I've downloaded the maintenance manual from Lenovo and it's not a 5 minute task to fix it). Hence I need a quick workaround before I go insane from weird mouse events.

My original plan was to disable the mouse driver, but in Karmic it's built in so this would require rebuilding the kernel with the driver compiled out and I've been resisting this kind of ugly hack. However, tonight I found a better workaround to disable the touchpad:

1. Find the Synaptics Mouse using:

xinput list

This dumps out a load of info, e.g.:

SynPS/2 Synaptics TouchPad" id=8 [XExtensionPointer]
Type is TOUCHPAD
Num_buttons is 12
Num_axes is 2
Mode is Relative
Motion_buffer is 256
Axis 0 :
Min_value is 1472
Max_value is 5472
Resolution is 1
Axis 1 :
Min_value is 1408
Max_value is 4448
Resolution is 1

and a lot more besides.

2. This enabled me to find the textual handle and id number (8) to my touchpad. I then listed the properties using:

xinput list-props "SynPS/2 Synaptics TouchPad"

which dumps out all the properties, e.g.:

Device Enabled (115): 1
Synaptics Edges (260): 1752, 5192, 1620, 4236
Synaptics Finger (261): 24, 29, 255
Synaptics Tap Time (262): 180
Synaptics Tap Move (263): 221
Synaptics Tap Durations (264): 180, 180, 100

..and a lot more properties too..

I then spotted the "Device Enabled" property, which I then disabled using:

xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Device Enabled" 8 0

..and my touchpad was disabled! No more annoying stray events!

3. I next added this rune into the Startup Applications Preferences (from System->Preferences->Startup Applications).

Hopefully when I've got a spare hour or two and the kids aren't around I will strip this laptop down and fix the touchpad. Meanwhile, I'm happy to get control back from my misbehaving touchpad!

LED console driver

Some of this afternoon I've been hacking away writing a *very* simple console driver that pulses the keyboard LED using asynchronous pulses identical to that of RS232 serial. The idea is to hook up a photo diode and convert the light signal into asynchronous pulses into a USB serial dongle that's connected to a debugging host PC. Then I should be able to do printk() over the keyboard LED and be able do debug the kernel when I don't have the luxury of a video console. :-)

OK, so the speed may be very slow, ~300 baud, but it may just work. If it does, it will get me out of all sorts of painful debugging holes where I don't have the luxury of being able to debug using POST codes by writing to port 0x80.

I will let you know if it works once we get some kit to read the LED reliably and at speed...

Debugging the Linux Kernel over a Serial Console.

There are times when the luxury of a text based console just do not exist when debugging kernels on a PC and one has to dump debug out over a serial console. This used to be relatively straight forward a few years ago since every PC had a DB9 serial port and bit banging data over this was fairly low tech and just worked.

However, the modern PC does not have such legacy ports anymore, so one has to fall back to using a USB serial dongle instead. One can purchase such kit quite easily, I use a PL2303 based USB serial dongle, it's fairly inexpensive (about £3.00) and Linux provides a serial console tty driver for this. I attach one serial console to the PC that needs debugging and connect this via a null modem cable to my host PC which captures the debug using a serial console terminal such as minicom.

Install minicom on your host debugging machine:

sudo apt-get install minicom

and build a debug kernel for the target machine with the following config options:

CONFIG_USB_SERIAL_CONSOLE=y  
CONFIG_USB_SERIAL=y

and enable the appropriate driver, e.g.:

CONFIG_USB_SERIAL_PL2303=y

Install this one the target machine to debug. Configure minicom to run at 115200 baud, 8 bits per char, no parity stop bits, no flow control and start it up on the host. Boot the target machine with the following kernel boot parameters:

console=tty console=ttyUSB0,115200n8

..and hopefully you will see all the console text appear in the minicom terminal, albeit rather slowly(!) You may see some dropped characters as flow control has been turned off.

This kind of debug is especially useful when you just cannot get normal VGA console debug output. However, because it's over USB, it can be a little useless for debug in the late stages of suspend or early stages of resume or boot.

I've used this technique to dump scheduling state over the console during X and network hangs. All in all, rather basic and crude, but it's another tool in my box for sorting out problematic kernel issues.

USB persist, Webcams and Suspend/Resume

The other day I was looking at a problem with a laptop running Ubuntu Hardy that had an integrated USB webcam that failed to work over a suspend/resume cycle. A webcam viewer application such as Cheese or UCView was being run during the suspend/resume and had locked up after the resume. A workaround was to enable USB persist on this device as described in Documentation/usb/persist.txt in the kernel source.

From my understanding, if a USB host controller loses power (e.g. during a system syspend) then it is treated as if it has been unplugged. For devices like integrated webcams we know it cannot be unplugged, so we can enable USB persist.

USB host controllers can get reset after suspend to RAM on some systems, and you will see kernel messages such as "root hub lost power or was reset" (use dmesg to see this). In such cases, USB persist can save the day.

UBS persist keeps the USB device's core data structures persistent when power session disruption occurs. Essentially on resume the kernel checks if persist is set on the device, if so, it does a USB port reset, re-enumates and checks if the device is the same as the one before (e.g. checking descriptors and information such as produce and vendor IDs) it then re-uses the original device data structures.

USB persist for a device can be enables by echo'ing 1 to the devices persist file as root:

echo 1 >/sys/bus/usb/devices/.../power/persist

Below is a shell script to set USB persist for a device with a given IDVENDOR and IDPRODUCT USB ID.

#!/bin/sh
IDVENDOR=0123
IDPRODUCT=0afb
for I in /sys/bus/usb/devices/*/*
do
if [ -e $I/idVendor -a -e $I/idProduct ]; then
idvendor=`cat $I/idVendor`
idproduct=`cat $I/idProduct`
if [ x$idvendor = x$IDVENDOR -a x$idproduct = x$IDPRODUCT ]; then
if [ -e $I/../power/persist ]; then
echo 1 > $I/power/persist
fi
if [ -e $I/../power/persist ] ; then
echo 1 > $I/../power/persist
fi
fi
fi
done

Note that doing USB persist on pluggable USB devices is NOT recommended as it can cause kernel panics!

The Linux-Ready Firmware Development Kit

The Linux-Ready Firmware Developer Kit is a useful tool to check if important firmware features on your PC are correct or not.

One can download a live CD which will run the tests without the need to install any software on your machine whatsoever. Alternatively you can grab the source and build it yourself.

To build and run it on my Ubuntu server I first had to install xutils-dev, bison, flex and iasl. I simply downloaded the source tar ball, gunzip and untar'd it and ran make. To run it from the build directory one has run as root and do:

export LD_LIBRARY_PATH=.
./biostest

It runs a bunch of tests, such as checking the ethernet functionality, CPU frequency scaling, fan testing, HPET configuration checking, MTRR and BIOS checks to name but a few. Hopefully with this kit one can determine if a BIOS is configured correctly for Linux before shipping the BIOS on PCs. We can only hope :-)

Hacking the Chumby

A few weeks ago I was given a Chumby. What is a Chumby? Well it's a nifty little Linux based internet enabled device that plays Flash Lite widgets. The Chumby has a small 3.5" 320x240 colour touch screen, a couple of USB 2.0 ports, stereo 2W speakers and a headphone socket. The processor is a 350Mhz Freescale MX21ADS (ARM926EJ-Sid(wb) rev 4 (v5l)) and on board is 64 MB SDRAM and 64MB NAND flash.

It also has a squeeze sensor and a motion sensor (accelerometer) - the latter is used for interaction, such as games Widgets.

Hacking this device is fairly straight forward, there is a hidden button on a configuration screen that selects a hidden options menu. This has an option to allow one to enable SSH, and once enabled one can SSH in as root and start playing! To keep SSH enabled even after a reboot one needs to touch the file /psp/start_sshd.
Another hidden option enables a simple Web Server. I've hacked this to allow me to check the system status, it's rather crude, but it works!

The Chumby Website allows one to select from over a 1000 widgets - you simply add these to one of your personal Chumby channels and the device downloads these Flash Lite widgets which are free to use. There are a huge range of widgets, ranging from internet radio players, clocks, RSS news feed viewers, webcam viewers, games, photo viewers and more beside!

As for hacking, there is a Wiki with plenty of information on how to download the GCC toolchain and kernel source - with which one can start porting apps. It's early days for me - I've only rebuilt the kernel and ported bash, but I'm looking to do some hoopy things, such as get a C64 emulator ported - there may be enough horsepower on this device for it to work. Watch this space!

Chumby References:

Chumby Industries
Wikipedia
ARM

Debugging Intel X Hangs

While I was at UDS I learnt some useful X debugging tricks for Intel based chipsets. One can debug the driver using debugfs as follows:

1. ssh into the machine with locked-up X and then mount debugfs:

$ sudo mount -t debugfs none /sys/kernel/debug

2. Find the dri debug directory:

$ cd /sys/kernel/debug/dri/0

3. Check for a system lock up by looking to see if sequence numbers are advancing or not:

$ cat i915_gem_seqno
Current sequence: 1732368
Waiter sequence: 0
IRQ sequence: 1732364

If the sequence numbers are not increasing then we know that the GPU has locked up for some reason.

4. If that's working, then check to see what the X server is doing:

$ cat /proc/pid-of-X-server/wchan

(where 'pid-of-X-server' is the process id of the X server)

This will show you what it is waiting for. If you see it changing then it's not an X hang.

5. Look at the interrupt activity

$ cat i915_gem_interrupt

Check that the masks are restored correctly after a resume - interrupts may be masked and hence not able to respond to interrupts.

The IRQ sequence generally is a little behind Waiter sequence - if IRQ sequence does not increment it's a GPU hang. The Current Sequence *SHOULD NOT* be zero. Waiter Sequence is zero when there is nothing queued up to process.

Also, check out http://intellinuxgraphics.org/documentation.html

Brother 2170W Laser Printer

A little while ago my old cranky HP InkJet printer finally died and it was time to look for a new printer. This time around I was looking for a simple black and white laser printer as these work out cheaper per page than colour Inkjets. Lack of colour is a bonus in my books as it stops the kids wanting to print out lots of pictures from Tuxpaint and Gcompris :-)

As a luxury option I wanted to see if I could find a reasonably priced laser printer that also had Wifi connectivity to give me a little more flexibility in where I put the box.

After some searching around I ended up buying a Brother HL2170W, which can print upto 22 pages a minute and has very good driver support in Ubuntu. My Ubuntu Intrepid and Jaunty laptops found the printer on the network with no effort at all, can configuring it was a no-brainer. Once configured, one can easily check the printer's status and do further configuration using the printers web based admin tool, as shown below:

The downside was that when it associates with my ancient and flakey 3COM Office Connect router/Access Point the router occasionally crashes when using WPA2 PSK(!). I don't believe this is a fault with the Brother's Wifi at all. Until I'm brave enough to reflash my router with a firmware upgrade or buy a better one I won't be using my wireless laser printer wirelessly :-(

The Brother so far is reliable and just works as it should, with zero hassles. The only downside is that the printer has 32MB of memory, so it's a little slow at printing very large graphics.

The full specifications of the printer can be found here.

UPDATE: 24th Aug 2009. I've re-flashed my Wireless Access Point and I still cannot get the printer to associate. When I have time I will experimenting with another bit of wireless kit and see what's going on. Hmph.

Bonnie++, a useful benchmarking tool

When it comes to figuring out how a file system behaves on a HDD or SSD I generally first turn to Bonnie++ as a straight forward and easy to use disk benchmarking tool.

Bonnie performs a series of tests, covering character and block read/write I/O, database style I/O operations as well as tests for multiple file creation/deletion and some random access I/O operations.

To install bonnie for Ubuntu, simply do:

sudo apt-get install bonnie

To get a fair I/O benchmark (without the memory cache interfering with results) Bonnie generates test files that are at least twice the size of the memory of your machine. This can be a problem if you have servers with a lot of memory as the generated files can be huge and testing can therefore take quite a while. I generally boot a system and specify ~1GB of memory using the mem=1024M kernel parameter just so that Bonnie only tests with a 2GB file to speed my testing up.

My rule of thumb is to run Bonnie at least 3 times and take an average of the results. If you see a large variation in your results double check that there isn't some service running that is interfering with the benchmarking.

Bonnie references:

http://www.coker.com.au/bonnie++
http://linux.com/archive/feature/139742