Fixing Linux System Pauses

A couple of weeks ago I was looking at a bug where a netbook would just seem to randomly hang and would only come alive when a key was pressed. After poking around a bit I recalled that my colleague Stefan Bader had seen this issue before, and he told me to try booting with the Linux boot parameter acpi_skip_timer_override. Lo and behold, this workaround worked.

So what was going on? Well, it's a BIOS issue. The BIOS seemed to be claiming that IRQ0 was routed to another IRQ on the IO-APIC and in fact, this was not so. Generally speaking, documentaion for most chipsets is not disclosed, so it's impossible to know how the chipset is configured and worse still, how to fix the problem with a quirk. There are a couple of patches in the kernel (e.g. for the patch for the HP NX6325) where this problem is worked around with a quirk, but for other machines, one has to work around this problem with appropriate boot options.

We see this bug manifest itself because modern kernels use a tickless timer and we hit a state where all the CPUs have gone into a deep C state and need a timer interrupt to wake them up. However, if the routing of the timer interrupt is misconfigured then then CPU is not woken up, hence the hang until we generate an external interrupt, for example, by pressing a key.

One can debug this by booting with kernel boot parameter "debug lapic=debug". This will make Linux dump out the interrupt routing on the IO-APIC and it's worth using to understand what's going on under the bonnet.

Boot options that are worth trying to work around this option are:

acpi_skip_timer_override
- this ignores the IRQ zero / pin 2 interrupt override

hpet=disable
- disable HPET and use PIT instead

idle=poll
- forces a polled idle loop (so CPU won't go into deep C state), hence uses power and makes the system run hot (not recommended)

idle=halt
- halt is forced to be used for CPU idle, C2/C3 states won't be used again. This may only work on systems without C1E (Enhanced Halt State).

So, if you ever see Linux hanging around and not waking from it's idle state, try one of the above and see if this solves your problem.

Amusing LaunchPad Bug

Every so often while working on bugs reported in LaunchPad one stumbles on some interesting or just plain weird bug reports. LaunchPad bug #107648 probably the most amusing one I read of late.

Bug #1 is the longest running bug (and will take a few minutes to fetch), and will take a while to fix, but we are working on it. :-)

BBC article on Open Source

Open Source software has made it onto the BBC's radar again with an article on their Click programme. It's commendable that they devote some airtime to it; the programme seemed be concluding that Open Source has missed opportunities because it has not marketed or promoted itself. This is quite frustrating, as the BBC normally reports on new software releases from Microsoft and Apple, but pays little attention to the new releases of software such as Open Office or new distro releases.

Also, it amused me that the article mentioned that the Asus Eee-PC has been "quite" successful. In fact, the Linux driven netbook revolution caught Microsoft by surprise and caused a massive change in the way people do low-cost mobile computing. Microsoft just has XP, with Linux there are lots of revolutionary desktop experiences which are tailored for this computing model.

It's a start. The BBC Click programme did mention it would like to hear your comments, so please watch the programme, and post them positive and helpful feedback - hopefully we can focus the BBC to devote more airtime to the benefits of Open Source computing!

Understanding Linux Early Resume Hacks

When Linux resumes from a suspend, it can be configured to do some really grim BIOS related hacks while in real mode; these hacks are controlled by the acpi_sleep kernel boot parameter.

The settings are as follows:

acpi_sleep=s3_bios
This makes the kernel set the video by directly calling into the video BIOS ROM at address 0xc000.

acpi_sleep=s3_mode
This makes the kernel set the video mode by BIOS interrupt int $10, ax=0x4f02 (select VESA video mode).

acpi_sleep=s3_beep
This causes the kernel to beep. Depending on the kernel version you either get a beep or a beeped Morse code message "...-" (which is Morse for "V" for some reason).

These acpi_sleep settings map to values:

s3_bios 1
s3_mode 2
s3_beep 4

and can be set by writing these magic values to /proc/sys/kernel/acpi_video_flags before you do the next suspend; this is a little more flexible than setting the kernel boot parameter acpi_sleep.

s3_beep is checked for first - so if you want to make sure your machine is coming out of the BIOS and executing the initial kernel resume code, set this option and you will hear a beep during the very early resume execution as long as you have an internal speaker in your PC.

s3_bios is then next checked and if enabled, the kernel calls directly into the video BIOS ROM to force the video state.

Finally, s3_mode is checked, and if enabled a VESA mode setting BIOS interrupt is executed to set the set video mode.

Some BIOSs just croak and die if you do s3_bios and s3_mode calls - which is understandable as the behaviour for doing such calls during a resume is undefined. It may work, it may not!