Firmware Test Suite for Ubuntu 11.04

The  Firmware Test Suite (fwts) is a tool I've been working on over the past few months that does automated testing of PC firmware.  The main aim is to check for errors in BIOS code and ACPI tables.  I have put most effort into ACPI testing as my analysis shows that it constitutes over 90% of firmware related errors reported against the kernel in LaunchPad.

Most of the key features for Ubuntu 11.04 are now in the Firmware Test Suite, so now seems an appropriate time to mention them.

Automated ACPI Method Testing.

The ACPI DSDT and SSDT tables contain ACPI Machine Language (AML) bytecode.  This code is executed by the Linux kernel by the ACPI driver and is an abstract interface between the operating system and the underlying machine hardware, such as the embedded controller, the closed proprietary BIOS and various hardware registers.   The AML can describe hardware layout and contains methods that operate in a manner that should conform to the ACPI specification.   For example, there are methods to handle power management,  thermal control, battery status and much more besides.   The main problem we face is when these implementations are buggy and may not conform to the specification.   The kernel can execute the AML code and so if the AML is broken you may get a machine that does not work correctly.

I've added a method test to fwts that will extract the AML byte code from the DSDT and then using the ACPICA execution engine run these in user space and check methods are returning the expected return types and in some cases (where appropriate) check expected return values.  Over 70 standard methods described in version 4.0a of the ACPI specification are sanity checked.  The code also checks mutex acquire/release counts to see if methods are incorrectly not releasing locks.

To run these test using 11.04 fwts, use:

sudo fwts method

..and the results are appended to the file results.log.  (One has to use sudo to be able to read the ACPI tables).

Another way to sanity check the AML code is to extract it, dis-assemble and then re-assemble it using the ACPICA iasl compiler.  A lot of AML is compiled using the Microsoft AML compiler and this appears to be less strict. Hence, recompiling with a more strict compiler can pick up errors.   The syntaxcheck test existed in fwts for Ubuntu 10.10, but I've re-worked this by integrating in the ACPICA core into the tool and then enhancing the error reporting.  For most error messages produced by the compiler fwts now also outputs 8 lines of disassembled code around the error (to provide context) and also outputs extra feedback advice.  This advice is based on looking at all the reasons why this error can be emitted by the compiler and explaining these with more context.

To run this test, use:

sudo fwts syntaxcheck

..again, output is appended to results.log

Since the ACPICA core is integrated in, we can also use this to easily dump out the disassembled AML byte code from all the ACPI tables.  To do so use:

sudo fwts --disassemble-aml

and the AML code for each table is dumped into files DSDT.dsl and SSDTn.dsl (where n is the Nth SSDT).  This is much easier than running acpixtract, acpidump and the iasl -d on the DSDT and SSDT tables.

The final ACPI table test added to fwts in Ubuntu 11.04 performs about 40 sanity checks on configuration data in common ACPI tables.  To run this test use:

sudo fwts acpitables

 

..and this checks various configurations, e.g. of HPET, SCI interrupt, GPE blocks, reset register + reset values, APIC flags, etc.

Although it's strictly not a test, fwts now includes a tool that will dump out the ACPI tables in an annotated form so that one can examine the configuration settings in a more human readable form than just looking at the raw data in a hex editor. To do use:

sudo fwts acpidump

UEFI

The Firmware Test Suite also needs to work on UEFI firmware based machines which are now becoming far more common.  I've re-worked some of the underlying code in fwts to work with UEFI and standard BIOS.  My next step is to extend fwts to add in more UEFI only tests, but this is still "work-in-progress".

Other goodness

As it is an evolving tool, fwts includes a lot more refinements (and bug fixes) compared the first cut that came out in Ubuntu 10.10.   The log format to be less verbose and there is the new "-" option that dumps the log straight to stdout rather than appending it to a log file.

To see all the tests built into fwts, you can use the "--show-tests-full" option, rather than the more terse option "--show-tests".  

Originally fwts ran silently and this caused some confusion as it was not obvious what was happening.  For Ubuntu 11.04  fwts now runs in verbose mode and reports a brief per test progress information.  The silent mode can be enabled using the "-q" or "--quiet" options.

One can also run the tool on dumped ACPI tables from other machines allowing to remotely diagnose bugs just from raw table data.  Dump the tables using:

sudo fwts --dump

..this produces a dump of kernel log, dmidecode, lspci and the ACPI tables. One can read in the ACPI tables into fwts and run appropriate ACPI specific tests on this data, e.g.

fwts --dumpfile=acpidump.log syntaxcheck method acpitables

There are many more refinements and minor new features.  I recommend reading the manual page to get more familiar with the tool.

To install fwts in Ubuntu 11.04 use:

sudo apt-get install fwts

Enjoy!

[ Update ]

I've written a fwts reference guide, available here: https://wiki.ubuntu.com/Kernel/Reference/fwts