Today we were trying to do some debugging by getting some tones out of a laptop speaker by frobbing bit 1 of port 0x61 on the keyboard controller. Rather unexpectedly I got no sound whatsoever out of the speaker, yet I had managed to do so the day before. So I double checked what had changed since the day before:
1. Was it because I upgraded my kernel?
2. Did I unexpected disabled the speaker when tweaking BIOS settings?
3. Was it something interfering with my port 0x61 bit twiddling?
4. Was the hardware now broken?
As per usual, I first assumed that the most complex parts of the system were to blame as they normally can go wrong in the most subtle way. After a lot of fiddling around I discovered that the PC speaker only worked when I plugged the AC power into the laptop. Now that wasn't obvious.
I suspect I should have applied Occam's Razor to this problem to begin with. We live and learn...
Wednesday, 29 June 2011
Friday, 24 June 2011
dstat - a replacement for vmstat, iostat and ifstat
Normally when I see a problem to do with CPU, I/O, network or memory resource hogging I turn to my trusty tools such as vmstat, iostate or ifstat to check system behaviour.
I stumbled upon dstat today, which boasts to be "a versatile replacement for vmstat, iostat and ifstat.". So let's see what it can do. First, install with:
Running dstat with no arguments displays CPU stats (user,system,idle,wait,hardware interrupts, software interrupts), Disk I/O stats (read/write), Network stats (receive,send), Paging stats (in/out) and System stats (interrupts, context switches). Not bad at all.
Dstat even highlights in colour the active values, so you don't miss relevant deltas in the statistics being churned out by the tool. Colours can be disabled with the --nocolor option.
But there is more. There are a plethora of options to show various system activities, such as:
-l load average
-m memory stats (used, buffers, cache, free)
-p process stats (runnable, uninterruptible, new)
--aio aio stats (asynchronous I/O)
--fs filesystem stats (open files, inodes)
--ipc ipc stats (message queue, semaphores, shared memory)
--socket socket stats (total, tcp, udp, raw, ip-fragments)
--tcp TCP stats (listen, established, syn, time_wait, close)
--udp UDP stats (listen, active)
--vm virtual memory stats (hard pagefaults, soft pagefaults, allocated, free)
..to name but a few. As it is, this already is more functional that vmstat, iostat and ifstat. But that's not all - dstat is extensible by the use of dstat plugins and the packaged version of dstat comes shipped with a few already, for example:
--battery battery capacity
--cpufreq CPU frequency
--top-bio-adv top block I/O activity by process
--top-cpu-adv most expensive CPU process
--top-io most expensive I/O process
--top-latency process with highest latency
..and many more besides - consult the manual page for dstat to see more.
So, dstat really is a Swiss army knife - a useful tool to get to know and use whenever you need to quickly spot misbehaving processes or devices.
I stumbled upon dstat today, which boasts to be "a versatile replacement for vmstat, iostat and ifstat.". So let's see what it can do. First, install with:
sudo apt-get install dstat
Running dstat with no arguments displays CPU stats (user,system,idle,wait,hardware interrupts, software interrupts), Disk I/O stats (read/write), Network stats (receive,send), Paging stats (in/out) and System stats (interrupts, context switches). Not bad at all.
Dstat even highlights in colour the active values, so you don't miss relevant deltas in the statistics being churned out by the tool. Colours can be disabled with the --nocolor option.
But there is more. There are a plethora of options to show various system activities, such as:
-l load average
-m memory stats (used, buffers, cache, free)
-p process stats (runnable, uninterruptible, new)
--aio aio stats (asynchronous I/O)
--fs filesystem stats (open files, inodes)
--ipc ipc stats (message queue, semaphores, shared memory)
--socket socket stats (total, tcp, udp, raw, ip-fragments)
--tcp TCP stats (listen, established, syn, time_wait, close)
--udp UDP stats (listen, active)
--vm virtual memory stats (hard pagefaults, soft pagefaults, allocated, free)
..to name but a few. As it is, this already is more functional that vmstat, iostat and ifstat. But that's not all - dstat is extensible by the use of dstat plugins and the packaged version of dstat comes shipped with a few already, for example:
--battery battery capacity
--cpufreq CPU frequency
--top-bio-adv top block I/O activity by process
--top-cpu-adv most expensive CPU process
--top-io most expensive I/O process
--top-latency process with highest latency
..and many more besides - consult the manual page for dstat to see more.
So, dstat really is a Swiss army knife - a useful tool to get to know and use whenever you need to quickly spot misbehaving processes or devices.
Thursday, 23 June 2011
Transitioning a PC to S5 (revisited)
Back in February I wrote about turning off a PC using the Intel I/O Controller Hub and had some example code to do this, and it was a dirty hack. I've revised this code to be more aligned with how the Linux kernel does this, namely:
1. Cater for the possible existence of PM1b_EVT_BLK and PM1b_CNT_BLK registers.
2. Clear WAKE_STATUS before transitioning to S5
3. Instead of setting SLP_TYP and SLP_EN on, one sets SLP_TYPE and then finally sets SLP using separate writes.
The refined program requires 4 arguments, namely the port address of the PM1a_EVT_BLK, PM1b_EVT_BLK, PM1a_CNT_BLK and PM1b_CNT_BLK. If the PM1b_* ports are not defined, then these should be 0.
To find these ports run either:
or do:
sudo fwts acpidump - | grep PM1 | grep Block:
For example, on a Sandybridge laptop, I have PM1a_EVT_BLK = 0x400, PM1a_CNT_BLK = 0x404 and the PM1b_* ports are not defined, so I use:
sudo ./halt 0x400 0 0x404 0
..and this will transition the laptop to the S5 state very quickly. Needless to say, make sure you have sync'd and/or unmounted your filesystems before doing this.
The PM1_* port addresses from /proc/ioport and the fwts acpidump come from the PM1* configuration data from the ACPI FACP, so if this is wrong, then powering down the machine won't work. So, if you can't shutdown your machine using this example code then it's possible the FACP is wrong. At this point, one should sanity check the port addresses using the appropriate Southbridge data sheet for your machine - generally speaking look for:
PM1_STS—Power Management 1 Status Register (aka PM1a_EVT_BLK)
PM1_CNT—Power Management 1 Control (aka PM1a_CNT_BLK)
..however these are offsets from PMBASE which are defined in the LPC Interface PCI Register Address Map so you may require a little bit of work to figure out the addresses of these registers on your hardware.
1. Cater for the possible existence of PM1b_EVT_BLK and PM1b_CNT_BLK registers.
2. Clear WAKE_STATUS before transitioning to S5
3. Instead of setting SLP_TYP and SLP_EN on, one sets SLP_TYPE and then finally sets SLP using separate writes.
The refined program requires 4 arguments, namely the port address of the PM1a_EVT_BLK, PM1b_EVT_BLK, PM1a_CNT_BLK and PM1b_CNT_BLK. If the PM1b_* ports are not defined, then these should be 0.
To find these ports run either:
cat /proc/ioports | grep PM1
or do:
sudo fwts acpidump - | grep PM1 | grep Block:
For example, on a Sandybridge laptop, I have PM1a_EVT_BLK = 0x400, PM1a_CNT_BLK = 0x404 and the PM1b_* ports are not defined, so I use:
sudo ./halt 0x400 0 0x404 0
..and this will transition the laptop to the S5 state very quickly. Needless to say, make sure you have sync'd and/or unmounted your filesystems before doing this.
The PM1_* port addresses from /proc/ioport and the fwts acpidump come from the PM1* configuration data from the ACPI FACP, so if this is wrong, then powering down the machine won't work. So, if you can't shutdown your machine using this example code then it's possible the FACP is wrong. At this point, one should sanity check the port addresses using the appropriate Southbridge data sheet for your machine - generally speaking look for:
PM1_STS—Power Management 1 Status Register (aka PM1a_EVT_BLK)
PM1_CNT—Power Management 1 Control (aka PM1a_CNT_BLK)
..however these are offsets from PMBASE which are defined in the LPC Interface PCI Register Address Map so you may require a little bit of work to figure out the addresses of these registers on your hardware.
Wednesday, 22 June 2011
Using SystemTap to do runtime AML tracing
The ACPI engine in the kernel can be debugged by building with CONFIG_ACPI_DEBUG and configuring /sys/module/acpi/parameters/debug_layer and /sys/module/acpi/parameters/debug_level appropriately. This can provide a wealth of data and is generally a very powerful debug state tracing mechanism. However, there are times when one wants to get a little more debug data out or perhaps just drill down on a specific core area of functionality without being swamped by too much ACPI debug. This is where tools like SystemTap are useful.
SystemTap is a very powerful tool that allows one to add extra debug instrumentation into a running kernel without the hassle and overhead of rebuilding a kernel with debug printk() statements in. It allows very quick turnaround in writing debug and one does not have to reboot a machine to load a new kernel since the debug is loaded and unloaded dynamically.
SystemTap has its own scripting language for writing debug scripts, but for specialised hackery it provides a mechanism ('guru mode') to embed C directly which can be called from the SystemTap script. The SystemTap language is fairly small and easy to understand and one easily becoming proficient with the language in a day.
The only downside is that one requires a .ddeb kernel package which is huge since it contains all the necessary kernel debug information.
Over the past week I have been looking at debugging various aspects of the ACPI core, such as fulling tracing suspend/resume and dumping out executed AML code at run time. I was able to quickly prototype a SystemTap script that dumps out AML opcodes on the Oneiric kernel - this saved me the usual build of a debug kernel with CONFIG_ACPI_DEBUG enabled and then capturing the appropriate debug and wading through copious amounts of debug data.
Conclusion: Some initial investment in time and effort is required to understand SystemTap (and to get to grips with the more useful features in 'guru mode'). However, one can be far more productive because the debug cycle is made far more efficient. Also, SystemTap provides plenty of functionality to allow very detailed and targeted debugging scripts.
SystemTap is a very powerful tool that allows one to add extra debug instrumentation into a running kernel without the hassle and overhead of rebuilding a kernel with debug printk() statements in. It allows very quick turnaround in writing debug and one does not have to reboot a machine to load a new kernel since the debug is loaded and unloaded dynamically.
SystemTap has its own scripting language for writing debug scripts, but for specialised hackery it provides a mechanism ('guru mode') to embed C directly which can be called from the SystemTap script. The SystemTap language is fairly small and easy to understand and one easily becoming proficient with the language in a day.
The only downside is that one requires a .ddeb kernel package which is huge since it contains all the necessary kernel debug information.
Over the past week I have been looking at debugging various aspects of the ACPI core, such as fulling tracing suspend/resume and dumping out executed AML code at run time. I was able to quickly prototype a SystemTap script that dumps out AML opcodes on the Oneiric kernel - this saved me the usual build of a debug kernel with CONFIG_ACPI_DEBUG enabled and then capturing the appropriate debug and wading through copious amounts of debug data.
Conclusion: Some initial investment in time and effort is required to understand SystemTap (and to get to grips with the more useful features in 'guru mode'). However, one can be far more productive because the debug cycle is made far more efficient. Also, SystemTap provides plenty of functionality to allow very detailed and targeted debugging scripts.
Wednesday, 15 June 2011
Dumping the contents of the Embedded Controller
Dumping the contents of the Embedded Controller (EC) can be useful when debugging some x86 BIOS/kernel related issues. At a hardware level to get access to the EC memory one goes via the EC command/status and data port. As a side note, one can determine these ports as follows:
cat /proc/ioports | grep EC
The preferred way to access these is via the ACPI EC driver in drivers/acpi/ec.c which is used by the ACPI driver to handle read/write operations to the EC memory region.
In addition to this driver, there the ec_sys module that provides a useful debugfs interface to allow one to read + write to the EC memory. Write support is enabled with the ec_sys module parameter 'write_support' but it is generally discouraged as one may be poking data into memory may break things in an unpredictable manner, hence by default write support is disabled.
So, to dump the contents of the first EC (assuming debugfs is mounted), do:
Simple!
As a bonus, the General Purpose Event bits are also readable from /sys/kernel/debug/ec/ec0/gpe.
Before I stumbled upon ec_sys.c I used a SystemTap script to execute ec_read() in ec.c to do the reading directly. Yes it's ugly and stupid, but it does prove SystemTap is a very useful tool.
cat /proc/ioports | grep EC
The preferred way to access these is via the ACPI EC driver in drivers/acpi/ec.c which is used by the ACPI driver to handle read/write operations to the EC memory region.
In addition to this driver, there the ec_sys module that provides a useful debugfs interface to allow one to read + write to the EC memory. Write support is enabled with the ec_sys module parameter 'write_support' but it is generally discouraged as one may be poking data into memory may break things in an unpredictable manner, hence by default write support is disabled.
So, to dump the contents of the first EC (assuming debugfs is mounted), do:
sudo modprobe ec_sys
sudo od -t x1 /sys/kernel/debug/ec/ec0/io
Simple!
As a bonus, the General Purpose Event bits are also readable from /sys/kernel/debug/ec/ec0/gpe.
Before I stumbled upon ec_sys.c I used a SystemTap script to execute ec_read() in ec.c to do the reading directly. Yes it's ugly and stupid, but it does prove SystemTap is a very useful tool.
Subscribe to:
Posts (Atom)