Suspend and Resume - always fun to debug

Debugging suspend/resume issues is always non-trivial when a machine hangs up during the resume. Generally, one would like to save some debug state in the resume stage which gets saved when one has to reboot a hung machine. However, the only real place one can shove state is in the real time clock (RTC) which gives one only ~24 bits of debug state.

Ubuntu kernels have PM_TRACE debug enabled, allowing one to turn on debug tracing via:

echo 1 > /sys/power/pm_trace

When a machine then hangs in the resume stage some magic is stashed in the RTC and then one has a ~3 minute window to reboot the machine. On the reboot, the hashed magic stored in the RTC is converted back into a debug message which can be read using the dmesg command and looking for the "Magic number" message:

$ dmesg | less

Magic number is: 13:773:539
hash matches device usbdev2.1_ep00

This usually provides enough information to allow one to start corning the issue.