Friday 11 March 2011

Making sense of PCIe ASPM

PCI Express based serial linked devices can be managed by Active State Power Management (ASPM) to extend battery life on mobile devices such as laptops and netbooks.  ASPM is a power management protocol that allows an operating system's power management to place the link physical layer into a low power mode and it has the ability to instruct other devices on the link to go into a lower power mode too.
The plus side is that we save power with ASPM, however, it will introduce some latency as the bus needs time to be woken up when in a low power state.
The PCIe specification (version 2.0) defines two power modes:
  • L0s, which set low power mode in in direction on the link (usually from the physical link layer controller downstream)
  • L1, which sets low power mode in both directions on the link, however there is greater wakeup latency.
Section 5.2.9.3 "IA-PC Boot Architecture Flags" of the version 4.0a of the ACPI specification for the PCIe ASPM Controls (bit 4 of IAPC_BOOT_ARCH field of the FADT) states:  "If set, indicates to OSPM that it must not enable OSPM ASPM control on this platform."

In the document ASPM implementation on Windows Vista the ASPM Controls are described as follows: "If the BIOS indicates that the platform does not properly support ASPM, the operating system disables ASPM. This can be conveyed in the ACPI fixed ACPI description table (FADT) table IAPC_BOOT_ARCH flags by setting the PCIe ASPM Controls bit to 1".

Unfortunately we may have PCIe devices that are unreliable under ASPM control, hence the ASPM Control state bit is set in the ACPI FADT to inform the operating system to not control ASPM.

The Linux ASPM driver implements the nitty-gritty details of ASPM - by default it reads the ASPM config from the PCI configuration space. The ASPM config is contained in one of the Capabilities List items, the head of this is pointed to by CapPntr (offset 0x34 in the Configuration Header).

Currently Linux forcibly clears ASPM state on the device if the Control state bit is set. A recent patch by Matthew Garrett refers to the "PCI Express In Depth for Windows Vista and Beyond" in which it seems to imply that Windows won't use the PCIe features (such as ASPM) unless the full control is granted via the _OSC control.    Garrett's patch only clears ASPM state if we have _OSC control and the ASPM Control state bit is set, and his test show that this seems to save several Watts on a ThinkPad X220.  We're currently crowd-sourcing a test of this patch here.

Linux also provides some ASPM driver kernel parameters to allow some level of tweakability.  The following kernel parameters can be used:
  • "pcie_aspm=off" - disables ASPM
  • "pcie_aspm=default" - use default firmware configuration as set in the PCI Express Capabalities list item with ID 0x10
  • "pcie_aspm=performance"  - disables ASPM and clock power management
  • "pcie_aspm=powersave" - highest power saving mode, enable ASPM and clock power management
"pcie_aspm=off" has seemed to help some users with some PCIe devices that case Linux hang at boot time.  I deeply suspect that the IACP_BOOT_ARCH flag may be telling Linux to do this but it's being ignored.
I measured "pcie_aspm=powersave" on one of my netbooks and I believe I get a small amount of power saving, powertop seemed to indicate about 0.2 Watts being saved, but this could be within the margin of error in my measurements.
Anyhow, if you want to tinker with more power savings, maybe overriding the firmware configured defaults with "pcie_aspm=powersave" may help. It's worth a try.

UPDATE:

Thanks to Alex Hung for pointing out some mistakes in my original blog post.

7 comments:

  1. I am curious: Do you think the PCI spec can be used to turn off a device completely? I am asking because, I'd like to add a power-hungry GFX to my Desktop that I'd like to turn off occasionally and use the on-board ones.

    ReplyDelete
  2. why the latency? is there a way to design devices with a much faster poweron time?

    ReplyDelete
  3. @Rogelio, I'm not a hardware engineer, but I suspect the latencies are down to powering up the PLLs and getting reference clocks re-synchronized and other link related housekeeping.

    ReplyDelete
  4. Ran across this post after I rolled my own version of 3.2-rc2 with MG's patches. Booting with pcie_aspm=powersave didn't do a lot for me, however, changing /sys/module/pcie_aspm/parameters/policy to powersave shaved approx 2-3 watts off. Measured with a wattmeter on the mains plug.

    ReplyDelete
  5. @Colin,

    "PCIe ASPM Controls" does not indicate "BIOS can indicate that ASPM should be disabled".

    Instead, it means that BIOS tells OSPM, i.e. Linux or Windows, not to control ASPM settings ("If set, indicates to OSPM that it must not enable OSPM ASPM control on this platform" from ACPI 4.0a)

    I suspect the primary purpose of this bit is to workaround some faulty PCIE devices that causes the instability. BIOS can force to disable L0s and/or L1 and OSPM will not enable them. I saw many such cases before myself.

    ~Alex Hung

    ReplyDelete
  6. @Alex, many thanks for spotting my mistakes. I've re-worked the article.

    ReplyDelete
  7. I'm having trouble with ASPM on Skylake and noticed the root port says "ASPM not supported" does that mean UEFI/Linux disabled it or that the hardware itself does not support it?

    00:1b.0 0604: 8086:a167 (rev f1) (prog-if 00 [Normal decode])
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- Reset- FastB2B-
    PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
    Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
    DevCap: MaxPayload 256 bytes, PhantFunc 0
    ExtTag- RBE+
    DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
    RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
    MaxPayload 128 bytes, MaxReadReq 128 bytes
    DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
    LnkCap: Port #17, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <1us, L1 <16us
    ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
    LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
    ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
    LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-

    (was too long for comment) - Linux 4.10

    ReplyDelete