The plus side is that we save power with ASPM, however, it will introduce some latency as the bus needs time to be woken up when in a low power state.
The PCIe specification (version 2.0) defines two power modes:
- L0s, which set low power mode in in direction on the link (usually from the physical link layer controller downstream)
- L1, which sets low power mode in both directions on the link, however there is greater wakeup latency.
In the document ASPM implementation on Windows Vista the ASPM Controls are described as follows: "If the BIOS indicates that the platform does not properly support ASPM, the operating system disables ASPM. This can be conveyed in the ACPI fixed ACPI description table (FADT) table IAPC_BOOT_ARCH flags by setting the PCIe ASPM Controls bit to 1".
Unfortunately we may have PCIe devices that are unreliable under ASPM control, hence the ASPM Control state bit is set in the ACPI FADT to inform the operating system to not control ASPM.
The Linux ASPM driver implements the nitty-gritty details of ASPM - by default it reads the ASPM config from the PCI configuration space. The ASPM config is contained in one of the Capabilities List items, the head of this is pointed to by CapPntr (offset 0x34 in the Configuration Header).
Currently Linux forcibly clears ASPM state on the device if the Control state bit is set. A recent patch by Matthew Garrett refers to the "PCI Express In Depth for Windows Vista and Beyond" in which it seems to imply that Windows won't use the PCIe features (such as ASPM) unless the full control is granted via the _OSC control. Garrett's patch only clears ASPM state if we have _OSC control and the ASPM Control state bit is set, and his test show that this seems to save several Watts on a ThinkPad X220. We're currently crowd-sourcing a test of this patch here.
Linux also provides some ASPM driver kernel parameters to allow some level of tweakability. The following kernel parameters can be used:
- "pcie_aspm=off" - disables ASPM
- "pcie_aspm=default" - use default firmware configuration as set in the PCI Express Capabalities list item with ID 0x10
- "pcie_aspm=performance" - disables ASPM and clock power management
- "pcie_aspm=powersave" - highest power saving mode, enable ASPM and clock power management
I measured "pcie_aspm=powersave" on one of my netbooks and I believe I get a small amount of power saving, powertop seemed to indicate about 0.2 Watts being saved, but this could be within the margin of error in my measurements.
Anyhow, if you want to tinker with more power savings, maybe overriding the firmware configured defaults with "pcie_aspm=powersave" may help. It's worth a try.
UPDATE:
Thanks to Alex Hung for pointing out some mistakes in my original blog post.
I am curious: Do you think the PCI spec can be used to turn off a device completely? I am asking because, I'd like to add a power-hungry GFX to my Desktop that I'd like to turn off occasionally and use the on-board ones.
ReplyDeletewhy the latency? is there a way to design devices with a much faster poweron time?
ReplyDelete@Rogelio, I'm not a hardware engineer, but I suspect the latencies are down to powering up the PLLs and getting reference clocks re-synchronized and other link related housekeeping.
ReplyDeleteRan across this post after I rolled my own version of 3.2-rc2 with MG's patches. Booting with pcie_aspm=powersave didn't do a lot for me, however, changing /sys/module/pcie_aspm/parameters/policy to powersave shaved approx 2-3 watts off. Measured with a wattmeter on the mains plug.
ReplyDelete@Colin,
ReplyDelete"PCIe ASPM Controls" does not indicate "BIOS can indicate that ASPM should be disabled".
Instead, it means that BIOS tells OSPM, i.e. Linux or Windows, not to control ASPM settings ("If set, indicates to OSPM that it must not enable OSPM ASPM control on this platform" from ACPI 4.0a)
I suspect the primary purpose of this bit is to workaround some faulty PCIE devices that causes the instability. BIOS can force to disable L0s and/or L1 and OSPM will not enable them. I saw many such cases before myself.
~Alex Hung
@Alex, many thanks for spotting my mistakes. I've re-worked the article.
ReplyDeleteI'm having trouble with ASPM on Skylake and noticed the root port says "ASPM not supported" does that mean UEFI/Linux disabled it or that the hardware itself does not support it?
ReplyDelete00:1b.0 0604: 8086:a167 (rev f1) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #17, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <1us, L1 <16us
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
(was too long for comment) - Linux 4.10