The early boot requires loading and decompressing the kernel and initramfs from the boot storage device. This speed is dependent on several factors, speed of loading an image from the boot device, the CPU and memory/cache speed for decompression and the compression type.
Generally speaking, the smallest (best) compression takes longer to decompress due to the extra complexity in the compression algorithm. Thus we have a trade-off between load time vs decompression time.
For slow rotational media (such as a 5400 RPM HDD) with a slow CPU the loading time can be the dominant factor. For faster devices (such as a SSD) with a slow CPU, decompression time may be the dominate factor. For devices with fast 7200-10000 RPM HDDs with fast CPUs, the time to seek to the data starts to dominate the load time, so load times for different compressed kernel sizes is only slightly different in load time.
The Ubuntu kernel team ran several experiments benchmarking several x86 configurations using the x86 TSC (Time Stamp Counter) to measure kernel load and decompression time for 6 different compression types: BZIP2, GZIP, LZ4, LZMA, LZMO and XZ. BZIP2, LZMA and XZ are slow to decompress so they got ruled out very quickly from further tests.
In compression size, GZIP produces the smallest compressed kernel size, followed by LZO (~16% larger) and LZ4 (~25% larger). With decompression time, LZ4 is over 7 times faster than GZIP, and LZO being ~1.25 times faster then GZIP on x86.
In absolute wall-clock times, the following kernel load and decompress results were observed:
Lenovo x220 laptop, 5400 RPM HDD:
LZ4 best, 0.24s faster than the GZIP total time of 1.57s
Lenovo x220 laptop, SSD:
LZ4 best, 0.29s faster than the GZIP total time of 0.87s
Xeon 8 thread desktop with 7200 RPM HDD:
LZ4 best, 0.05s faster than the GZIP total time of 0.32s
VM on a Xeon 8 thread desktop host with SSD RAID ZFD backing store:
LZ4 best, 0.05s faster than the GZIP total time of 0.24s
Even with slow spinning media
and a slow CPU, the longer load time of the LZ4 kernel is overcome by
the far faster decompression time.
As media gets faster, the load time difference between GZIP, LZ4 and
LZO diminishes and the decompression time becomes the dominant speed
factor with LZ4 the clear winner.
For Ubuntu 19.10 Eoan Ermine, LZ4 will be the default decompression for x86, ppc64el and s390 kernels and for the initramfs too.
References:
Analysis: https://kernel.ubuntu.com/~cking/boot-speed-eoan-5.3/kernel-compression-method.txt
Data: https://kernel.ubuntu.com/~cking/boot-speed-eoan-5.3/boot-speed-compression-5.3-rc4.ods
Which LZ4 level do you use? -9 seemed to be optimal in my tests.
ReplyDeleteHi Colin,
ReplyDeleteI'm a little confused about what I'm seeing here. I boot Xen PV guests by booting a grub image that's compiled as a PV Xen guest, and that pvgrub then fishes in the guest's block devices for a grub/cfg which it parses.
This has worked fine for several years.
I didn't look at 19.10 because I'm not generally an ubuntu user, but looking at 20.04, these LZ4-compressed kernels now no longer boot for me. They do this when they parse the guest's grub.cfg:
Loading Linux 5.4.0-26-generic ...
error: not xen image.
Loading initial ramdisk ...
error: you need to load the kernel first.
I have verified that this is definitely because of problems with LZ4 compression support because I decompressed that kernel, and I can boot the decompressed image fine.
So my question is, where does the problem lie? I think it must be in my copy of grub, but I fail to find any evidence that upstream grub supports LZ4 compression either. Is there any chance you could point me to a link to the source file where that support was added? Or where Ubuntu added the support?
I am aware of this bug in the Xen hypervisor:
https://lore.kernel.org/xen-devel/20191206062535.GA13261@gentoo-tp.home/T/
but as far as I can see that is related only to direct kernel load, i.e. where the kernel lives in the dom0 filesystem and is referenced directly in the guest's configuration. As mentioned, I use a grub image as a guest kernel and this finds the guest's kernel.
Thanks,
Andy