At Meta, we use sample-based profiling with
perf to guide performance optimization work – otherwise we would need even more servers than the ones we already have! This 2018 engineering blog post on BOLT is still highly relevant.
For a primer on
perf, I highly recommend reading Brendan Gregg’s perf examples page.
perf source code is part of the Linux source tree; there are standalone releases but Linux distributions tend to build it from the entire kernel tarball instead (e.g. Debian, Fedora, Ubuntu). Take note of this, it will become relevant later.
perf has excellent backward compatibility: a performance profile taken with an older version can be read by a newer version. Forward compatibility is not a given though.
And when you are in the middle of transitioning between two CentOS Stream releases, you might end up in a situation where a performance profile is taken on a system running a newer
perf (CentOS Stream 9 has 5.14.0-331.el9 while CentOS Stream 8 has 4.18.0-499.el8) – but you might want to run the analysis on a system with the older
Enter Hyperscale SIG
At Meta, we run CentOS Stream on our servers, with packages from the Hyperscale SIG layered on top. This allows us to selectively upgrade system packages as necessary -
systemd is a prime example, but also, yes,
Our initial plan was to simply maintain the same kernel for both
hsx.el8 (experimental repository for CentOS Stream 8) and
hsx.el9 (experimental repository for CentOS Stream 9).
perf is really a byproduct here; the main reason to have a kernel package in Hyperscale is to enable certain functionalities (like the Btrfs file system) that are not enabled in the stock CentOS Stream kernel.
hsx.el8 kernel is lagging behind, unfortunately (TL;DR - we need to redo the way we rebase changes from newer kernels). To the point that, as it turns out,
perf-5.14.0-76.hs2.hsx.el9 is no longer compatible with
perf-5.14.0-331.el9! Remember, the
x.y.z version number does not tell you everything, as features can be backported.
We initially thought a small fix would fix the issue, but it turns out the kernel no longer even builds thanks to Hyperscale also shipping an updated
pahole. Kudos to one of my colleagues, Stefan Roesch, for blogging about the workaround - with an adapted patch we at least have a buildable kernel again… and of course it does not solve the issue. Plus, iterating on changes is slow… being part of the kernel package means when you rebuild, you have to also rebuild the Linux kernel and all the kernel modules.
Can we just build
Yes, yes you can. Remember upstream provides standalone releases? They are even signed!
… not so easy: unfortunately trying to get this built with all the relevant distro compiler settings… takes a while to get right. Not something you want to do while you have to fix an actual issue, certainly!
Luckily, there is a middle ground: when looking to just rebuild the Fedora kernel to see if its
perf would work fine on CentOS Stream, I noticed that… aha… of course, Fedora no longer builds
perf and other tools with the kernel, it’s part of the kernel-tools source package instead.
Home run! … and… again, not so fast. Initially, everything seems fine. I have to disable
LIBBPF_DYNAMIC at least for
c8s because the system
libbpf is too old, and surprisingly, flipping
0 does not work - you have to remove the entire definition - but after that this builds in
mock against Hyperscale targets! Time to build on CBS. Drats.
- bpftool has to be disabled - it fails only on CBS. Still need to troubleshoot why.
- coresight has to be disabled on c8s
After that we finally have working builds… for both
hs.el9 … and then discovered that the
hs.el8 build pulls in newer modular Perl RPMs… while on our servers we disable modular repos by default. So these won’t install.
One last change: disable Perl support.
And we finally have builds of
perf-6.3.3 for both CentOS Stream 8 and 9, built with exactly the same settings, that anyone can install simply by enabling our repositories.
These are no longer part of our experimental kernel, so you can simply enable our main repos to get this without exposing yourself to experimental packages. And since these are based on the Fedora
kernel-tools, and Fedora runs a mostly stock kernel, you can rest assured that the code won’t deviate from what you can find in the kernel source tree for the corresponding version.
And yes, I definitely was not anticipating all the twists and turns! The journey and the destination are both worth it though.
This post is day 24 of my #100DaysToOffload challenge. Visit https://100daystooffload.com to get more info, or to get involved.