At Meta, we use sample-based profiling with perf to guide performance optimization work – otherwise we would need even more servers than the ones we already have! This 2018 engineering blog post on BOLT is still highly relevant.

For a primer on perf, I highly recommend reading Brendan Gregg’s perf examples page.

The perf source code is part of the Linux source tree; there are standalone releases but Linux distributions tend to build it from the entire kernel tarball instead (e.g. Debian, Fedora, Ubuntu). Take note of this, it will become relevant later.

The problem

perf has excellent backward compatibility: a performance profile taken with an older version can be read by a newer version. Forward compatibility is not a given though.

And when you are in the middle of transitioning between two CentOS Stream releases, you might end up in a situation where a performance profile is taken on a system running a newer perf (CentOS Stream 9 has 5.14.0-331.el9 while CentOS Stream 8 has 4.18.0-499.el8) – but you might want to run the analysis on a system with the older perf…

Enter Hyperscale SIG

At Meta, we run CentOS Stream on our servers, with packages from the Hyperscale SIG layered on top. This allows us to selectively upgrade system packages as necessary - systemd is a prime example, but also, yes, perf.

Our initial plan was to simply maintain the same kernel for both hsx.el8 (experimental repository for CentOS Stream 8) and hsx.el9 (experimental repository for CentOS Stream 9). perf is really a byproduct here; the main reason to have a kernel package in Hyperscale is to enable certain functionalities (like the Btrfs file system) that are not enabled in the stock CentOS Stream kernel.

The hsx.el8 kernel is lagging behind, unfortunately (TL;DR - we need to redo the way we rebase changes from newer kernels). To the point that, as it turns out, perf-5.14.0-76.hs2.hsx.el9 is no longer compatible with perf-5.14.0-331.el9! Remember, the x.y.z version number does not tell you everything, as features can be backported.

We initially thought a small fix would fix the issue, but it turns out the kernel no longer even builds thanks to Hyperscale also shipping an updated pahole. Kudos to one of my colleagues, Stefan Roesch, for blogging about the workaround - with an adapted patch we at least have a buildable kernel again… and of course it does not solve the issue. Plus, iterating on changes is slow… being part of the kernel package means when you rebuild, you have to also rebuild the Linux kernel and all the kernel modules.

Can we just build perf itself?

Yes, yes you can. Remember upstream provides standalone releases? They are even signed!

… not so easy: unfortunately trying to get this built with all the relevant distro compiler settings… takes a while to get right. Not something you want to do while you have to fix an actual issue, certainly!

Luckily, there is a middle ground: when looking to just rebuild the Fedora kernel to see if its perf would work fine on CentOS Stream, I noticed that… aha… of course, Fedora no longer builds perf and other tools with the kernel, it’s part of the kernel-tools source package instead.

Home run! … and… again, not so fast. Initially, everything seems fine. I have to disable LIBBPF_DYNAMIC at least for c8s because the system libbpf is too old, and surprisingly, flipping LIBBPF_DYNAMIC=1 to 0 does not work - you have to remove the entire definition - but after that this builds in mock against Hyperscale targets! Time to build on CBS. Drats.

After that we finally have working builds… for both hs.el8 and hs.el9 … and then discovered that the hs.el8 build pulls in newer modular Perl RPMs… while on our servers we disable modular repos by default. So these won’t install.

One last change: disable Perl support.

And we finally have builds of perf-6.3.3 for both CentOS Stream 8 and 9, built with exactly the same settings, that anyone can install simply by enabling our repositories.

These are no longer part of our experimental kernel, so you can simply enable our main repos to get this without exposing yourself to experimental packages. And since these are based on the Fedora kernel-tools, and Fedora runs a mostly stock kernel, you can rest assured that the code won’t deviate from what you can find in the kernel source tree for the corresponding version.

And yes, I definitely was not anticipating all the twists and turns! The journey and the destination are both worth it though.

This post is day 24 of my #100DaysToOffload challenge. Visit https://100daystooffload.com to get more info, or to get involved.

Have a comment on one of my posts? Start a discussion in my public inbox by sending an email to ~michel-slm/public-inbox@lists.sr.ht [mailing list etiquette]

Posts are also tooted to @michel_slm@floss.social or @michel_slm@social.coop