Bisecting a kernel DPM regression on 12-year-old hardware, with the upstream maintainer along for the ride
Published on September 24, 2026
This one starts with the same GPUs from the
passthrough guide: a pair of
AMD FirePro D500 (Tahiti, GCN 1.0, 2013) inside a
Mac Pro 6,1 that now runs Proxmox as a home server. Once
passthrough worked, the obvious next question was whether one of those GPUs
could pull its weight running a local LLM — llama.cpp
over Vulkan, via the open-source RADV driver.
It could, until a long prompt made it crash the whole
VM with a vk::DeviceLostError. Reporting that crash upstream
opened a five-day thread with a RADV maintainer that eventually split into
three separate bugs, cost the home server seven unexplained hard crashes,
and ended with my own name in a kernel commit message next to the words
“FirePro D500.”
This is the story of the middle bug: a GPU that would not leave first gear.
tok/s lost when the GPU got stuck in its lowest power state
exact commit isolated, out of 5 candidates, over 8 reboots
full host crashes along the way, with zero trace in the logs
patches sent to the amd-gfx mailing list, awaiting merge
The original bug was straightforward to describe:
feed llama.cpp a prompt past roughly 600–800 tokens on
the D500 over RADV, and the compute context died with
vk::DeviceLostError. I filed it against the RADV maintainer
Timur Kristóf on Mesa’s GitLab. His diagnosis:
outdated Mesa, plus a GFX6 soft-reset fix of his own that had just landed in
Linux 7.3 and should fix it.
It did — a 1161-token prompt that used to hang the whole host now ran clean, on one GPU and on two. But validating a fix on a release-candidate kernel meant booting into it, and that is where the collateral damage started. Passing the GPU through to a desktop VM under 7.3-rc3 now triggered a PCIe AER error during the GPU reset — AER stands for Advanced Error Reporting, the PCIe bus’s own mechanism for flagging corrupted or lost traffic on the link — and the guest driver would shut the device down mid-session, reproduced 4 times out of 4 on the RC kernel, 0 times out of 1 on production. That report is still sitting in a drawer (see the closing section), because the very next thing I found made it look small by comparison.
Before closing the original issue, Timur asked for an apples-to-apples comparison: same command, same prompt, same GPU, only the kernel changing. The numbers did not match my assumption at all.
A quick note on the notation: P0, P3,
and so on are the GPU’s DPM (dynamic power management) states —
P0 is idle/lowest clocks, and the number climbs as the card
ramps up for load, up to whatever the highest state supported here is
(P3). A healthy card idles at P0 and jumps to a
higher state within a couple of seconds of real work landing on it.
| Production kernel | 7.3-rc3 | |
|---|---|---|
| DPM under load | P0 → P3 in ~2s (sclk 30 → 72.5, mclk 15 → 127) | stuck at P0 the entire time |
| Generation speed | 21.7 tok/s | 6.1 tok/s |
The throughput drop I had been blaming on a longer
prompt had nothing to do with context length. The card’s dynamic
power management (DPM) simply never left its lowest clock
state under the new kernel — confirmed by polling
amdgpu_pm_info once a second while the load ran.
Timur asked for three things: force the highest power
state by hand, bisect between kernel 6.19 and 7.3, and write down exact
repro steps. Forcing the state turned out to be the cleanest signal of all
— on production, echo high > power_dpm_force_performance_level
is accepted; on 7.3-rc3, it returns EINVAL immediately
(Linux’s generic “invalid argument” error code — the
kernel refusing the request outright, not a value formatted wrong).
No throughput measurement required, just an error code.
# production kernel: accepted echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level # 7.3-rc3: Invalid argument echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level -bash: echo: write error: Invalid argument
A release-level bisect narrowed it to between 7.1
(behaves like production) and 7.2 (already broken). Timur then pointed out
that only a handful of commits had touched si_dpm.c in that
window and asked me to revert them and find the guilty one. I found five
candidates, not four — and rather than argue about the discrepancy,
just tested all five.
This meant compiling a custom kernel locally for the first time in this project (on a spare 24-core LXC container — the Mac Pro’s own Xeon, ironically, building the kernel that would later fix its own GPU). Eight reboots and a binary search later, one commit stood out as necessary and sufficient on its own:
e6c5d36756e7 drm/amd/pm/si: Fix updating clock limits from power states
A patch that, going by its own commit message, was meant to fix laptops. It broke a desktop instead.
Timur asked for two dmesg captures with driver
debugging enabled, one on each side of the commit, to see the actual
difference in the DPM tables sent to the card’s embedded controller
(the SMC). Reading through si_dpm.c with those logs in hand,
the chain became visible: the D500’s VBIOS ships a
battery power state with clocks locked at the
minimum — harmless on its own, since nothing on a desktop should ever
select it. The new commit started using that state to populate the
driver’s notion of “maximum clocks allowed on DC (battery)
power,” which had previously fallen back to the generous AC limits.
With real, tiny DC limits now in place, the driver marked the
high-performance power state as not DC-compatible —
unconditionally, with no regard for whether the machine was actually
running on battery.
My first fix attempt, suggested by Timur as an
isolation test, was to hardcode that compatibility flag to
true. It worked completely: 21.3 tok/s, DPM
climbing to P3 exactly like production. I posted it as a positive result.
Timur pushed back anyway: that flag, he said, “shouldn’t matter
unless the SMC thinks it’s running on battery” — and a
Mac Pro is never running on battery. My hardcode was papering over a
different flag entirely, one neither of us had been looking at.
The driver deals with AC/DC in two unrelated places, and the names are close enough to actively mislead:
| Flag | Scope | What it actually does |
|---|---|---|
dc_compatible / PPSMC_SWSTATE_FLAG_DC |
Per power-state | The one everyone (myself included) assumed mattered. Turned out to be irrelevant on this hardware. |
PPSMC_SYSTEMFLAG_GPIO_DC |
System-wide, set once at boot | Tells the SMC “trust a physical GPIO pin to know if you’re on battery.” The real gate. |
The system-wide flag is set whenever the VBIOS
declares the HARDWAREDC platform capability — which the
D500’s VBIOS does, despite the card living in a desktop with no
battery and no such GPIO wired to anything. With that flag set, the SMC
trusts a pin that permanently reads “not AC,” and ignores
everything else: the per-state flag, and any software notification.
Two isolation builds nailed it down. Both cleared
PPSMC_SYSTEMFLAG_GPIO_DC and left dc_compatible
at its normal (broken) value; one of them also sent an explicit
“running on AC” message to the SMC, the other sent nothing at
all. Both fully fixed the regression. Neither the per-state flag nor the
software notification mattered in the slightest — only the
system-wide GPIO flag did.
Before posting that finding, there was an obvious
objection to anticipate: unconditionally clearing
PPSMC_SYSTEMFLAG_GPIO_DC whenever HARDWAREDC is
set would also break real laptops with a genuinely wired GPIO. So the
report included a proposed scope — gate the fix on this card’s
PCI subsystem ID, Apple’s 106b:0126.
Timur agreed with the root cause point for point and
proposed a cleaner scope of his own: gate it on the
AMD_IS_MOBILITY device flag instead, which the driver already
uses to tell laptop SKUs from desktop ones — covering any GCN1 desktop
board with a similarly confused VBIOS, not just this one Mac Pro.
- if (adev->pm.dpm.platform_caps & ATOM_PP_PLATFORM_CAP_HARDWAREDC) + if ((adev->flags & AMD_IS_MOBILITY) && + (adev->pm.dpm.platform_caps & ATOM_PP_PLATFORM_CAP_HARDWAREDC)) table->systemFlags |= PPSMC_SYSTEMFLAG_GPIO_DC;
Two lines. Tested on a clean release-candidate tree
with no other patch applied: auto now climbs to P3 at 21.32
tok/s, matching production exactly; high forced is accepted,
no more EINVAL. Full parity restored.
Timur folded the fix into a proper commit, citing the FirePro D500 by name in the message, and sent a two-patch series to the amd-gfx mailing list — the actual review path into the mainline kernel. The second patch is a complementary hardening of the same code, not something I tested myself; both are currently awaiting review.
Two threads from this same week never got resolved.
The PCIe AER bug from the very first section — a passthrough GPU
reset that goes fatal under the RC kernel — has a full writeup with
dmesg from both host and guest, but I still haven’t decided whether it
belongs on bugzilla.kernel.org or on the
linux-pci mailing list, so it hasn’t been posted yet.
And along the way, the home server itself hard-crashed
seven times with absolutely nothing in the logs — no panic, no OOM, no
AER, pstore empty every time. Most of those correlated with a
specific sequence in my own tooling (binding and unbinding the GPU between
the kernel driver and vfio-pci in quick succession around a
reboot), and removing that dangerous transition from the automation fixed
it structurally. The underlying kernel-level cause of why that
sequence hangs the host, though, was never root-caused — it was simply
avoided.
dc_compatible hardcode fixed the symptom completely. It took
a maintainer who understood the hardware better than the test results did
to say “that shouldn’t be the reason” and send everyone
looking one layer deeper.power_dpm_force_performance_level high gave a hard
EINVAL instead of a throughput number, every subsequent test
took seconds instead of a full inference run.A note on how this was written: the investigation, decisions, and testing described above are my own. The writing itself was drafted and edited with the help of generative AI (Claude), based on my notes and the real project history.