OPINIONATED OMARCHY
TROUBLESHOOTING CORPUS
790
RECORDS
790
AUDITED
0
UNCHECKED
2388
SOURCES
LIVE
apps-services medium common DISPUTE

Fix Podman GPU containers after an NVIDIA driver update (stale CDI spec)

omarchyarchendeavouroscachyosmanjaropodmannvidiadesktoplaptop
CORRECTED An auditor found the fix wrong or incomplete and rewrote it.
AUDIT NOTE

The problem and the remedy are right: I confirmed nvidia-container-toolkit ships usr/share/libalpm/hooks/nvidia-ctk-cdi.hook and usr/share/libalpm/scripts/nvidia-ctk-cdi, and that the package contains no nvidia-cdi-refresh.service, so the Arch-specific framing holds and nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml is exactly what the hook's own warning tells you to run. But the cause misstates when the spec goes stale: I read the hook, and it triggers on Install AND Upgrade of nvidia-utils, nvidia-container-toolkit, opencl-nvidia, egl-gbm and egl-wayland, so 'the driver was installed after the toolkit' is precisely the case the hook does handle. The real fragility is that when the version changed the script does not regenerate: it rewrites /etc/cdi/nvidia.yaml with a plain sed string substitution of the old libcuda version, and prints a warning saying to regenerate by hand if problems appear.

The cause above was not rewritten and may still contain the error described. The fix below is the corrected version.
RISK

A regenerated spec only matches the driver that is loaded *right now*. If you regenerate it after a pacman -Syu but before rebooting into the new kernel/driver, it will break again on reboot. Regenerate after the reboot, or just let the pacman hook do it and reboot.

Symptom

podman run --rm --device nvidia.com/gpu=all archlinux nvidia-smi -L fails with Error: setting up CDI devices: unresolvable CDI devices nvidia.com/gpu=all, or the container starts but nvidia-smi inside it reports Failed to initialize NVML: Driver/library version mismatch. It worked before the last update.

Cause

Podman resolves nvidia.com/gpu=... through a CDI spec, on Arch /etc/cdi/nvidia.yaml, maintained by the nvidia-ctk-cdi pacman hook in nvidia-container-toolkit. The hook does fire on nvidia-utils/nvidia-container-toolkit/opencl-nvidia/egl-* install and upgrade, but when it detects a driver version change it does not regenerate the spec. It patches the old version string in place with sed and warns you to regenerate manually. So the spec is stale or mangled when that substitution went wrong, when the hook was skipped (pacman --nohooks, or a driver installed outside pacman / an out-of-band DKMS rebuild), or when the file was never generated. There is no nvidia-cdi-refresh.service on Arch to fix it up at boot.

Fix

Same commands, with one extra check first. Compare the driver version baked into the spec against the running driver, because the pacman hook may have sed-patched it rather than regenerating:


nvidia-ctk cdi list
grep -m1 'libcuda.so' /etc/cdi/nvidia.yaml
readlink -f /usr/lib/libcuda.so

If those versions disagree, or the paths in the file do not exist, regenerate rather than patch:


sudo mkdir -p /etc/cdi
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
nvidia-ctk cdi list

Then test:


podman run --rm --device nvidia.com/gpu=all --security-opt=label=disable archlinux nvidia-smi -L

If nvidia-ctk cdi generate itself fails, the host driver is the problem (nvidia-smi, lsmod | grep ^nvidia, pacman -Q nvidia-utils nvidia-container-toolkit). nvidia-ctk --debug cdi list for names that still will not resolve. Do not combine CDI GPUs with --userns nomap or --userns auto.

Verify

nvidia-ctk cdi list shows nvidia.com/gpu=all plus one entry per GPU, and the test container prints your GPU's UUID line.

Sources

podman-nvidia-cdi-spec-stale