NVRM: API mismatch after a partial upgrade or an un-rebooted kernel update
Everything load-bearing checks out. Arch System_maintenance line 89 says verbatim "never run pacman -Sy; instead, always use pacman -Syu" and lines 101-102 name pacman -Sy package and -Sy then -S as the two unsupported forms, which are exactly the record's two desync paths. I read bin/omarchy-update-pacman-guard on quattro and the analysis is precisely right: it sets is_system_upgrade only when (( has_sync && has_sysupgrade )), so sudo pacman -Sy nvidia exits 0 and passes through unchecked. kernel-modules-hook is real (extra 0.1.7) and is even an optdepend of limine-mkinitcpio-hook. limine-mkinitcpio and limine-update are real binaries (gitlab.com/Zesko/limine-entry-tool, install/arch-linux/limine-mkinitcpio-hook/usr/bin/). dkms remove nvidia/<ver> --all, dkms autoinstall -k, dkms status, /proc/driver/nvidia/version and modinfo -F version nvidia are all valid. Limine's own CONFIG.md confirms editor_enabled "Defaults to yes", and per-entry cmdline is a real option, so the TTY/nomodeset escape hatch is sound. No destructive commands. The dkms removal is by explicit version, not an rm -rf.
Do NOT "fix" a partial upgrade by symlinking library sonames. The Arch wiki warns explicitly against this. Sonames are bumped precisely because they are incompatible. If the -Syu that follows the bad -Sy fails halfway, you are now in a genuine partial-upgrade state and must resolve the error and finish the transaction before running any other pacman operation. Booting with nomodeset gives you a low-resolution text console only. Do not leave it in the permanent command line. dkms remove --all deletes built modules for every kernel. If you remove the wrong version you lose the working module too, so read dkms status first.
Symptom
After an update the desktop never comes back. You get a black screen, or you get dumped at a TTY. nvidia-smi says:
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
and sudo dmesg | grep NVRM shows:
NVRM: API mismatch: this kernel module has the version 580.95.05, but
NVRM: this NVIDIA driver component has the version 590.48. Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.
Or modprobe just cannot find it at all:
modprobe: FATAL: Module nvidia not found in directory /lib/modules/6.18.4-arch1-1
Cause
The NVIDIA driver ships as two halves that must be the exact same version: the userspace libraries (nvidia-utils) and the kernel module (nvidia, nvidia-dkms, nvidia-open-dkms). Two things desynchronise them.
- A partial upgrade.
pacman -Sy nvidia, orpacman -Syfollowed bypacman -S <anything>, refreshes the database and pulls one half forward while leaving the other behind. The Arch wiki is explicit that partial upgrades are unsupported and that you must never runpacman -Sy. On Omarchy this is not covered by the update guard:bin/omarchy-update-pacman-guardonly aborts when both a sync flag and a sysupgrade flag are present, sosudo pacman -Sy nvidiasails straight past it.
- A kernel package upgraded underneath a running kernel. Pacman deletes
/usr/lib/modules/<old-version>/when it installs the new kernel, so the still-running kernel loses every module it has not already loaded. That produces theModule nvidia not found in directory /lib/modules/$(uname -r)variant until you reboot.
A third, rarer variant: a stale DKMS build from an older driver version still sitting in /usr/lib/modules/<kernel>/updates/dkms/ shadowing the correctly-versioned packaged module.
This is distinct from a DKMS *build failure*. Here the build succeeded, the versions just do not line up.
Fix
**1. Confirm the mismatch before touching anything.**
uname -r
pacman -Q linux linux-headers nvidia-utils nvidia-dkms nvidia-open-dkms 2>/dev/null
cat /proc/driver/nvidia/version 2>/dev/null # what the loaded module thinks it is
modinfo -F version nvidia 2>/dev/null # what the on-disk module is
dkms status
If /proc/driver/nvidia/version and pacman -Q nvidia-utils disagree, it is a version skew. If modinfo says "module not found", the running kernel lost its modules.
**2. Complete the upgrade properly, never pacman -Sy <pkg>.**
# Omarchy 4:
omarchy update
# plain Arch / EndeavourOS / CachyOS / Manjaro:
sudo pacman -Syu
If you are stuck on a black screen with no shell: press Ctrl+Alt+F2 for a TTY, or at the Limine menu select the entry and add nomodeset to the command line (Limine's editor, with the key hints on the menu's help line) to get a text console, then run the upgrade there.
**3. Clear a stale DKMS build if one is shadowing the packaged module.**
dkms status
find /usr/lib/modules -path '*updates/dkms*' -name 'nvidia*.ko*' -printf '%p\n'
# remove the old version by name, e.g.:
sudo dkms remove nvidia/580.95.05 --all
sudo dkms autoinstall -k "$(uname -r)"
**4. Rebuild the initramfs so early-KMS modules match, then reboot.**
# Omarchy 4 (Limine + UKI):
sudo limine-mkinitcpio
sudo limine-update
# plain mkinitcpio systems:
sudo mkinitcpio -P
sudo reboot
**5. Stop it happening again.** Install the hook that keeps the running kernel's modules on disk across an upgrade:
sudo pacman -S kernel-modules-hook
Verify
After reboot: nvidia-smi prints the device table, cat /proc/driver/nvidia/version reports the same version as pacman -Q nvidia-utils, cat /sys/module/nvidia_drm/parameters/modeset returns Y, and dmesg | grep -i 'API mismatch' returns nothing.
Sources
- https://wiki.archlinux.org/title/System_maintenance
- https://wiki.archlinux.org/title/General_troubleshooting
- https://bbs.archlinux.org/viewtopic.php?id=291394
- https://bbs.archlinux.org/viewtopic.php?id=261042
- https://github.com/basecamp/omarchy/blob/quattro/bin/omarchy-update-pacman-guard
- https://wiki.archlinux.org/title/NVIDIA