Fix Docker refusing --gpus all with "could not select device driver"
Checked on this Omarchy 4 workstation (omarchy 4.0.2-1, kernel 7.1.9-arch1-2, docker 1:29.7.2-1, nvidia-open-dkms 610.57.04-1, RTX 3090) and against four primary sources. Confirmed on this machine: the problem is reproducible by construction, because /usr/share/omarchy/install/omarchy-base.packages lists docker, docker-buildx and docker-compose, /usr/share/omarchy/install/hardware/nvidia.sh installs nvidia-open-dkms and nvidia-utils when lspci sees an NVIDIA card, no Omarchy package list mentions nvidia-container-toolkit, pacman -Q nvidia-container-toolkit reports it is not installed, /etc/cdi does not exist, no hook binary is on PATH, and nvidia-smi works and reports the card. Two defects. First, the cause is wrong: --gpus does not require a runtimes entry in /etc/docker/daemon.json. moby's daemon/devices_nvidia_linux.go (read from the moby repo) registers its nvidia device driver purely from an exec.LookPath for nvidia-cdi-hook or nvidia-container-runtime-hook and returns nil when neither exists, and the Arch wiki says the same, that after installing the package you can use --gpus OR register the runtime. Second and worse on Omarchy 4: /etc/docker/daemon.json already exists and is owned by omarchy-settings 4.0.2-1 (confirmed with pacman -Qo), carrying "dns": ["172.17.0.1"] and "bip": "172.17.0.1/16", so pasting the record's whole-file runtimes-only JSON deletes exactly the two keys that the corpus record docker-container-dns-blocked-by-ufw and /usr/share/omarchy/install/config/firewall.sh depend on, and breaks container DNS everywhere. That is correct generic Arch advice mis-applied to Omarchy 4, because on plain Arch the file does not exist. The nvidia-ctk path is safe: the toolkit's pkg/config/engine/docker/option.go loadConfig reads and json-decodes the existing daemon.json before AddRuntime merges the key, confirmed from source. Third, the verify is broken on stock Omarchy: docker info here prints Server: permission denied while trying to connect to the docker API at unix:///var/run/docker.sock and docker info | grep -i runtime matches nothing, because /usr/share/omarchy/install/config/docker.sh deliberately leaves the user out of the docker group, so every docker command in the fix needs sudo. What held: all package paths are right, confirmed against the Arch package file list, which ships /usr/bin/nvidia-ctk, /usr/bin/nvidia-container-runtime, /usr/bin/nvidia-container-runtime-hook and /etc/nvidia-container-runtime/config.toml. The wiki supports the NVML --device workaround verbatim, and all three device nodes exist here. The Arch PKGBUILD and its nvidia-ctk-cdi.hook confirm the CDI file is generated automatically on install and refreshed on nvidia-utils upgrades, which is new information worth having in the fix. All four cited URLs return 200 and support their claims, so nothing is removed, but the BBS thread is a single post with no replies, so the Docker Desktop branch rests on one unanswered report and is now scoped as such. Severity and frequency left alone: the consequence is still a failed container, not a damaged machine, and the new destructive path belongs in danger. NOT exercised: I have no sudo, so I could not install the toolkit, restart dockerd, run nvidia-ctk runtime configure, or start any container. The merge behaviour of nvidia-ctk is read from its source rather than observed, and the NVML workaround is untested here.
On Omarchy 4 /etc/docker/daemon.json is owned by omarchy-settings and already sets dns and bip. Overwriting it with a runtimes-only file removes both and breaks DNS in every container, because Omarchy's ufw rules permit port 53 only to 172.17.0.1. Use sudo nvidia-ctk runtime configure --runtime=docker, which merges into the existing file, or add the key by hand and keep everything else. Being a packaged file it can also arrive as a .pacnew during omarchy update, so a hand-edited copy has to be merged rather than ignored. Whatever you edit, the file must stay valid JSON. A stray trailing comma makes docker.service fail to start with every container down, so validate with python3 -m json.tool /etc/docker/daemon.json before restarting. dockerd also refuses to start if the same option is set both in daemon.json and as a flag in the unit.
Symptom
Any GPU container fails immediately:
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
nvidia-smi on the host works fine. Sometimes the container starts but then prints Failed to initialize NVML: Unknown Error.
Cause
Docker has no built-in NVIDIA support. dockerd registers its nvidia device driver only when one of the NVIDIA Container Toolkit hook binaries is on its PATH: moby's daemon/devices_nvidia_linux.go runs exec.LookPath for nvidia-cdi-hook and for nvidia-container-runtime-hook, and when it finds neither it returns no device driver at all, which is the could not select device driver message. On Arch those binaries come from nvidia-container-toolkit, a separate package that nothing pulls in. Omarchy 4 installs docker, docker-buildx, docker-compose and the NVIDIA driver, but never nvidia-container-toolkit, so an Omarchy machine with a working nvidia-smi still fails every GPU container. Registering an nvidia entry under runtimes in /etc/docker/daemon.json is a different feature: it is what --runtime=nvidia selects, and --gpus does not need it.
Fix
Install the toolkit and restart the daemon:
sudo pacman -S --needed nvidia-container-toolkit
sudo systemctl restart docker.service
That is the whole fix for --gpus. The package ships /usr/bin/nvidia-container-runtime-hook and /usr/bin/nvidia-cdi-hook, and its ALPM hook /usr/share/libalpm/hooks/nvidia-ctk-cdi.hook regenerates /etc/cdi/nvidia.yaml on install and on every nvidia-utils upgrade.
Omarchy does not put your user in the docker group, so run the client under sudo unless you opted in through Setup > Security > Sudoless Docker:
sudo docker run --rm --gpus all nvidia/cuda:12.1.1-runtime-ubuntu22.04 nvidia-smi
If you also want --runtime=nvidia, register the runtime with nvidia-ctk rather than by hand:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker.service
nvidia-ctk reads the existing /etc/docker/daemon.json, decodes it, and adds only the runtimes key, so the rest of the file survives.
**Do not paste a whole-file daemon.json over the top on Omarchy 4.** The file is shipped by omarchy-settings and already carries the daemon's DNS and bridge settings:
pacman -Qo /etc/docker/daemon.json # omarchy-settings
cat /etc/docker/daemon.json
{
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "5" },
"dns": ["172.17.0.1"],
"bip": "172.17.0.1/16"
}
Dropping dns and bip breaks name resolution in every container, because Omarchy's ufw rules allow port 53 only to 172.17.0.1. Add the key and keep the rest:
{
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "5" },
"dns": ["172.17.0.1"],
"bip": "172.17.0.1/16",
"runtimes": {
"nvidia": {
"path": "/usr/bin/nvidia-container-runtime",
"runtimeArgs": []
}
}
}
On plain Arch, EndeavourOS or CachyOS the file usually does not exist at all, and the Arch wiki's whole-file form is safe there.
If you get Failed to initialize NVML: Unknown Error instead, pass the device nodes explicitly:
sudo docker run --rm --gpus all \
--device /dev/nvidiactl:/dev/nvidiactl \
--device /dev/nvidia-uvm:/dev/nvidia-uvm \
--device /dev/nvidia0:/dev/nvidia0 \
nvidia/cuda:12.1.1-runtime-ubuntu22.04 nvidia-smi
If you installed Docker Desktop yourself, which Omarchy does not ship, the same error can mean you are talking to Desktop's VM, and that VM has no host GPU:
docker context ls
docker context use default
Verify
sudo docker run --rm --gpus all nvidia/cuda:12.1.1-runtime-ubuntu22.04 nvidia-smi prints your GPU. Do not use docker info | grep -i runtimes as the test. It reports the --runtime=nvidia registration rather than --gpus support, and on a stock Omarchy 4 install docker info cannot read the Server section at all, printing permission denied while trying to connect to the docker API at unix:///var/run/docker.sock, because your user is not in the docker group. Under sudo, sudo docker info | grep -i runtimes lists nvidia only if you ran nvidia-ctk runtime configure.
Sources
- https://wiki.archlinux.org/title/Docker
- https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
- https://bbs.archlinux.org/viewtopic.php?id=300693
- https://archlinux.org/packages/extra/x86_64/nvidia-container-toolkit/files/
- https://archlinux.org/packages/extra/x86_64/nvidia-container-toolkit/json/
- https://github.com/moby/moby/blob/master/daemon/devices_nvidia_linux.go
- https://github.com/NVIDIA/nvidia-container-toolkit/blob/main/pkg/config/engine/docker/option.go
- https://github.com/NVIDIA/nvidia-container-toolkit/blob/main/pkg/config/engine/docker/docker.go
- https://gitlab.archlinux.org/archlinux/packaging/packages/nvidia-container-toolkit/-/raw/main/PKGBUILD
- https://gitlab.archlinux.org/archlinux/packaging/packages/nvidia-container-toolkit/-/raw/main/nvidia-ctk-cdi.hook