OPINIONATED OMARCHY
TROUBLESHOOTING CORPUS
790
RECORDS
790
AUDITED
0
UNCHECKED
2388
SOURCES
LIVE
apps-services medium very-common DISPUTE

Fix Docker refusing --gpus all with "could not select device driver"

omarchyarchendeavouroscachyosmanjarodockernvidiadesktoplaptop
CORRECTED An auditor found the fix wrong or incomplete and rewrote it.
AUDIT NOTE

Checked on this Omarchy 4 workstation (omarchy 4.0.2-1, kernel 7.1.9-arch1-2, docker 1:29.7.2-1, nvidia-open-dkms 610.57.04-1, RTX 3090) and against four primary sources. Confirmed on this machine: the problem is reproducible by construction, because /usr/share/omarchy/install/omarchy-base.packages lists docker, docker-buildx and docker-compose, /usr/share/omarchy/install/hardware/nvidia.sh installs nvidia-open-dkms and nvidia-utils when lspci sees an NVIDIA card, no Omarchy package list mentions nvidia-container-toolkit, pacman -Q nvidia-container-toolkit reports it is not installed, /etc/cdi does not exist, no hook binary is on PATH, and nvidia-smi works and reports the card. Two defects. First, the cause is wrong: --gpus does not require a runtimes entry in /etc/docker/daemon.json. moby's daemon/devices_nvidia_linux.go (read from the moby repo) registers its nvidia device driver purely from an exec.LookPath for nvidia-cdi-hook or nvidia-container-runtime-hook and returns nil when neither exists, and the Arch wiki says the same, that after installing the package you can use --gpus OR register the runtime. Second and worse on Omarchy 4: /etc/docker/daemon.json already exists and is owned by omarchy-settings 4.0.2-1 (confirmed with pacman -Qo), carrying "dns": ["172.17.0.1"] and "bip": "172.17.0.1/16", so pasting the record's whole-file runtimes-only JSON deletes exactly the two keys that the corpus record docker-container-dns-blocked-by-ufw and /usr/share/omarchy/install/config/firewall.sh depend on, and breaks container DNS everywhere. That is correct generic Arch advice mis-applied to Omarchy 4, because on plain Arch the file does not exist. The nvidia-ctk path is safe: the toolkit's pkg/config/engine/docker/option.go loadConfig reads and json-decodes the existing daemon.json before AddRuntime merges the key, confirmed from source. Third, the verify is broken on stock Omarchy: docker info here prints Server: permission denied while trying to connect to the docker API at unix:///var/run/docker.sock and docker info | grep -i runtime matches nothing, because /usr/share/omarchy/install/config/docker.sh deliberately leaves the user out of the docker group, so every docker command in the fix needs sudo. What held: all package paths are right, confirmed against the Arch package file list, which ships /usr/bin/nvidia-ctk, /usr/bin/nvidia-container-runtime, /usr/bin/nvidia-container-runtime-hook and /etc/nvidia-container-runtime/config.toml. The wiki supports the NVML --device workaround verbatim, and all three device nodes exist here. The Arch PKGBUILD and its nvidia-ctk-cdi.hook confirm the CDI file is generated automatically on install and refreshed on nvidia-utils upgrades, which is new information worth having in the fix. All four cited URLs return 200 and support their claims, so nothing is removed, but the BBS thread is a single post with no replies, so the Docker Desktop branch rests on one unanswered report and is now scoped as such. Severity and frequency left alone: the consequence is still a failed container, not a damaged machine, and the new destructive path belongs in danger. NOT exercised: I have no sudo, so I could not install the toolkit, restart dockerd, run nvidia-ctk runtime configure, or start any container. The merge behaviour of nvidia-ctk is read from its source rather than observed, and the NVML workaround is untested here.

The cause above was rewritten on 2026-09-11 to match this note; the fix was corrected by the audit itself.
RISK

On Omarchy 4 /etc/docker/daemon.json is owned by omarchy-settings and already sets dns and bip. Overwriting it with a runtimes-only file removes both and breaks DNS in every container, because Omarchy's ufw rules permit port 53 only to 172.17.0.1. Use sudo nvidia-ctk runtime configure --runtime=docker, which merges into the existing file, or add the key by hand and keep everything else. Being a packaged file it can also arrive as a .pacnew during omarchy update, so a hand-edited copy has to be merged rather than ignored. Whatever you edit, the file must stay valid JSON. A stray trailing comma makes docker.service fail to start with every container down, so validate with python3 -m json.tool /etc/docker/daemon.json before restarting. dockerd also refuses to start if the same option is set both in daemon.json and as a flag in the unit.

Symptom

Any GPU container fails immediately:


docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

nvidia-smi on the host works fine. Sometimes the container starts but then prints Failed to initialize NVML: Unknown Error.

Cause

Docker has no built-in NVIDIA support. dockerd registers its nvidia device driver only when one of the NVIDIA Container Toolkit hook binaries is on its PATH: moby's daemon/devices_nvidia_linux.go runs exec.LookPath for nvidia-cdi-hook and for nvidia-container-runtime-hook, and when it finds neither it returns no device driver at all, which is the could not select device driver message. On Arch those binaries come from nvidia-container-toolkit, a separate package that nothing pulls in. Omarchy 4 installs docker, docker-buildx, docker-compose and the NVIDIA driver, but never nvidia-container-toolkit, so an Omarchy machine with a working nvidia-smi still fails every GPU container. Registering an nvidia entry under runtimes in /etc/docker/daemon.json is a different feature: it is what --runtime=nvidia selects, and --gpus does not need it.

Fix

Install the toolkit and restart the daemon:


sudo pacman -S --needed nvidia-container-toolkit
sudo systemctl restart docker.service

That is the whole fix for --gpus. The package ships /usr/bin/nvidia-container-runtime-hook and /usr/bin/nvidia-cdi-hook, and its ALPM hook /usr/share/libalpm/hooks/nvidia-ctk-cdi.hook regenerates /etc/cdi/nvidia.yaml on install and on every nvidia-utils upgrade.

Omarchy does not put your user in the docker group, so run the client under sudo unless you opted in through Setup > Security > Sudoless Docker:


sudo docker run --rm --gpus all nvidia/cuda:12.1.1-runtime-ubuntu22.04 nvidia-smi

If you also want --runtime=nvidia, register the runtime with nvidia-ctk rather than by hand:


sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker.service

nvidia-ctk reads the existing /etc/docker/daemon.json, decodes it, and adds only the runtimes key, so the rest of the file survives.

**Do not paste a whole-file daemon.json over the top on Omarchy 4.** The file is shipped by omarchy-settings and already carries the daemon's DNS and bridge settings:


pacman -Qo /etc/docker/daemon.json     # omarchy-settings
cat /etc/docker/daemon.json

{
    "log-driver": "json-file",
    "log-opts": { "max-size": "10m", "max-file": "5" },
    "dns": ["172.17.0.1"],
    "bip": "172.17.0.1/16"
}

Dropping dns and bip breaks name resolution in every container, because Omarchy's ufw rules allow port 53 only to 172.17.0.1. Add the key and keep the rest:


{
    "log-driver": "json-file",
    "log-opts": { "max-size": "10m", "max-file": "5" },
    "dns": ["172.17.0.1"],
    "bip": "172.17.0.1/16",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

On plain Arch, EndeavourOS or CachyOS the file usually does not exist at all, and the Arch wiki's whole-file form is safe there.

If you get Failed to initialize NVML: Unknown Error instead, pass the device nodes explicitly:


sudo docker run --rm --gpus all \
  --device /dev/nvidiactl:/dev/nvidiactl \
  --device /dev/nvidia-uvm:/dev/nvidia-uvm \
  --device /dev/nvidia0:/dev/nvidia0 \
  nvidia/cuda:12.1.1-runtime-ubuntu22.04 nvidia-smi

If you installed Docker Desktop yourself, which Omarchy does not ship, the same error can mean you are talking to Desktop's VM, and that VM has no host GPU:


docker context ls
docker context use default

Verify

sudo docker run --rm --gpus all nvidia/cuda:12.1.1-runtime-ubuntu22.04 nvidia-smi prints your GPU. Do not use docker info | grep -i runtimes as the test. It reports the --runtime=nvidia registration rather than --gpus support, and on a stock Omarchy 4 install docker info cannot read the Server section at all, printing permission denied while trying to connect to the docker API at unix:///var/run/docker.sock, because your user is not in the docker group. Under sudo, sudo docker info | grep -i runtimes lists nvidia only if you ran nvidia-ctk runtime configure.

Sources

docker-gpu-could-not-select-device-driver