Failed to Initialize NVML: Unknown Error on GPU Nodes

Here is NVIDIA's reproduction, container id elided, including the five --device flags, which the notice says are deliberate: they narrow the problem down to this specific issue.

$ docker run -d --rm --runtime=nvidia --gpus all \
    --device=/dev/nvidia-uvm \
    --device=/dev/nvidia-uvm-tools \
    --device=/dev/nvidia-modeset \
    --device=/dev/nvidiactl \
    --device=/dev/nvidia0 \
    nvcr.io/nvidia/cuda:12.0.0-base-ubuntu20.04 bash -c "while [ true ]; do nvidia-smi -L; sleep 5; done"

$ sudo systemctl daemon-reload

$ docker logs <container-id>
GPU 0: Tesla K80 (UUID: GPU-05ea3312-64dd-a4e7-bc72-46d2f6050147)
GPU 0: Tesla K80 (UUID: GPU-05ea3312-64dd-a4e7-bc72-46d2f6050147)
GPU 0: Tesla K80 (UUID: GPU-05ea3312-64dd-a4e7-bc72-46d2f6050147)
GPU 0: Tesla K80 (UUID: GPU-05ea3312-64dd-a4e7-bc72-46d2f6050147)
Failed to initialize NVML: Unknown Error
Failed to initialize NVML: Unknown Error

Two documented facts separate this from a crash. The container needs to be deleted once the issue occurs, and when it is restarted, manually or automatically depending on the use of a container orchestration platform, it regains access to the GPU. Neither involves the process dying.

The Kubernetes half of the reproduction makes the consequence concrete.

apiVersion: v1
kind: Pod
metadata:
  name: cuda-nvidia-smi-loop
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda
    image: "nvcr.io/nvidia/cuda:12.0.0-base-ubuntu20.04"
    command: ["/bin/sh", "-c"]
    args: ["while true; do nvidia-smi -L; sleep 5; done"]
    resources:
      limits:
        nvidia.com/gpu: 1

restartPolicy: OnFailure is load-bearing. The loop never exits non-zero, so the policy never fires and the pod sits there printing the error, while every instrument watching exits, exit codes or restart counts reports a healthy workload. Same shape as an agent that returns a confident completion message having written nothing: a process that keeps running while producing nothing is a detection problem, not a crash. The check has to read the resource the process is supposed to be using.

Do not try to read a diagnosis out of the message. NVML_ERROR_UNKNOWN is 999 in nvml.h, commented as an internal driver error, and nvmlInit_v2 is documented as returning it on any unexpected error, next to NVML_ERROR_DRIVER_NOT_LOADED when the driver is not running and NVML_ERROR_NO_PERMISSION when NVML cannot talk to the driver. It is the catch-all return, and it tells you initialisation failed and nothing else. Another error string whose real fix is upstream of the message: Transformers does not recognise an architecture that vLLM is being asked to serve.

NVIDIA's troubleshooting page states the cause in two sentences, and every mitigation follows from them. When using the NVIDIA Container Runtime Hook, meaning the Docker --gpus flag or the NVIDIA Container Runtime in legacy mode, to inject requested GPUs and driver libraries into a container, the hook makes modifications, including setting up cgroup access, to the container without the low-level runtime such as runc being aware of these changes. The result is that updates to the container can remove access to the requested GPUs.

The device access is granted out of band. The low-level runtime holds a container config that does not mention those devices, so the next time anything rewrites the container's cgroup state from that config, the access the hook added is not in the input and is not carried forward.

Beside that cause sits one variant, in a warning box on the same page: certain runc versions show similar behaviour with the systemd cgroup driver when /dev/char symlinks for the required devices are missing on the system. The page adds two things about it. The behaviour persisted even if device nodes were requested on the command line, which is why NVIDIA's own reproduction passes five --device flags and still fails. And newer runc versions do not show this behaviour, while newer NVIDIA driver versions ensure that the required symlinks are present, reducing the likelihood of the specific issue occurring for affected runc versions.

A cause, a variant and a trigger are not three causes. The page documents one cause heading, one warning-boxed variant and one named trigger, under a section structure of problem, Affected Environments, Mitigations and Workarounds. Getting that right changes what you build: three independent root causes would justify keeping three independent mitigations, and what is actually in front of you is one mechanism with one adjacent condition.

The page also names no affected or fixed version of the NVIDIA Container Toolkit, runc, containerd, Docker or the kubelet anywhere in this section. Every version reference in it is qualitative: certain runc versions, newer runc versions, newer NVIDIA driver versions. This post does not supply a number for any of them either, because none is published.

The trigger sentence is verbatim from the Affected Environments section. On certain systems, this behavior is not limited to explicit container updates such as adjusting CPU and memory limits for a container. On systems where systemd is used to manage the cgroups of the container, reloading the systemd unit files (systemctl daemon-reload) is sufficient to trigger container updates and cause a loss of GPU access.

Read that as a scoping statement rather than a headline. daemon-reload is a trigger, not the cause, and treating it as the cause misleads anyone whose trigger was a docker update or a limits change. What they share is that something rewrote the container's cgroup state from a config that never knew about the injected devices.

The precondition is the cgroup driver, and NVIDIA gives a check per engine. For Docker:

$ docker info
...
Cgroup Driver: systemd
Cgroup Version: 1

For containerd it is SystemdCgroup = true under the nvidia runtime options in /etc/containerd/config.toml, with the effective value readable through sudo crictl info:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
  BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime"
  ...
  SystemdCgroup = true

For cri-o, it is cgroup_manager = "systemd" under [crio.runtime] in /etc/crio/crio.conf or /etc/crio/crio.conf.d/00-default.

Two exclusions come straight from the notice, and either one ends the investigation. If the system is not using systemd to manage cgroups, then it is not subject to this issue. And Podman environments use crun by default and are not subject to this issue unless runc is configured as the low-level container runtime.

NVIDIA Container Toolkit v1.18.0 was published on 2025-10-21, and its first listed high-level change is the one that matters: the default mode of the NVIDIA Container Runtime has been updated to make use of a just-in-time-generated CDI specification instead of defaulting to the legacy mode. The changelog entry under v1.18.0-rc.1 reads "Default to jit-cdi mode in the nvidia runtime". CDI injection is the mitigation NVIDIA names on the troubleshooting page, and it works structurally: when CDI is used to inject devices into a container, the required device nodes are included in the modifications made to the container config, so even if the container is updated it still has access to the required devices. The devices stop being an out-of-band addition and become part of the input the update rewrites from.

On a node built after that date with nvidia-container-runtime.mode left at auto, the CDI mitigation is already the default and there is nothing to enable.

Stopping there ships a false conclusion. The same v1.18.0-rc.1 changelog carries a second entry directly beside the first: "Resolve to legacy by default in nvidia-container-runtime-hook". At tag v1.20.0, the newest non-prerelease toolkit release on or before this post's date and published on 2026-08-13, cmd/nvidia-container-runtime-hook/hook_config.go builds its mode resolver with info.WithDefaultMode(info.LegacyRuntimeMode), and the resolver in internal/info/auto.go returns its own defaultMode for the NVML platform when the mode is auto. The two paths read the same mode = "auto" from the same config.toml and reach different answers: jit-cdi for the NVIDIA Container Runtime, legacy for the hook.

docker run --gpus all invokes the hook. On a fully current v1.20.0 node with an untouched config it is still on the exposed path, which is consistent with NVIDIA's own scoping sentence naming the Docker --gpus flag and the NVIDIA Container Runtime in legacy mode as the two hook paths in the same breath.

Five paths, two of them safe by construction. The verdict is short: leave the mode at auto on a v1.18.0 or later node, and move workloads from --gpus to --runtime=nvidia. Doing one without the other buys nothing.

Do not infer the mode from the package version. Read it.

# Print the effective config, including the resolved key set
$ sudo nvidia-ctk config

# The key that decides exposure
[nvidia-container-runtime]
  mode = "auto"

# Installed toolkit version
$ dpkg -l | grep nvidia-container-toolkit

nvidia-ctk config with no --set and no --output writes the configuration to stdout: that is the read, and --in-place is what turns it into a write. There is no --list flag and no config get subcommand, and a valueless --set nvidia-container-runtime.mode is rejected with expected key=value. The file itself lives at /etc/nvidia-container-runtime/config.toml.

mode = "auto" in that file is the documented default, but it is not proof of the resolved mode, because auto resolves differently per invocation path. Make the runtime say it out loud instead. The debug log path defaults to /dev/null, so nothing is written until you point it somewhere.

$ sudo nvidia-ctk config --in-place \
    --set nvidia-container-runtime.debug=/var/log/nvidia-container-runtime.log \
    --set nvidia-container-runtime.log-level=debug

$ docker run --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all \
    nvcr.io/nvidia/cuda:12.0.0-base-ubuntu20.04 nvidia-smi -L

$ grep -E "Auto-detected mode as|Using requested mode" /var/log/nvidia-container-runtime.log
Auto-detected mode as 'jit-cdi'

Those two strings are the resolver's only statements of the mode it picked. Auto-detected mode as means the config said auto and the resolver picked; Using requested mode means someone set the mode explicitly and the resolver short-circuited on it. That short-circuit is the sharp edge: an explicit mode = "legacy" is honoured on every version, v1.20.0 included, so upgrading the toolkit under it changes nothing. Legacy gets pinned that way by documents rather than by software: a runbook written when legacy was the default, or a configuration-management template that sets the key to be explicit about it.

A separate signal tells you someone moved the mode but not the workloads. If config.toml sets the mode explicitly to jit-cdi, cdi or csv and a container is still started with docker --gpus, the hook refuses:

invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime (e.g. specify the --runtime=nvidia flag) instead

main.go wraps that assertion in log.Panicf, so the container fails to start rather than starting without GPUs. This is not the NVML error under another name; it is the loud version of the same disagreement. The quiet failure happens when the mode is auto, the loud one when it is not.

The answer belongs in the image build rather than in a runbook, the same reasoning we applied to pinning and snapshotting a self-managed inference node for an air-gapped estate. Deriving a minimum safe version for a GPU server from the vendor's record is its own exercise, covered in hardening Triton against the vendor's published record; here the minimum toolkit version is v1.18.0 and the boundary is a single date.

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.0-1
sudo apt-get install -y \
    nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

$ sudo nvidia-ctk runtime configure --runtime=docker
$ sudo systemctl restart docker

[Unit]
Description=Trigger CDI refresh on NVIDIA driver or toolkit install / upgrade events

[Path]
PathChanged=/lib/modules/%v/modules.dep
PathChanged=/lib/modules/%v/modules.dep.bin
PathChanged=/usr/bin/nvidia-ctk

[Install]
WantedBy=multi-user.target

$ systemctl is-enabled nvidia-cdi-refresh.path
$ systemctl is-active nvidia-cdi-refresh.path
$ ls -l /var/run/cdi/nvidia.yaml
$ nvidia-ctk cdi list

Some nodes cannot take a toolkit upgrade this quarter. What follows is the pre-v1.18 workaround set, a set rather than a menu: each entry covers a different slice and none covers everything.

The Docker cgroup driver change reads like a fix, and NVIDIA limits it in the sentence immediately after publishing it: this approach does not change the behavior for explicit container updates, and a container still loses access to devices in this case. It covers the daemon-reload trigger and nothing beyond it.

{
    "exec-opts": ["native.cgroupdriver=cgroupfs"]
}

The symlink utility shipped in toolkit v1.12.0 and addresses the variant rather than the cause. NVIDIA's guidance is that it should be configured to run at boot on each node where GPUs will be used in containers, and that it requires the NVIDIA driver kernel modules to have been loaded when it runs.

$ sudo nvidia-ctk system create-dev-char-symlinks --create-all

# /lib/udev/rules.d/71-nvidia-dev-char.rules
ACTION=="add", DEVPATH=="/bus/pci/drivers/nvidia", RUN+="/usr/bin/nvidia-ctk system create-dev-char-symlinks --create-all"

Two notes on that command, both of which bite a copy-paste. The --watch option was removed under v1.18.0-rc.1 and does not exist at v1.20.0, where the command registers exactly --dev-char-path, --driver-root, --create-all, --load-kernel-modules, --create-device-nodes and --dry-run. And the driver-container variant in NVIDIA's own notice names a create-dev-symlinks subcommand; the only two subcommands under nvidia-ctk system at v1.20.0 are create-dev-char-symlinks and create-device-nodes. Use the real one with --driver-root, or leave the variant out.

None of the above tells you it happened. The failure is silent by construction, so the node has to check itself, and the check has to run where the trigger is rather than on a dashboard refresh. The script below is this post's own construction, not an NVIDIA artifact, though every interface in it is standard.

#!/bin/sh
# /usr/local/bin/assert-gpu-visible
# Exit non-zero if any running container that was given GPUs can no longer see them.
set -eu
rc=0
for c in $(docker ps -q); do
  docker inspect --format '{{.Id}} {{.HostConfig.Runtime}}' "$c" | grep -q ' nvidia$' || continue
  if ! docker exec "$c" nvidia-smi -L 2>&1 | grep -qv 'Failed to initialize NVML'; then
    echo "gpu-revoked container=$c" >&2
    rc=1
  fi
done
exit $rc

The logic is checkable by hand. On a healthy container nvidia-smi -L prints one line per device and no NVML line, so grep -qv finds a non-matching line, succeeds, and the negated condition does not fire. On a revoked container the only output is the NVML line, grep -qv finds nothing, and the condition fires. It greps the prefix rather than the full string deliberately, so it also catches the Insufficient Permissions failure, a different item with a different remedy.

State its limits honestly. It detects, it does not prevent. And it only inspects containers whose HostConfig.Runtime is nvidia, so the --gpus containers on the exposed hook path are exactly the ones it cannot see. That is an argument for moving them, not for widening the filter.

Wire it to the transaction rather than to a timer, so the check runs inside the maintenance window that can cause the failure.

# /etc/systemd/system/assert-gpu-visible.service
[Unit]
Description=Assert running GPU containers still see their devices

[Service]
Type=oneshot
ExecStart=/usr/local/bin/assert-gpu-visible

# /etc/apt/apt.conf.d/99-assert-gpu-visible
DPkg::Post-Invoke { "systemctl start --no-block assert-gpu-visible.service || true"; };

Both of those are constructed too. The systemd unit and APT's DPkg::Post-Invoke hook are standard interfaces, and if you would rather not add a package hook, the unit alone, invoked from your configuration-management run, is enough.

Deployment location does not change the mechanism at all: the cgroup revocation is identical on a rented instance and on a box on your own floor. What it changes is node age and evidence. Rented short-lived nodes are rebuilt from a fresh image, so their toolkit is whatever was current at that rebuild, and a three-year-old install is not among the outcomes. On hardware a regulated team keeps for years, the toolkit version is whatever the build engineer installed at rack time, anything imaged before 2025-10-21 is on the legacy default unless someone changed it deliberately, and the thing issuing daemon-reload is that team's own patch automation. Then the governance half: on rented infrastructure the provider's control plane records a node replacement, so a GPU going away has a timestamp somewhere. On your own floor nothing records it unless you write it, which is part of the ledger behind what on-premise actually costs against renting the same capacity, and it sits beside the other decisions you own once the hardware is yours, including whether confidential computing on the GPU is worth its overhead.

So treat a routine OS patch window on a GPU node as a GPU-availability event for every container already running on that box, and put it in the change record as one, with the assertion's output as the evidence that it did or did not happen. That is what makes the failure traceable afterwards, the same discipline as tracing a production model failure back to the change that caused it. The rest of the tooling ground around this sits in the AI development tools pillar.

This week, on one GPU node: run sudo nvidia-ctk config, read the mode line, and write the answer next to that node in the change record. If it says anything other than auto, you have found a node where upgrading the toolkit will not change a thing.