Last Updated: Aug 25, 2026
Introduction
The NVIDIA device plugin health-checks every GPU it manages against the driver's XID event stream. Any XID that is not on the plugin's ignore list marks that GPU unhealthy and permanently withdraws it from the node's allocatable capacity, and restarting the device plugin pod on that node is the only way to get it back.
As of this writing the list compiled into the plugin covers XIDs 13, 31, 43, 45, 68, and 109. Application-level XIDs such as 94, 137, and 145 are missing from it, so on an 8-GPU SXM node a single rank exiting abruptly mid-collective can fault its seven NVLink peers and leave the node reporting capacity: 8 with allocatable: 1 while every GPU is healthy.
If you have confirmed these XIDs are not affecting your workloads, extend the ignore list so the plugin stops withdrawing healthy capacity.
ℹ️ Note: Apply this through Helm rather than editing the device plugin DaemonSet. A
kubectl editon the DaemonSet works immediately but is reverted by the nexthelm upgradeof the GPU Operator, at which point GPUs start being withdrawn again.
Prerequisites
- Crusoe Managed Kubernetes (CMK) Cluster With the NVIDIA GPU Operator
- Helm 3.13 or Later and kubectl Configured Against the Cluster
- Cluster-Admin Permissions
These steps assume the default Crusoe installation: release gpu-operator in namespace nvidia-gpu-operator. Confirm with helm list -A if yours differs.
Instructions
Step 1: Check the Current Configuration
Read the ignore list from the ClusterPolicy and confirm the operator has finished reconciling:
kubectl get clusterpolicy cluster-policy \
-o jsonpath='{.spec.devicePlugin.env}' | jq
kubectl get clusterpolicy cluster-policy -o jsonpath='{.status.state}'A ready state with no DP_DISABLE_HEALTHCHECKS entry means the plugin is running the compiled-in defaults.
Then confirm what the running pods actually carry, which is not always the same thing:
kubectl -n nvidia-gpu-operator get pods -l app=nvidia-device-plugin-daemonset \ -o custom-columns=POD:.metadata.name,ENV:.spec.containers[0].env
Step 2: Apply the Fix
Export the current values, including chart defaults, and note the chart version:
helm -n nvidia-gpu-operator get values -a gpu-operator > gpu-operator-values.yaml helm -n nvidia-gpu-operator get metadata gpu-operator | grep -i version
Open gpu-operator-values.yaml and add one entry under devicePlugin.env, leaving everything else untouched:
devicePlugin:
env:
- name: DP_DISABLE_HEALTHCHECKS
value: "13,31,43,45,68,94,109,137,145"If the NVIDIA chart repository is not already configured on your machine, add it, then apply the file:
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update helm -n nvidia-gpu-operator upgrade gpu-operator nvidia/gpu-operator \ --version <CHART_VERSION> \ -f gpu-operator-values.yaml
⚠️ Warning: Edit the exported file rather than writing a values file from scratch. Helm merges maps but replaces lists, and
devicePlugin.envis a list, so a file containing only the new variable becomes the plugin's entire environment and silently drops the rest.--reuse-valuesdoes not prevent this, because it merges at the map level and not inside arrays.
💡 Tip: Pin
--versionto the version reported above. An unpinned chart reference resolves to whatever the repository currently holds, which can upgrade the whole GPU Operator as a side effect.
Step 3: Restart the Device Plugin Pods
Check the update strategy:
kubectl -n nvidia-gpu-operator get ds nvidia-device-plugin-daemonset \
-o jsonpath='{.spec.updateStrategy.type}'RollingUpdate restarts the pods automatically. OnDelete does not, and you must delete them one node at a time:
for p in $(kubectl -n nvidia-gpu-operator get pod \
-l app=nvidia-device-plugin-daemonset -o name); do
kubectl -n nvidia-gpu-operator delete $p
kubectl -n nvidia-gpu-operator rollout status \
ds/nvidia-device-plugin-daemonset --timeout=120s
doneDeleting the device plugin pod does not evict or interrupt running workloads. A new GPU pod binding to that node during the brief restart may fail admission and will retry automatically.
⚠️ Warning: Under
OnDelete, the ClusterPolicy and DaemonSet can showDP_DISABLE_HEALTHCHECKSwhile the running pods do not have it, so the configuration looks correct while the fix is inactive.kubectl rollout restarthas no effect under this strategy. Delete the pods.
Verify
Re-run the pod check from Step 1. Every device plugin pod should now list DP_DISABLE_HEALTHCHECKS, and capacity should match allocatable on every node:
kubectl get nodes -o custom-columns=NODE:.metadata.name,CAP:.status.capacity.'nvidia\.com/gpu',ALLOC:.status.allocatable.'nvidia\.com/gpu'
Any GPUs withdrawn before the change are restored by the pod restart in Step 3. To roll back, run helm -n nvidia-gpu-operator rollback gpu-operator <REVISION> and delete the plugin pods again.
ℹ️ Note: This changes how the device plugin reacts to an XID, not GPU or driver behavior. The XID errors still occur, are still written to the kernel ring buffer, and any XID-based alerting you have continues to fire. Only the capacity withdrawal stops.
Which XIDs Can Be Ignored
Add only XIDs that NVIDIA classifies as application-level. Ignoring a hardware XID leaves a genuinely failing GPU in the scheduling pool.
| Classification | XIDs | Meaning |
|---|---|---|
| Application-level, safe to ignore | 13, 31, 43, 45, 68, 94, 109, 137, 145 | Caused by application behavior. The GPU is healthy and the recommended action is to restart the application. |
| Hardware, never ignore | 48, 63, 64, 79, 95, 119, 120, 154, 171, 172 | Genuine hardware faults. The withdrawal is correct and the GPU needs investigation. |
XID 94 requires reading the message, not just the code. The application-level form reports Contained: SM, RST: No, and no physical address:
NVRM: Xid (PCI:0003:00:04): 94, Contained: SM (0x1). RST: No, D-RST: No NVRM: Xid (PCI:0003:00:04): 94, pid=52425, name=python, channel 0x0000000a
⚠️ Warning: An XID 94 reporting
RST: Yes,Contained: GSP, or a physical address is a hardware event. Check counters withnvidia-smi -q -d ECC,ROW_REMAPPER. If they have moved or a remap is pending, open a support ticket instead of ignoring the XID.
Example
An inference service is scaled by a queue-depth autoscaler, and the serving container does not implement a SIGTERM handler. On every scale-in, Kubernetes waits out the grace period and then kills all eight ranks at once, producing a sub-second XID 94 burst that withdraws seven of the node's eight GPUs.
Because one scale-in can delete replicas across several nodes in the same second, the bursts appear cluster-wide and read like a hardware incident. Capacity erodes over days until a job cannot be placed on a cluster that appears to have idle GPUs. Applying this configuration stops the erosion, and adding a SIGTERM handler removes the trigger.