How-To Configure NVIDIA Device Plugin to Ignore Specific XID Errors

Last Updated: Feb 5, 2026

Introduction

By default, the NVIDIA GPU Operator Device Plugin ignores a specific set of XID errors during health checks. If an XID error occurs that is not on this ignore list, the plugin marks the affected GPU as unhealthy, making it unavailable for new workloads.

This default list of ignored XIDs can be confirmed by running kubectl -n nvidia-gpu-operator logs ds/nvidia-device-plugin-daemonset command on your Crusoe Managed Kubernetes (CMK) cluster.

# kubectl -n nvidia-gpu-operator logs ds/nvidia-device-plugin-daemonset
......
nvidia-device-plugin I0205 01:10:51.403987 1 health.go:64] Ignoring the following XIDs for health checks: map[13:true 31:true 43:true 45:true 68:true 109:true]

In this case, the plugin is configured to bypass XIDs 13, 31, 43, 45, 68, and 109.

When an XID error that is not a part of the ignored list mentioned above is logged on a node, the GPUs on that node are marked as unhealthy and removed from resources available to be allocated on that node.

I0205 02:41:36.328742       1 health.go:169] XidCriticalError: Xid=94 on Device=GPU-4a5d290e-b800-d300-c3eb-63623f8e8239; marking device as unhealthy. 

If you determine that specific XID errors are not impacting your particular workload and want to prevent GPUs from being taken offline unnecessarily, you can manually expand the ignore list by following this article.

Prerequisites

• Crusoe Managed Kubernetes (CMK) cluster
• NVIDIA GPU Operator

Step-by-Step Instructions

Step 1: Edit the nvidia-device-plugin daemonset

kubectl edit ds -n nvidia-gpu-operator nvidia-device-plugin-daemonset

Step 2: Add the DP_DISABLE_HEALTHCHECKS environment variable to the nvidia-device-plugin container and specify the XIDs to ignore

apiVersion: apps/v1
kind: DaemonSet
metadata:
....
spec:
  template:
    spec:
      containers:
      - args:
        - /bin/entrypoint.sh
        command:
        - /bin/sh
        - -c
        env:
        - name: DP_DISABLE_HEALTHCHECKS #disable health checks for specific XIDs
          value: 13,31,43,45,68,94,109,137,145

Example

Additional Resources

https://github.com/NVIDIA/k8s-device-plugin/blob/fdbb5e093e3078547f2a89d0070cfcdda438b610/internal/rm/health.go#L233