Last Updated: Nov 06, 2025
Overview
The nvidia-cuda-validator-<> pods are not successful, i.e. not in Completed state. The logs for cuda-validation container in nvidia-cuda-validator-<> pod shows the following error logs
Failed to allocate device vector A (error code system not yet initialized)!
[Vector addition of 50000 elements]Prerequisites
- Crusoe Managed Kubernetes (CMK)
- B200 GPUs
Cause
The NVIDIA Fabric Manager process fails to start or encounters errors listing NVSwitches on the fabric. Currently, this is being investigated by NVIDIA, but our preliminary analysis suggests that this is a result of a race condition between NVIDIA Network operator pods setting up the network interfaces on the Kubernetes node and GPU operator pods attempting to start the fabric manager.
Step-by-Step Instructions
Restart the GPU driver daemonset pods
- On the affected nodes, delete the nvidia-driver-daemonset pods. Kubernetes will recreate the pods, and this has shown to address the issue.
NOTE: The following commands will restart nvidia-driver-daemonset pods across all the nodes. If only a subset of your nodes are impacted, please delete the pods on those nodes individually.
# kubectl -n nvidia-gpu-operator get daemonset | grep -i driver
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
nvidia-gpu-driver-ubuntu22.04-769dbf9846 4 4 4 4 4 crusoe.ai/accelerator=nvidia-l40s-48gb,feature.node.kubernetes.io/system-os_release.ID=ubuntu,feature.node.kubernetes.io/system-os_release.VERSION_ID=22.04,nvidia.com/gpu.deploy.driver=true,nvidia.com/gpu.present=true 35d
# kubectl delete pods -n nvidia-gpu-operator -l app=nvidia-gpu-driver-ubuntu22.04-769dbf9846
Additional Resources
We have raised a bug with NVIDIA to further investigate this issue.