CUDA Validator Fails With Error "Failed to allocate device vector"

Last Updated: Nov 4, 2025

Overview

While adding B200 nodes to an existing Crusoe Managed Kubernetes (CMK) cluster, you might see the cuda-validator pod failing to initialize. When checking the pod logs, you may observe the following error:

Failed to allocate device vector A (error code system not yet initialized)!
[Vector addition of 50000 elements]
stream closed EOF for nvidia-gpu-operator/nvidia-cuda-validator-zl2ds (cuda-validation)

This indicates that the NVIDIA CUDA validation container is unable to initialize the GPU due to driver incompatibility with the B200 GPU type.

Prerequisites

• Crusoe Managed Kubernetes (CMK)
• B200 VMs
• NVIDIA GPU Operator v25.3.0

Step-by-Step Instructions

The support for NVIDIA B200 GPUs was introduced starting with GPU driver version 570.133.20. Earlier driver versions packaged with GPU Operator v25.3.0 used drivers older than v570.133.20 and do not support B200 GPUs, resulting in CUDA initialization failures.

1. Verify GPU Driver Version

   • Run the following command inside the NVIDIA driver daemonset pod on a B200 node:

     $ kubectl exec -it <nvidia-driver-daemonset-xxxxx> -- nvidia-smi

   • Look for the Driver Version field in the output. Example:

     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 570.133.18    Driver Version: 570.133.18    CUDA Version: 12.6   |
     +-----------------------------------------------------------------------------+

     If the version is lower than 570.133.20, you must upgrade the GPU Operator.

2. Upgrade the NVIDIA GPU Operator

   • Upgrade to v25.3.1 or later (which includes the updated driver supporting B200 GPUs). If you used a custom values file during installation, include it in the upgrade.

     $ helm repo update
     $ helm upgrade gpu-operator nvidia/gpu-operator \
       -n nvidia-gpu-operator \
       --version 25.3.1 \
       -f <custom_values.yaml>

   • After the upgrade, confirm that the GPU driver pods have restarted and are using the new driver version:

     $ kubectl get pods -n nvidia-gpu-operator
     $ kubectl logs <nvidia-driver-daemonset-pod> -n nvidia-gpu-operator

     You should now see the correct driver version (>=570.133.20) when running nvidia-smi.

Additional Resources

• Installing NVIDIA GPU Operator

• NVIDIA GPU Operator Release Notes (v25.3.0)