Introduction
On B200 and B300 nodes there is a race condition between the GPU Operator and the Network Operator during pod initialization. If the GPU driver pod starts before MOFED has finished installing, the GPU and Network Operator pods stay Pending and CUDA calls fail with system not yet initialized.
NVIDIA has not yet published a fix. Until they do, the workaround is a Kyverno policy that injects a wait-for-mofed init container into the GPU driver pods, which forces the two operators to initialize in the right order.
ℹ️ Note: This applies to B200 (
nvidia-b200-180gb-sxm-ib) and B300 (nvidia-b300-288gb-sxm-ib,nvidia-b300-288gb-sxm-ib-noht) nodes only, and can be applied after cluster creation.
ℹ️ Note: Two other MOFED race conditions are documented separately and are not the same failure. This article covers the GPU Operator versus Network Operator ordering race on B200/B300, where driver pods stay
Pending. See Related Articles for the storage-module race and the driver-restart mitigation.
Prerequisites
-
kubectlPointed at the Target CMK Cluster (Verify Withkubectl config current-context) -
helmAvailable - Kyverno and
busyboxImages Available in CCR (Private Clusters Without Public Egress)
Instructions
Step 1: Install Kyverno (Skip if Already Present)
Check whether Kyverno is already running, and skip the install if it is:
kubectl get pods -n kyverno
On a private cluster with no egress to the public internet, install Kyverno with its images pulled from your own Crusoe Container Registry path. Substitute your region, organization and project — see How-To: Use the Crusoe Container Registry (CCR) for how to find yours:
export CCR=registry.<REGION>.ccr.crusoecloudcompute.com/<ORGANIZATION>/<PROJECT>/cmk helm install kyverno kyverno/kyverno \ --namespace kyverno --create-namespace \ --version 3.9.1 \ --set global.image.registry=$CCR \ --set test.image.registry=$CCR \ --set test.image.tag=v1.19.1 \ --set webhooksCleanup.image.registry=$CCR
Wait until the admission controller is Ready before applying the policy. Applying it earlier means the mutation will not fire:
kubectl rollout status -n kyverno deploy/kyverno-admission-controller --timeout=180s
Step 2: Apply the Mutating ClusterPolicy
Save the following as b200-wait-for-mofed-clusterpolicy.yaml, substituting your own CCR path in the image field:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: wait-for-mofed-ready
spec:
rules:
- name: add-mofed-wait-init-container
match:
any:
- resources:
kinds:
- Pod
namespaces:
- nvidia-gpu-operator
preconditions:
all:
- key: "{{ request.object.spec.nodeSelector.\"crusoe.ai/accelerator\" || '' }}"
operator: In
value:
- "nvidia-b200-180gb-sxm-ib"
- "nvidia-b300-288gb-sxm-ib"
- "nvidia-b300-288gb-sxm-ib-noht"
mutate:
patchStrategicMerge:
spec:
initContainers:
- name: wait-for-mofed
image: registry.<REGION>.ccr.crusoecloudcompute.com/<ORGANIZATION>/<PROJECT>/cmk/busybox:1.37.0-glibc
command:
- sh
- -c
- |
echo "Waiting for MOFED driver installation to complete..."
while [ ! -f /run/mellanox/drivers/.driver-ready ]; do
echo "MOFED not ready, waiting..."
sleep 5
done
echo "MOFED ready, proceeding with GPU driver installation"
volumeMounts:
- name: run-mellanox-drivers
mountPath: /run/mellanox/driversApply it and confirm the policy registered:
kubectl apply -f ./b200-wait-for-mofed-clusterpolicy.yaml kubectl get clusterpolicy wait-for-mofed-ready
The policy matches Pods in the nvidia-gpu-operator namespace whose nodeSelector["crusoe.ai/accelerator"] is a B200 or B300 accelerator, and injects an init container that loops until /run/mellanox/drivers/.driver-ready exists.
ℹ️ Note: The injected init container mounts a volume named
run-mellanox-drivers, which it expects the GPU driver pod to already declare. The NVIDIA driver DaemonSet does declare it today. If a future GPU Operator release renames that volume, pod admission will fail rather than degrade quietly — so check this first if driver pods stop being created after an operator upgrade.
Step 3: Delete the Already-Broken GPU Driver Pods
⚠️ Warning: Kyverno mutation fires only on pod creation. Driver pods that already lost the race are running without the init container, and the policy will not retroactively fix them. You have to delete them so the DaemonSet recreates them through the now-mutating admission path.
Find the driver DaemonSet name — the Ubuntu variant on these nodes:
kubectl get ds -n nvidia-gpu-operator | grep -i driver # e.g. nvidia-gpu-driver-ubuntu22.04-75b8b5cf5d
Confirm its label selector before deleting anything, since the key is not always app:
DS=nvidia-gpu-driver-ubuntu22.04-75b8b5cf5d # set to the real DS name
kubectl get ds -n nvidia-gpu-operator $DS -o jsonpath='{.spec.selector.matchLabels}'Then delete its pods:
kubectl delete pod -n nvidia-gpu-operator -l app=$DS
Or let the operator manage the recreate, which is equivalent:
kubectl rollout restart ds -n nvidia-gpu-operator $DS
New driver pods should now show the wait-for-mofed init container:
kubectl get pod -n nvidia-gpu-operator -l app=$DS -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.initContainers[*].name}{"\n"}{end}'⚠️ Warning: Order matters. Install Kyverno and apply the policy first, then delete the driver pods. If you delete first, the recreated pods are admitted before the policy exists and come back unmutated — leaving you exactly where you started.
Step 4: Verify
The init container blocks until MOFED is ready, so the first boot on a node is slower than usual. That is expected.
# No more crashing or stuck validators kubectl get pods -n nvidia-gpu-operator | grep -viE 'running|completed' # Fabric reaches Completed on a worker nvidia-smi -q -i 0 | grep -A2 Fabric
Pass criteria: Fabric State : Completed — not In Progress — on every B200/B300 worker, and nvidia-cuda-validator-* and nvidia-operator-validator-* no longer crashing.
Resolution
The Kyverno policy forces the GPU driver pod to wait for MOFED's /run/mellanox/drivers/.driver-ready marker before starting, so the NVLink devices are present by the time Fabric Manager is configured. Deleting the existing driver pods is what repairs already-affected nodes; nodes added later are protected automatically, because their driver pods are admitted through the policy from the start.
Once NVIDIA ships a fix and it is deployed to your clusters, remove the policy:
kubectl delete clusterpolicy wait-for-mofed-ready
Example
A new B200 node pool comes up and the GPU Operator never reaches a healthy state. kubectl get pods -n nvidia-gpu-operator shows the driver pod Running but nvidia-operator-validator and nvidia-cuda-validator crash-looping, and a test workload fails immediately with system not yet initialized. On the node itself, nvidia-smi -q -i 0 | grep -A2 Fabric reports State : In Progress rather than Completed.
Nothing is wrong with the hardware, and rebooting the node sometimes fixes it and sometimes does not — which is the signature of a race rather than a fault. The driver pod won the race against MOFED on this node, so the NVLink devices were not present when Fabric Manager was configured.
Installing Kyverno, applying the policy, and then deleting the driver pods brings the node right: the recreated driver pod shows a wait-for-mofed init container, sits in Init for a minute or two while MOFED finishes, and then starts. Fabric reports Completed, the validators stop crashing, and the workload runs. Nodes added to the pool afterwards come up correctly without intervention.