B200 VM Unreachable After Reboot Due to Unsupported Ubuntu 22.04 Driver Stack

Introduction

B200 VMs running the Ubuntu 22.04 curated image may become completely unreachable after a reboot — no SSH response, 100% packet loss on ping — while the Crusoe control plane continues to report the VM as RUNNING.

The Ubuntu 22.04 image ships with the NVIDIA 570 driver series and CUDA 12.8. The 570 series is no longer LTS-designated by NVIDIA and is not qualified against the current B200 host environment. On reboot, the guest OS's RDMA device stack fails to initialize correctly, causing rdma-ndd to crash-loop. Because the IB device stack never comes up cleanly, the VM's network interface never completes DHCP — leaving the VM alive at the hypervisor level but dark on the network.

Migrating to the Ubuntu 24.04 curated image, which runs the NVIDIA 580 LTS driver series with CUDA 13.x, resolves the issue.

Prerequisites

• Crusoe Infrastructure Cloud Account with Active B200 VM(s)
• Access to Crusoe Cloud Console or API to Stop and Recreate VMs

Instructions

Step 1: Confirm the Symptoms

Verify that your VM matches this failure pattern before proceeding.

Ping from a bastion or another VM in the same network returns 100% packet loss:

$ ping <YOUR_VM_HOSTNAME>
18 packets transmitted, 0 received, 100% packet loss

If you have access to the serial console, you will see rdma-ndd crash-looping repeatedly with no recovery:

[FAILED] Failed to start RDMA Node Description Daemon.
See 'systemctl status rdma-ndd.service' for details.
[  OK  ] Stopped RDMA Node Description Daemon.
         Starting RDMA Node Description Daemon...
[FAILED] Failed to start RDMA Node Description Daemon.
See 'systemctl status rdma-ndd.service' for details.
[  OK  ] Stopped RDMA Node Description Daemon.
         Starting RDMA Node Description Daemon...
[FAILED] Failed to start RDMA Node Description Daemon.

Additionally confirm:

• The VM shows as RUNNING in the Crusoe console or via the API.
• The issue appears or reappears after a reboot — the VM may have been reachable immediately after creation.
• The VM is a B200 instance in a region where the host OS has been updated.

ℹ️ Note: Migrating the VM to a different node will not resolve this issue. The failure is caused by the guest image's driver stack, not the underlying host hardware.

Step 2: Delete the Affected VM

Delete the VM using the Crusoe console or API before recreating it.

⚠️ Warning: Stopping and recreating a VM will not preserve any data stored in the VM's local memory or ephemeral storage. Ensure any critical data has been persisted to a persistent disk or external storage before proceeding.

Step 3: Recreate the VM Using the Ubuntu 24.04 Image

When recreating the VM, select the ubuntu24.04-nvidia-sxm-docker-b200 curated image. This image runs the NVIDIA 580 LTS driver series with CUDA 13.x, which is qualified against the current B200 host environment.

ℹ️ Note: The Ubuntu 22.04 image uses the NVIDIA 570 driver series, which is no longer LTS-designated by NVIDIA and is not qualified against the current B200 host environment. NVIDIA recommends all users migrate to the 580 LTS release.

Step 4: Verify Connectivity After Reboot

Once the VM is running, confirm SSH connectivity. Then perform a reboot to confirm the issue does not recur:

ssh <YOUR_VM_USER>@<YOUR_VM_IP>
sudo reboot

After the VM comes back up, verify SSH access again. If the VM becomes unreachable after reboot, open a support ticket.

Resolution

The Ubuntu 22.04 image's NVIDIA 570 driver stack is not qualified against the current B200 host environment, causing rdma-ndd to crash-loop on boot and the VM's network interface to never complete DHCP. Migrating to the Ubuntu 24.04 image, which ships with the qualified NVIDIA 580 LTS driver stack, resolves the incompatibility.