Introduction
Crusoe GPU instances receive their GPUs through PCIe passthrough, so the VM sees real hardware. But everything you'd normally use to name a machine is ephemeral. VM names, VM IDs, and Kubernetes node names all change when you rebuild a VM or recycle a node pool. The physical GPU underneath doesn't.
ℹ️ Note: On Crusoe Managed Kubernetes, a "node" is a VM in a node pool, not the physical host it runs on. This article uses "VM" for the virtual machine and "physical hardware" for the GPU board and host underneath it.
That gap matters when you're trying to tell a recurring hardware fault apart from a coincidence. If the same rank aborts every few days, or ECC errors keep appearing on a workload even after you've deleted and recreated the VM twice, the question is whether you're seeing one bad GPU or several. VM-level identifiers can't answer it.
GPU UUIDs and board serial numbers can. They're read from the GPU itself, so they stay constant across VM lifecycle events and give you a stable fingerprint to correlate against.
This FAQ covers what identifiers are available, how to read them, and how to use them to track hardware over time.
Prerequisites
- GPU Instance Running on Crusoe Cloud
- SSH or Terminal Access to the Instance
- NVIDIA Driver Installed
- CMK Cluster With
kubectlAccess (Kubernetes Questions Only)
Question 1: Is There a Kubernetes Node Label That Maps to the Physical Host?
Answer: No. Node labels on CMK describe the VM and its node pool, such as instance class, node pool ID and name, and project ID. None of them expose the physical host identity or its serial number.
You can confirm this for any node:
kubectl get node <NODE_NAME> --show-labels
Use the GPU-level identifiers below instead. They're more useful for hardware correlation anyway, because they identify the specific GPU board rather than the chassis.
Question 2: How Do I Read the GPU UUIDs and Serial Numbers?
Answer: Run nvidia-smi with a query for the fields you want:
nvidia-smi --query-gpu=index,name,uuid,serial --format=csv
Expected output:
index, name, uuid, serial 0, NVIDIA H100 80GB HBM3, GPU-af17e080-7c95-18cb-b596-6fe025fef395, 1655023013829 1, NVIDIA H100 80GB HBM3, GPU-087651a2-2340-3f5b-a1e9-aa35b29357dd, 1654023012475 2, NVIDIA H100 80GB HBM3, GPU-4546b4df-0dc9-2685-68b5-770e9a247778, 1654823008067 3, NVIDIA H100 80GB HBM3, GPU-4818ce2f-16b3-7cae-6526-f0fb7086c37f, 1651824050094
For a quicker look at just the UUIDs:
nvidia-smi -L
Expected output:
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-af17e080-7c95-18cb-b596-6fe025fef395) GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-087651a2-2340-3f5b-a1e9-aa35b29357dd) GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-4546b4df-0dc9-2685-68b5-770e9a247778) GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-4818ce2f-16b3-7cae-6526-f0fb7086c37f)
The index is the position of the GPU in this VM and isn't stable across reboots or rebuilds. The uuid and serial are the hardware identifiers.
ℹ️ Note: On AMD Instinct SKUs there is no
nvidia-smi. Userocm-smi --showuniqueid --showserialfor the equivalent per-GPU identifiers; everything else in this FAQ applies unchanged.
Question 3: Do These Identifiers Survive a VM Rebuild?
Answer: Yes. The UUID and serial are properties of the physical GPU board and are read from the hardware, so they don't change when the VM around them is destroyed and recreated. If a rebuilt VM lands on the same physical hardware, the same GPUs report the same UUIDs and serials.
If the values change after a rebuild, that tells you the VM came up on different physical hardware, which is itself useful information when you're tracking a fault.
ℹ️ Note: The GPU index is not an identifier. A GPU that was
index 3before a reboot can be a different physical board afterwards. Always correlate on UUID or serial, never on index.
Question 4: How Do I Tell Whether a Fault Keeps Hitting the Same GPU?
Answer: Capture a baseline while the node is healthy, then capture again when a fault occurs and compare.
Record the baseline at provisioning time and keep it somewhere outside the VM:
nvidia-smi --query-gpu=index,name,uuid,serial --format=csv > gpu-baseline-$(hostname)-$(date +%F).csv
When something fails, map the failure back to a GPU index first. An XID in the system log, a DCGM alert, or the aborting rank in a NCCL or RCCL job will each identify a GPU index. Then look that index up in a fresh nvidia-smi query to get its UUID and serial, and compare against your baseline and against any earlier incidents.
If the same serial shows up across multiple incidents, you're looking at one board with a persistent problem rather than a series of unrelated events.
Question 5: What Should I Include When I Open a Support Ticket About a GPU Fault?
Answer: Include all of the following:
- The UUID and serial of the affected GPU
- The VM ID
- The error itself — XID number, DCGM alert, or job failure output
- An NVIDIA bug report (see How-To Capture NVIDIA Bug Report via Command Center)
The UUID and serial let Crusoe Support match the GPU to its hardware history directly, without first working backwards from a VM that may already have been replaced.