How-To Map Guest NVMe Devices to PCIe BDF Topology

Introduction

On Crusoe's GPU VM types, local NVMe drives are attached via PCIe pass-through, which means the guest OS sees them as ordinary block devices (nvme0, nvme1, and so on). Every PCIe device is addressed by a BDF (Bus:Device:Function) identifier, a hierarchical address like 0002:00:05.0 that describes exactly where that device sits on the PCIe fabric. Your guest's nvme0/nvme1/etc. naming is assigned by enumeration order, not by physical layout, so two devices with adjacent numbers are not guaranteed to have adjacent or related BDFs.

This matters whenever you're designing a storage layout with a resilience requirement, most commonly RAID, since two drives that look independent from inside the guest may in fact share a common upstream PCIe bridge, switch, or root complex. If that shared upstream component ever has a problem, both drives can be affected simultaneously, which defeats the redundancy a mirrored RAID pair is meant to provide.

This article walks through resolving your guest-visible NVMe device names to their BDFs and tracing the upstream PCIe topology from there, so you can make informed decisions about which drives to pair together.

Prerequisites

• SSH Access to the VM.
• Sudo Privileges on the Guest OS.
• pciutils and nvme-cli Installed (Provides lspci and nvme).

Instructions

Step 1: List Your Guest NVMe Devices

lsblk -d -o NAME,SIZE,SERIAL,MODEL

This gives you the guest device names (nvme0n1, nvme1n1, and so on) along with each drive's serial number and model, useful for your own inventory later.

Step 2: Resolve Each Device's BDF

for i in 0 1 2 3 4 5 6 7; do
  echo -n "nvme$i -> "
  readlink -f /sys/class/nvme/nvme$i/device
done

Adjust the range to match however many NVMe devices your VM type has. The output resolves each guest device name to its BDF, embedded in the path, for example:

nvme0 -> /sys/devices/pci0002:00/0002:00:05.0
nvme1 -> /sys/devices/pci0002:00/0002:00:06.0

Here, 0002:00:05.0 and 0002:00:06.0 are the BDFs for nvme0 and nvme1 respectively. The 0002:00 prefix is the PCI domain and bus, and 05.0/06.0 are the device and function.

Step 3: Trace the Full Upstream PCIe Path From Each BDF

A shared domain prefix in the BDF (like 0002:00 above) is a starting signal, but it doesn't by itself prove two devices share the same upstream bridge or switch. Trace the full path with:

lspci -PP -s <BDF_FROM_STEP_2>

Repeat for each device's BDF. The -PP flag shows the complete chain from the root complex down to the device, letting you see exactly where two devices' upstream paths converge, or diverge, rather than inferring it from the domain number alone.

Step 4: Identify Which BDFs Share an Upstream Path

Compare the traced paths across your devices. Devices whose BDFs converge at the same upstream bridge or switch share more of the physical fabric than devices that only happen to sit in the same numeric domain. This is the grouping to use when reasoning about shared failure domains, not the raw domain number by itself.

Step 5 (Optional): Record Physical Serials for Your Own Inventory

sudo nvme id-ctrl /dev/nvme0 | grep -E '^sn|^mn|^fr'

Repeat for each device. This captures the drive's serial number, model, and firmware revision, useful to have on hand if you ever need to reference a specific physical drive in a support ticket.

Example

Suppose you're building a RAID10 array across 8 local NVMe drives and want each mirror pair to be resilient to an upstream PCIe fault, not just individual drive failure. After resolving BDFs and tracing topology with the steps above, you find that nvme0 through nvme3 (BDFs 0002:00:05.0 through 0002:00:08.0) converge on one upstream path, and nvme4 through nvme7 converge on a separate one. Instead of pairing nvme0 with nvme1 (both on the same upstream path) as a mirror, you could pair nvme0 with nvme4 (on separate upstream paths), so a single upstream fault can only ever affect one side of any given mirror, not both simultaneously.