Introduction
Crusoe GPU instances expose their local NVMe drives to the guest through VFIO passthrough. A common pattern is to combine all eight into a single mdadm RAID array for container image cache, kubelet state, or training scratch space.
mdadm writes a superblock to each member drive. It sits outside the data region and records the array UUID, member roles, an event counter, and the homehost — a hostname string that controls which machines may auto-assemble the array.
When an instance is deleted, its drives are sanitized and returned to the pool. Data regions are fully erased, but there is a brief handoff window in which the host can still see the drives before sanitization completes. If the superblocks declare that the array may be assembled anywhere, the host's own md driver can auto-assemble it and write superblock updates back — racing the erase step and sometimes winning.
The replacement instance then boots with a RAID array nobody created on it, usually degraded. The behavior is timing-dependent, so the same delete-and-replace cycle can produce a clean instance one time and a stale array the next.
ℹ️ Note: Only RAID metadata survives this window. Data regions are fully zeroed during sanitization, and no file contents carry over between instances.
Prerequisites
- Root or Sudo Access on the Affected Instance
-
mdadmandnvme-cliInstalled - Confirmation That the Array Holds No Needed Data
Instructions
Step 1: Confirm the Array Predates the Instance
First find what the kernel named the array. It is commonly md127 for an array assembled from foreign metadata, but it can be any mdN:
cat /proc/mdstat
Then inspect it, substituting the device you found:
sudo mdadm --detail /dev/md127
Output from an affected instance:
Creation Time : Thu Jul 30 19:32:05 2026
State : clean, degraded
Raid Devices : 8
Active Devices : 7
Name : any:ephemeral
Events : 6302
Number Major Minor RaidDevice State
- 0 0 0 removed
1 259 4 1 active sync set-B /dev/nvme1n1
2 259 6 2 active sync set-A /dev/nvme2n1Two fields identify a ghost array:
-
Creation Timeearlier than the instance's own creation time. An array built here cannot predate the instance. -
Nameprefixedany:. That prefix is the homehost stored in the superblock, andanyis what permits assembly on any host.
The array is usually degraded rather than complete, because sanitization succeeds on some drives and not others. Drives whose superblocks were erased appear as removed.
Step 2: Identify Which Drives Still Carry Superblocks
for dev in /dev/nvme[0-9]n1; do
echo "=== ${dev} ==="
sudo mdadm --examine "${dev}" 2>&1 | grep -E "Creation Time|Update Time|Events"
doneExample output:
=== /dev/nvme0n1 ===
mdadm: No md superblock detected on /dev/nvme0n1.
=== /dev/nvme1n1 ===
Creation Time : Thu Jul 30 19:32:05 2026
Update Time : Tue Sep 8 20:03:05 2026
Events : 6302
=== /dev/nvme2n1 ===
Creation Time : Thu Jul 30 19:32:05 2026
Update Time : Tue Sep 8 20:03:05 2026
Events : 6302
[nvme3n1 through nvme7n1 omitted — identical]Wiped drives return No md superblock detected. Drives carrying stale metadata return an identical Creation Time and Events counter across all of them, inherited from the previous instance's array.
Confirm the drive reported as removed is healthy, to rule out an actual fault. These query the drive controller, so they take the controller device rather than the namespace:
sudo nvme smart-log /dev/nvme0 sudo nvme error-log /dev/nvme0
On a ghost array these are clean — media_errors: 0 and num_err_log_entries: 0. The drive was never faulty; it simply has no superblock for md to match against.
⚠️ Warning: If these readings are not clean, or you plan to open a support ticket for any reason, capture a diagnostic bundle before continuing. Step 3 destroys the evidence permanently — see the diagnostics articles under Related Articles.
ℹ️ Note:
mdadm --manage /dev/md127 --re-add /dev/nvme0n1fails with--re-add ... is not possible. There is no superblock to re-add from. Rebuilding is the correct path.
Step 3: Stop the Ghost Array and Clear All Metadata
sudo mdadm --stop /dev/md127
for dev in /dev/nvme[0-9]n1; do sudo wipefs -a -f "${dev}"; done⚠️ Warning:
wipefs -a -fdestroys all filesystem and RAID signatures on the named devices. Confirm you are targeting the correct drives and that no data you need resides on this array.
Step 4: Recreate the Array Scoped to This Host
sudo mdadm --create /dev/md/ephemeral \
--level=10 --chunk=256 --raid-devices=8 \
--name=ephemeral --homehost=$(hostname) --run \
/dev/nvme{0,1,2,3,4,5,6,7}n1$(hostname) is evaluated on the instance, so the command works unmodified across a node pool. Omitting --homehost gives the same result, since mdadm defaults to the local hostname.
Confirm the Name field now carries the hostname rather than any:
sudo mdadm --detail /dev/md/ephemeral | grep Name
Persist the definition so the array assembles correctly on later boots:
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf sudo update-initramfs -u
💡 Tip: Update your build automation rather than patching nodes individually. Any node still provisioned with
--homehost=anyremains exposed on its next replacement cycle.
Resolution
--homehost=any disables mdadm's hostname check, declaring that any machine able to see the superblock may assemble the array. That is what allows the host to auto-assemble leftover metadata during the handoff window and write it back after sanitization has begun. Setting --homehost to the instance's own hostname restores the check.
This closes the path reliably from the guest side. Crusoe is separately hardening the host so the behavior does not depend on guest RAID configuration; that work is in progress. Your data was never at risk — only the metadata header persisted, and data regions are zeroed during sanitization.
Example
A Kubernetes node pool builds a RAID10 array on each node and bind-mounts /var/lib/containerd and /var/lib/kubelet onto it. A node is deleted and the pool provisions a replacement, which boots reporting:
md/raid10:md127: active with 7 out of 8 devices
No build script ran on this node. mdadm --detail shows a creation time weeks earlier than the instance, a Name of any:ephemeral, and one drive removed — though that drive enumerates normally and reports a clean SMART log. Steps 3 and 4 clear the stale array and rebuild it correctly, and dropping --homehost=any from the build script prevents recurrence.