Introduction
Combining the local NVMe drives on a Crusoe GPU instance into a single mdadm software RAID array is a common pattern. It presents one large ephemeral namespace instead of eight separate drives, which suits container image caches, kubelet state, and training scratch space.
That array is built and owned entirely inside your guest. Crusoe has no visibility into it — the drives are attached through VFIO passthrough, so the hypervisor sees no block traffic and has no awareness that an array exists at all. Any question about array state has to be answered from inside the instance.
md maintains its own bookkeeping, and it is worth understanding that this bookkeeping sits between two other layers that keep records of their own. Below it, each drive's firmware maintains a SMART and error log describing what the device experienced. Above it, the filesystem maintains its own state. md's records agree with neither automatically, and reading only one layer can be actively misleading.
The most common way this misleads: md stores a superblock on each member drive recording array identity, member roles, an event counter, and a per-device bad-block list. When a write to a member fails, md may record the affected sector range in that list rather than failing the drive out of the array. The array then continues to report itself as complete and healthy while specific regions are marked unreadable. A clean mdadm --detail does not, on its own, mean the array is clean.
This article covers collecting the full array state so that all three layers can be compared.
⚠️ Warning: Capture array state and kernel logs as soon as you notice the issue. The kernel ring buffer is a fixed-size circular log that overwrites itself continuously, so waiting costs evidence even if you change nothing. Capture before stopping the array, rebuilding it, rebooting, or replacing the instance — all four destroy evidence, and on ephemeral drives it is not recoverable.
Prerequisites
- Root or Sudo Access on the Affected Instance
-
mdadmInstalled - An Assembled or Previously Assembled Software RAID Array
- Approximate UTC Timestamp of the Event Being Investigated
On Ubuntu and Debian images, install mdadm if it is not already present:
sudo apt-get update && sudo apt-get install -y mdadm
Instructions
ℹ️ Note: Every step below writes into the
~/diagdirectory created in Step 1. Stay in that directory for the whole capture, or the output files end up scattered and the bundle in Step 7 misses them.
Step 1: Capture Kernel Logs Unfiltered
Do this first. The ring buffer is finite and journald is not persistent by default on most images.
mkdir -p ~/diag && cd ~/diag sudo dmesg -T > dmesg-full.log sudo journalctl -k --no-pager > journalctl-kernel.log sudo cp /var/log/kern.log* . 2>/dev/null
Capture the complete output. Do not pre-filter to md127 or to a device name — array events are interleaved with systemd mount activity, container-runtime operations, and PCIe messages that provide the surrounding context.
Step 2: Capture Array State
Find what the kernel named your array first. The examples below use md127, which is common but not guaranteed — substitute the device you actually see:
cat /proc/mdstat > mdstat.txt cat mdstat.txt
/proc/mdstat gives the live summary — array level, member list, device count, and any resync or check in progress:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : active raid10 nvme7n1[7] nvme6n1[6] nvme5n1[5] nvme4n1[4] nvme3n1[3] nvme2n1[2] nvme1n1[1] nvme0n1[0]
3750241280 blocks super 1.2 256K chunks 2 near-copies [8/8] [UUUUUUUU]
bitmap: 0/28 pages [0KB], 65536KB chunk
unused devices: <none>[8/8] is expected members over present members. [UUUUUUUU] is one character per member — U for up, _ for a missing or failed slot. Reading positionally, [8/7] [U_UUUUUU] would tell you the second member is absent.
A resync or check in progress appears here as an additional progress line, which the steady-state output above does not have:
[>....................] resync = 0.3% (12000896/3750241280) finish=302.8min speed=205728K/sec
Then capture the authoritative view, substituting your array device:
sudo mdadm --detail /dev/md127 > mdadm-detail.txt
/dev/md127:
Version : 1.2
Creation Time : Fri Aug 28 18:45:29 2026
Raid Level : raid10
Array Size : 3750241280 (3.49 TiB 3.84 TB)
Used Dev Size : 937560320 (894.13 GiB 960.06 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sat Aug 29 00:07:24 2026
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 256K
Consistency Policy : bitmap
Name : node01:ephemeral (local to host node01)
UUID : 2dde7981:c577f54d:1d664345:c40392d5
Events : 3666
Number Major Minor RaidDevice State
0 259 5 0 active sync set-A /dev/nvme0n1
1 259 4 1 active sync set-B /dev/nvme1n1
2 259 0 2 active sync set-A /dev/nvme2n1
3 259 7 3 active sync set-B /dev/nvme3n1
4 259 3 4 active sync set-A /dev/nvme4n1
5 259 2 5 active sync set-B /dev/nvme5n1
6 259 1 6 active sync set-A /dev/nvme6n1
7 259 6 7 active sync set-B /dev/nvme7n1This is a healthy, fully synced array: all eight members active, nothing failed, and no resync or check running.
ℹ️ Note: A newly built array reports
State: clean, resyncingwith aResync Statusline instead. This is expected rather than a fault — a fresh RAID10 performs an initial resync to bring all mirror copies into agreement, which on eight NVMe drives of this size takes several hours, and the array is fully usable throughout.
The set-A / set-B labels in the member table are worth noting. With Layout: near=2, consecutive device slots form mirror pairs — slots 0 and 1 hold the same data, as do 2 and 3, and so on. This is how you determine which two drives are copies of each other, which matters whenever a finding appears to affect a specific pair rather than the array as a whole.
Fields worth reading carefully in --detail:
-
State—clean,clean, degraded,active, resyncing, or similar. Degraded means at least one member is absent. -
Raid DevicesvsTotal DevicesvsActive Devices— a gap between them locates the problem. Slots listed asremovedshow which positions are unfilled. -
Failed Devices— distinct from missing. A member can be absent without ever having been marked failed. -
Creation TimeandEvents— the array's creation timestamp and update counter. Record both; these are the values you compare against each member in Step 3. -
Name— the array name assigned at creation, used to identify the array across reboots. -
LayoutandChunk Size— the mirror geometry described above, and the stripe unit. Record both so a finding on a specific pair can be mapped back to drives.
Step 3: Examine Each Member Individually
The array-level view does not show per-member metadata. Collect it directly.
for dev in /dev/nvme[0-9]n1; do
echo "=== ${dev} ===" >> mdadm-examine.txt
sudo mdadm --examine "${dev}" >> mdadm-examine.txt 2>&1
doneCompare Creation Time, Update Time, and Events across members. Values that match across all members indicate a consistent view of the array. Note any member that returns No md superblock detected instead of metadata, and include the complete output in your bundle either way.
Step 4: Collect the Bad-Block Lists
This is the step most often skipped and frequently the one that matters.
for dev in /dev/nvme[0-9]n1; do
echo "=== ${dev} ===" >> mdadm-badblocks.txt
sudo mdadm --examine-badblocks "${dev}" >> mdadm-badblocks.txt 2>&1
doneAn empty list is reported as no bad blocks. A populated list shows sector ranges and run lengths that md has marked unreadable on that member.
ℹ️ Note: The bad-block list is
md's own bookkeeping, not a drive-reported defect list. Entries here do not mean the drive has failing media — that question is answered by the drive's SMART and error logs, collected separately. Always gather both before drawing a conclusion about which layer is at fault.
⚠️ Warning: Each member's list has a fixed capacity of 512 entries. A member at 512/512 can no longer track new ranges, and
mdmay then fail that member out of the array. Note the entry count per member when you collect it.
Step 5: Capture Array Configuration and Mounts
findmnt > findmnt.txt cat /etc/fstab > fstab.txt cat /etc/mdadm/mdadm.conf > mdadm-conf.txt 2>/dev/null uname -r > kernel-version.txt
This establishes where the array is mounted and what depends on it. If the array backs /var/lib/kubelet or /var/lib/containerd, that relationship explains why an array-level problem surfaces as pod scheduling or image pull failures rather than as an obvious disk error.
mdadm.conf matters for a different reason: if the array definition was never written there, the array will not assemble predictably on the next boot regardless of its current health.
Step 6: Collect Drive-Level Health
Array diagnostics are incomplete without the layer beneath them. Follow How-To Collect NVMe Drive Diagnostics and add that output to the same ~/diag bundle.
Step 7: Bundle and Attach to Your Support Ticket
cd ~ && tar czf raid-diag-$(date -u +%Y%m%dT%H%M%SZ).tgz diag/
Attach the archive to your Crusoe support ticket with the approximate UTC timestamp in the ticket body. Attach the file rather than pasting output inline.
Example
Pods on one node start failing to pull images, with containerd reporting write errors into /var/lib/containerd. The array backing it reports itself healthy: /proc/mdstat shows [8/8] [UUUUUUUU] and mdadm --detail reports State: clean with zero failed devices. On that evidence alone the array looks fine and the problem looks like a containerd bug.
Step 4 is what settles it. mdadm --examine-badblocks on each member returns empty for seven drives and a populated list of sector ranges for nvme5n1 — regions md has marked unreadable without failing the drive out. That is exactly the state the introduction warns about: a complete, clean-looking array with specific regions that cannot be read. Any file whose blocks land in those ranges fails, which is why the symptom looked like a random subset of image pulls rather than a disk fault.
Pairing that with the NVMe capture from Step 6 separates the two possible causes. If nvme5's SMART log shows non-zero media_errors, the drive is failing and the bad-block entries are downstream of that. If the SMART log is clean, the write failures came from somewhere else — a transient transport event, or pressure during a resync — and the drive itself is healthy. Either way the finding is actionable, and neither conclusion is reachable from the array-level view alone.