Introduction
Local NVMe drives on Crusoe GPU instances are attached through VFIO passthrough. Your guest kernel owns the block layer end to end and issues commands directly to the physical drives, with no hypervisor emulation in the path.
That architecture has a direct consequence for troubleshooting. Crusoe can independently verify PCIe link speed and width, AER and IOMMU fault counters, drive enumeration, and host-level hardware events. Crusoe cannot see anything your guest kernel observes about the drives. For any drive-level question that arises inside the instance, the capture you take from inside the instance is the only record that exists.
There are two independent sources of truth worth understanding before you collect anything.
The drive's own controller maintains a SMART log and an error log in firmware. These record what the device itself experienced, are unaffected by anything happening in your kernel, and persist across guest reboots.
Your guest kernel maintains a separate record of what it observed while talking to the drive. The two can disagree, and when they do, that disagreement is itself diagnostic information. Collecting only one of them tells half the story.
This article covers collecting both, plus the enumeration and topology context that makes them interpretable.
⚠️ Warning: Capture kernel logs as soon as you notice the issue, and always before rebooting or replacing the instance. The kernel ring buffer is a fixed-size circular log — on a busy node it overwrites itself continuously and can age out hours of history on its own, so evidence disappears whether or not you touch anything. A reboot clears it outright, and replacing the instance destroys the ephemeral drives along with every trace of what happened.
Prerequisites
- Root or Sudo Access on the Affected Instance
-
nvme-cliInstalled - Approximate UTC Timestamp of the Event Being Investigated
On Ubuntu and Debian images, install nvme-cli if it is not already present:
sudo apt-get update && sudo apt-get install -y nvme-cli
Instructions
ℹ️ Note: Every step below writes into the
~/diagdirectory created in Step 1. Stay in that directory for the whole capture, or the output files end up scattered and the bundle in Step 5 misses them.
Step 1: Capture Kernel Logs Unfiltered
Do this first. Everything else can wait; the ring buffer cannot.
mkdir -p ~/diag && cd ~/diag sudo dmesg -T > dmesg-full.log sudo journalctl -k --no-pager > journalctl-kernel.log sudo cp /var/log/kern.log* . 2>/dev/null sudo cp /var/log/syslog* . 2>/dev/null
Capture the complete output. Do not pre-filter to a device name or an error string.
ℹ️ Note: A capture narrowed to
grep nvmelooks complete but silently drops the PCIe, IOMMU, memory-mapping,systemd, and container-runtime lines immediately surrounding the event. Those adjacent lines are frequently what makes a capture diagnostic rather than merely confirmatory. If the logs are large, compress rather than trim.
Step 2: Confirm Drive Enumeration
Establish which drives the guest can see and at what capacity.
sudo nvme list > nvme-list.txt lsblk -f > lsblk.txt sudo dmesg -T | grep -i nvme > nvme-enumeration.txt
nvme list reports the model, serial, firmware revision, and namespace size for every drive. lsblk -f shows how those drives are currently consumed — filesystem, array membership, or unused.
The dmesg extract in this step is a supplement to the full capture from Step 1, not a replacement for it. On a healthy instance it shows only PCI function assignment and queue setup at boot:
nvme nvme0: pci function 0002:00:05.0 nvme nvme0: 176/0/0 default/read/poll queues
Step 3: Collect Drive Health and Error Logs
These query the drive controller directly, so they take the controller device (/dev/nvme0) rather than the namespace (/dev/nvme0n1).
for dev in /dev/nvme[0-9]; do
[ -e "${dev}" ] || continue
echo "=== ${dev} ===" >> nvme-smart.txt
sudo nvme smart-log "${dev}" >> nvme-smart.txt
echo "=== ${dev} ===" >> nvme-errors.txt
sudo nvme error-log "${dev}" >> nvme-errors.txt
doneReading the SMART Log
A healthy drive returns output along these lines:
Smart Log for NVME device:nvme2 namespace-id:ffffffff critical_warning : 0 temperature : 28 C (301 Kelvin) available_spare : 100% available_spare_threshold : 10% percentage_used : 0% endurance group critical warning summary: 0 data_units_read : 45,083,472 data_units_written : 54,427,492 host_read_commands : 138,540,005 host_write_commands : 177,128,485 controller_busy_time : 530 power_cycles : 64 power_on_hours : 21,575 unsafe_shutdowns : 41 media_errors : 0 num_err_log_entries : 0 Warning Temperature Time : 0 Critical Composite Temperature Time : 0
The fields that carry the most signal:
-
critical_warning— non-zero indicates the drive is reporting a fault condition such as thermal throttling, degraded reliability, or exhausted spare capacity. -
media_errors— count of unrecoverable data errors the drive itself detected. Zero means the media has not reported a problem. -
num_err_log_entries— how many error log entries the drive has actually recorded. Zero means the drive has never failed a command in its lifetime. -
percentage_used— endurance consumed as a proportion of rated write life. -
available_spare/available_spare_threshold— remaining over-provisioned blocks. Approaching the threshold indicates genuine wear. -
unsafe_shutdowns— expected to be non-zero on ephemeral drives that have been reassigned between instances, and not by itself a fault indicator.
ℹ️ Note:
power_on_hours,power_cycles, andunsafe_shutdownsreflect the drive's entire service life, not your instance's use of it. Local NVMe drives are reassigned between instances, so a drive attached to a node you created this morning will still report accumulated hours and shutdown counts. High values in these fields are normal and are not a fault signal on their own.
Reading the Error Log
The error log is a fixed-size ring. Output from a healthy drive looks like this:
Error Log Entries for device:nvme2 entries:64 ................. Entry[ 0] ................. error_count : 0 sqid : 0 cmdid : 0 status_field : 0(SUCCESS: The command completed successfully) phase_tag : 0 parm_err_loc : 0 lba : 0 nsid : 0 vs : 0 trtype : The transport type is not indicated or the error is not transport related. cs : 0 trtype_spec_info: 0 ................. [Entries 1 through 63 omitted — identical, all zero]
⚠️ Warning:
entries:64is the capacity of the log ring, not a count of errors. A healthy drive prints all 64 slots, every one of them zeroed. Readerror_countandstatus_fieldwithin each entry to determine whether anything was actually recorded — a populated entry carries a non-zeroerror_count, a failingstatus_field, and thelbaat which the command failed.
💡 Tip: Check
num_err_log_entriesin the SMART log first. If it reads0, the error log contains nothing and you can skip reading all 64 entries — but still include the file in your bundle, since an empty error log is itself a meaningful finding.
💡 Tip: Collect from all eight drives even when you suspect only one. A single drive's readings are hard to interpret in isolation; the same readings across seven healthy siblings make an outlier obvious immediately.
Step 4: Capture PCIe Topology
When a question involves a specific subset of drives, the mapping between guest device names and PCIe addresses is needed to interpret it. Follow How-To Map Guest NVMe Devices to PCIe BDF Topology and save that output into ~/diag with the rest of the bundle.
Step 5: Bundle and Attach to Your Support Ticket
cd ~ && tar czf nvme-diag-$(date -u +%Y%m%dT%H%M%SZ).tgz diag/
Attach the archive to your Crusoe support ticket and include the approximate UTC timestamp in the ticket body. Attach the file rather than pasting log contents inline — pasted logs are frequently truncated, and truncation usually removes the section that matters.
Step 6: Enable Persistent Kernel Logging
Do this on every instance before you need it. A common reason a drive question goes unanswered is that the evidence rotated away before anyone went looking.
sudo mkdir -p /var/log/journal sudo sed -i 's/^#\?Storage=.*/Storage=persistent/' /etc/systemd/journald.conf sudo systemctl restart systemd-journald
Creating /var/log/journal is the part that matters: with the default Storage=auto, journald writes to disk only when that directory exists. Setting Storage=persistent makes the intent explicit so the behavior does not depend on the image's default.
Confirm it took effect:
journalctl --disk-usage
A persistent journal reports usage under /var/log/journal. If it still reports /run/log/journal, logs are in memory only and will not survive a reboot.
💡 Tip: If you provision instances from a node pool or automation, apply this in your bootstrap rather than per node. Instances are typically replaced faster than a manual change can propagate, and a replacement loses the setting along with everything else.
Example
A training job writing to local NVMe starts reporting I/O errors on one node partway through a run. The kernel log shows blk_update_request: I/O error against nvme3n1, and the job aborts.
The capture takes about a minute. dmesg-full.log shows the I/O errors arriving in a burst, with no PCIe AER or IOMMU messages anywhere near them. nvme-list.txt shows all eight drives enumerated at full capacity. The SMART logs are where the answer is: seven drives report media_errors: 0 and num_err_log_entries: 0, while nvme3 reports a non-zero media_errors count and a matching num_err_log_entries. The error log for nvme3 carries populated entries with a failing status_field and the lba of each failed command.
That combination — the drive's own firmware reporting media errors, the seven siblings clean, and no transport-layer faults — points at the drive rather than at the PCIe path or the guest's block layer, and it is exactly what Crusoe Support needs to act on the hardware. Had the capture been narrowed to grep nvme, the absence of AER messages would not have been demonstrable, and ruling out the transport is half of what makes this conclusive.