Introduction
This FAQ covers what Crusoe can and cannot observe about the local NVMe drives attached to your GPU instance, and what that means when you report a storage issue.
Question 1: Why Can't Crusoe See the Storage Errors My Kernel Is Reporting?
Answer: Local NVMe drives are attached through VFIO passthrough. The drive is bound to the host's vfio-pci driver, isolated behind the IOMMU, and mapped directly into your instance, where your guest kernel loads its own nvme driver and talks to the controller directly.
Unlike typical virtualized block storage — where the hypervisor emulates a disk and therefore sits in the I/O path — there is no emulation layer here. Queues live in guest memory and DMA moves data straight between the drive and your instance. Everything above the PCIe transport is invisible to the host; everything at and below it remains observable to Crusoe.
Question 2: What Can Crusoe Check on Its Own When I Report a Storage Issue?
Answer: The following, with no input from you:
- PCIe link health — negotiated speed and width, and whether a link has degraded.
- PCIe error counters — correctable and uncorrectable AER events on the drive and every port upstream of it.
- IOMMU and DMAR faults — translation failures or protection violations on the passed-through devices.
- Device enumeration and binding — whether all expected drives are present, correctly bound, and reporting their expected firmware revision.
- Host kernel and platform logs — anything the host observed at the transport or platform layer.
- BMC and system event logs — thermal excursions, power events, and hardware faults recorded out of band.
- Offline drive diagnostics — full SMART and error log reads, plus link and error checks, once the node is drained.
A genuine hardware fault on a drive or its PCIe path shows up here.
Question 3: What Can't Crusoe See?
Answer: Everything your guest kernel observes, while your instance is running:
- Guest kernel logs of any kind, including block layer messages
- I/O errors, command completions, timeouts, or retries as your kernel experienced them
- Filesystem state, mount status, or remount events
- Software RAID, LVM, or any other guest storage stack and its metadata
- Drive SMART and error logs, which answer to the guest while the drive is passed through
- Per-drive throughput, latency, or queue depth as measured inside your instance
- Which workloads, containers, or files were involved
Question 4: Can Crusoe Read the SMART Data From My Drives?
Answer: Not while your instance is running. From provisioning until the instance stops, no host nvme driver is attached — the host cannot poll for SMART data, see command completions, or account for I/O. The drives are unbound, sanitized, and returned to the pool when the instance stops, and only in that window can the host query them directly.
ℹ️ Note: This is why hardware diagnostics on a specific drive require draining the node. It is not a scheduling preference; the host genuinely cannot read the drive while it is passed through to a running instance.
Question 5: Why Not Emulate the Drives So Crusoe Has Visibility Into Them?
Answer: The isolation that stops the host observing your I/O is the same isolation that stops it interfering. IOMMU-enforced device assignment is a tenancy boundary, and keeping the host out of your data path is both a security and a performance property. Emulation would restore visibility at the cost of a software layer between your workload and the hardware on every I/O, which is not a reasonable trade for these workloads. The asymmetry is a consequence of direct hardware access, and it does place a real burden on you during troubleshooting.
Question 6: What Should I Capture Before Opening a Support Ticket?
Answer: For anything above the PCIe transport, your guest-side capture is the only record that will ever exist. Crusoe checks the host side and will ask you for the guest side, so sending both with your initial report removes a round trip — often several.
What to collect depends on what you are running:
- How-To Collect NVMe Drive Diagnostics — for any drive-level question.
-
How-To Collect Software RAID Diagnostics — additionally, if the drives are combined into an
mdadmarray.
The one thing worth doing before you need it is making journald persistent:
sudo mkdir -p /var/log/journal sudo sed -i 's/^#\?Storage=.*/Storage=persistent/' /etc/systemd/journald.conf sudo systemctl restart systemd-journald
The kernel ring buffer is finite and journald is not persistent by default on most images, so evidence from an incident noticed hours later has frequently already rotated away.
Question 7: I Already Rebooted the Instance. Can the Issue Still Be Investigated?
Answer: Partially. Crusoe can still check everything in Question 2 — those records live on the host and survive your instance. Anything your guest observed is gone, since a reboot clears the kernel ring buffer unless persistent logging was already enabled.
⚠️ Warning: Replacing or rebuilding an instance is worse than rebooting it. It destroys the ephemeral drives along with every record of what happened on them, including array metadata and filesystem state. Capture diagnostics before taking recovery actions, even when restoring service is the priority.