NVMe Storage Read-Only Errors on H100 Instances

Last Updated: Jan 15, 2026

Overview

This guide is for addressing a specific, now-rare issue on some H100 instances where the local NVMe storage disk unexpectedly becomes read-only. This was historically observed on a small portion of our fleet with a specific combination of system BIOS and Micron NVMe drives. While Crusoe has proactively upgraded the vast majority of the fleet to remediate this, it's possible to encounter it in rare cases.

The primary symptom you'll see from within your VM is the inability to write new data to the local disk, especially under heavy I/O load. When you check the kernel logs (dmesg), you will likely find a series of I/O errors that ultimately result in the filesystem being remounted as read-only. This guide will help you collect the necessary logs for our team to verify the underlying hardware and firmware, and take action if needed.

Prerequisites

• You will need SSH access to the affected Virtual Machine.

• The nvme-cli utility should be installed. If it is not, you can install it on Ubuntu/Debian-based systems with the following command:

  # sudo apt-get update && sudo apt-get install nvme-cli -y

Steps

1. Connect to the Affected VM
   • Use SSH to log in to the instance where the read-only filesystem issue is occurring.
2. Identify the NVMe Drive Model

   • Run the following command to list the NVMe devices and their model numbers. This helps our team identify if the hardware is a model known to be affected by this issue.

     $ sudo nvme list

   • Copy the output of this command. It will look similar to the following:

     Node                  SN                   Model                                    Namespace Usage                      Format           FW Rev
     --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
     /dev/nvme0n1          [...]                Micron_7450_MTFDKCC3T8TGH                  1           3.84  TB /   3.84  TB    512   B +  0 B   E3MU005
     /dev/nvme1n1          [...]                Micron_7450_MTFDKCC3T8TGH                  1           3.84  TB /   3.84  TB    512   B +  0 B   E3MU005
     [...]

3. Capture Kernel Diagnostic Logs

   • Next, run the following command to display and capture messages from the kernel ring buffer. This log contains detailed information about hardware events and the errors that led to the read-only state.

     $ dmesg

   • Copy the entire output. Key indicators of this issue to look for in the logs include:
     • blk_update_request: I/O error, dev nvme2n1, sector [...]
     • Buffer I/O error on device md127, logical block [...]
     • EXT4-fs warning (device md127): ext4_end_bio:344: I/O error 10 writing to inode [...]
     • EXT4-fs error (device md127): ext4_journal_check_start:83: comm python: Detected aborted journal
     • And finally: EXT4-fs (md127): Remounting filesystem read-only
4. Open a Support Ticket
   • Please open a support ticket with our team using the pre-filled link in the 'Resolution' section below.
   • In the ticket, paste the full output from both the nvme list and dmesg commands into the appropriate sections. Be sure to also include the name or ID of the affected VM.

Resolution

1. Crusoe Triage and Investigation
   • Once you've submitted the ticket with the required logs, our Cloud Support Engineering team will analyze the diagnostics to confirm if the symptoms and hardware profile point to this known firmware incompatibility.
2. Scheduled Host Maintenance
   • If the issue is confirmed, we will coordinate with you to schedule a maintenance window. During this time, you will need to power down the affected VM. Our infrastructure team will then perform the necessary BIOS or firmware upgrades on the underlying host to resolve the PCIe configuration issue.
3. Confirmation and Follow-Up
   • After the maintenance is complete, we'll notify you that it's safe to start your VM again. It looks like this maintenance typically does the trick, but if the problem reappears on the same instance, just reply to your ticket to let us know.

Additional Resources

• Create a Pre-filled Support Ticket for This Issue