How-To Validate GPU ECC and Row Remap Status Using nvidia-smi

Introduction

When Crusoe detects a GPU hardware health event on your node — such as a pending HBM memory row remap — you may want to independently verify the current state of your GPU from inside your VM. This article explains how to use nvidia-smi to check for pending row remaps, offlined memory pages, and related ECC error counters, and how to interpret what you find.

Understanding this output is useful both for confirming the condition Crusoe has alerted you to and for ongoing monitoring of GPU memory health during long-running workloads.

ℹ️ Note: The commands in this article must be run from inside your VM as a user with sudo privileges. They query the NVIDIA driver's view of the GPU and will reflect the current state as seen from the guest OS.

Prerequisites

• SSH Access to Your VM
• NVIDIA Driver Installed and Running Inside the VM
• nvidia-smi Available in the VM's PATH
• sudo Privileges Inside the VM

Instructions

Step 1: Check Row Remap Status

Run the following command to check the current row remap state for all GPUs on your node:

nvidia-smi -q | grep -A 6 "Remapped Rows"

You will see a Remapped Rows block for each GPU. Example output for a GPU with a pending remap:

Remapped Rows
    Correctable Error             : 0
    Uncorrectable Error           : 1
    Pending                       : Yes
    Remapping Failure Occurred    : No

Interpreting the fields:

• Correctable Error — The number of remap entries recorded in the GPU's InfoROM due to correctable (single-bit) ECC errors. A non-zero value is generally low severity but worth monitoring.
• Uncorrectable Error — The number of remap entries recorded due to uncorrectable (double-bit) ECC errors. A non-zero value means at least one bad HBM memory cell has been detected and a hardware repair has been scheduled.
• Pending: Yes — The hardware repair has been scheduled and durably recorded to the GPU's InfoROM, but has not yet been committed. The repair will take effect on the next full node power cycle. This is the field that triggers a GPURowRemapPending alert from Crusoe's monitoring systems.
• Pending: No — No pending remap. If this follows a node power cycle, it confirms the hardware repair was successfully committed.
• Remapping Failure Occurred: No — The repair is viable. The GPU is stable and no further escalation is required at this time.
• Remapping Failure Occurred: Yes — The remap entry failed to write to the InfoROM (XID 64). This is a serious condition. Contact Crusoe Support immediately and reference your open ticket.

⚠️ Warning: If Remapping Failure Occurred is Yes, do not attempt to continue running workloads on this GPU. Open a support ticket immediately if you do not already have one open.

ℹ️ Note: The Correctable Error and Uncorrectable Error counts reflect entries recorded in the InfoROM — not the number of rows physically remapped in hardware yet. All pending entries are committed together on the next full node power cycle.

Step 2: Check Offlined Page Status

Run the following command to confirm whether the affected memory page has been offlined by the NVIDIA driver:

nvidia-smi -q | grep -A 3 "Retired Pages"

Example output:

Retired Pages
    Single Bit ECC                : N/A
    Double Bit ECC                : N/A
    Pending Page Blacklist        : Yes

Interpreting the fields:

• Pending Page Blacklist: Yes — The NVIDIA driver has offlined the affected memory page. It will not be allocated to any current or future workload on this node. This protection is applied immediately when the error is detected and does not require a reboot to take effect.
• Pending Page Blacklist: No — No pages are currently pending blacklist. If this follows a node power cycle after a pending remap was committed, this is expected and confirms the repair is complete and the page has been reclaimed.

ℹ️ Note: On Ampere and later GPU architectures (H100, H200, B200), Single Bit ECC and Double Bit ECC under Retired Pages will show N/A — page retirement is a legacy mechanism replaced by row remapping on these GPUs. This is expected output and does not indicate a problem.

Step 3: Check for Relevant Xid Errors in System Logs

Xid errors are GPU error codes logged by the NVIDIA driver to the system journal. Checking for these can help you correlate a workload crash or instability with a specific GPU hardware event.

Run the following to check for recent GPU Xid errors:

sudo dmesg | grep -i "xid"

Or, if your system uses journalctl:

sudo journalctl -k | grep -i "xid"

Relevant Xid codes for ECC and row remap events:

• Xid 48 — An uncorrectable ECC error occurred. This is the underlying error that triggers a row remap entry.
• Xid 63 — A row remap entry was successfully written to the InfoROM. This confirms the hardware repair has been durably scheduled.
• Xid 64 — A row remap entry failed to write to the InfoROM. Contact Crusoe Support immediately.
• Xid 94 — A contained ECC error occurred. The application that encountered it is affected and must be restarted, but other workloads on the GPU are unaffected.
• Xid 95 — An uncontained ECC error occurred. This is more severe — contact Crusoe Support immediately.
• Xid 160 — GPU memory has been successfully marked for repair.

ℹ️ Note: Xid 94 is the most common error associated with a workload crash at the time a bad memory cell is first encountered. If your workload crashed unexpectedly, check your dmesg logs for Xid 94 or Xid 48 around the time of the crash.

Step 4: Run a Full GPU Query (Optional)

For a complete snapshot of GPU health state, run:

nvidia-smi -q

This outputs the full status of all GPUs on the node, including ECC error counters, clock speeds, temperature, memory usage, and process information. If you are opening a support ticket or have been asked by Crusoe Support to provide GPU state information, this is the recommended command to run and share output from.

Example

You receive a notification from Crusoe Support that a GPURowRemapPending event has been detected on GPU_SXM_5 on your node. You SSH into your VM and run:

nvidia-smi -q | grep -A 6 "Remapped Rows"

You see the following in the output block corresponding to GPU 5:

Remapped Rows
    Correctable Error             : 0
    Uncorrectable Error           : 1
    Pending                       : Yes
    Remapping Failure Occurred    : No

You then run:

nvidia-smi -q | grep -A 3 "Retired Pages"

And see:

Retired Pages
    Single Bit ECC                : N/A
    Double Bit ECC                : N/A
    Pending Page Blacklist        : Yes

This confirms:

• One uncorrectable ECC error has been detected and a remap entry recorded to the InfoROM.
• The hardware repair is pending and will be committed on the next full node power cycle.
• The affected memory page has been offlined by the driver and will not be reallocated to your workloads.
• No remap failure has occurred — the GPU is stable and the repair is viable.

You reply to the Crusoe Support ticket with this output and confirm your preferred next step (migrate to spare node or continue on current node).

Additional Resources

• NVIDIA GPU Memory Error Management Documentation
• How-To Respond to Uncorrectable ECC Errors