Introduction
If you have received a notification from Crusoe Support about a pending HBM memory row remap on your node, or if you have encountered GPU ECC errors during a workload run, this article explains what is happening at the hardware level, why it occurs, and what it means for your workloads.
Crusoe's GPU fleet uses High Bandwidth Memory (HBM) — a high-density, high-throughput memory technology integrated directly onto the GPU package alongside the compute dies. HBM is purpose-built for the memory bandwidth demands of large-scale AI/ML training and inference workloads. Because it is soldered directly to the GPU package rather than installed as a removable module, managing HBM degradation requires a different approach than traditional DRAM — one that is handled automatically in hardware and firmware, with Crusoe monitoring and responding on your behalf.
Understanding how this works helps you interpret GPU health alerts, make informed decisions about your workloads, and know when to act.
Prerequisites
- Basic Familiarity With GPU-Accelerated Workloads
- A Crusoe VM With One or More NVIDIA GPUs Attached
Instructions
Step 1: Understand What HBM Is and Why It Degrades
HBM is stacked DRAM integrated directly onto the GPU package via a silicon interposer. On NVIDIA data center GPUs such as the H100, H200, and B200, HBM provides the memory bandwidth required for large model weights, activations, and gradients to move between GPU compute cores at the speeds necessary for efficient training and inference.
Like all DRAM, HBM memory cells can degrade over time. Degradation can result from manufacturing variation, accumulated electrical stress, or cosmic ray-induced bit flips. When a memory cell degrades to the point where it can no longer reliably store data, it produces an ECC (Error Correcting Code) error.
There are two classes of HBM ECC errors:
- Correctable errors (single-bit) — The ECC circuitry detects and corrects the error automatically. Your workload is unaffected. These are worth monitoring as an early indicator of a cell that may eventually produce an uncorrectable error.
- Uncorrectable errors (double-bit) — The ECC circuitry detects the error but cannot correct it. The application that encountered the bad memory location is affected — typically resulting in a crash or abnormal exit. This is the class of error that triggers a row remap.
Step 2: Understand What a Row Remap Is
Row remapping is a hardware reliability mechanism built into NVIDIA Ampere and later GPU architectures (H100, H200, B200). Every HBM memory bank on these GPUs is provisioned at the factory with a set of spare rows — additional memory rows held in reserve specifically for this purpose.
When an uncorrectable ECC error is detected on a memory cell, the GPU schedules a repair: the degraded memory row will be replaced with one of those spare hardware rows. This is called a row remap. The repair is permanent — once committed, it persists for the life of the GPU — and it does not leave a hole in the GPU's memory address space. From your workload's perspective, the memory is simply fixed.
Row remapping replaces an older mechanism called page retirement used on pre-Ampere GPUs. Page retirement permanently removed the bad memory page from the available address space, reducing total GPU memory capacity over time. Row remapping avoids this by physically replacing the bad row with a spare, reclaiming the full address space after the repair is committed.
ℹ️ Note: NVIDIA H100, H200, and B200 GPUs support up to 512 total row remappings before RMA (Return to Manufacturer) criteria apply. A single pending remap is a routine hardware maintenance event — not an indication that the GPU is failing.
Step 3: Understand the Two-Phase Repair Process
Row remapping happens in two distinct phases. Understanding both helps clarify why a node can remain operational after a remap event is detected, and what risk — if any — remains during that time.
Phase 1 — Dynamic Page Offlining (Immediate, No Reboot Required)
The moment an uncorrectable ECC error is detected, the NVIDIA driver immediately marks the affected memory page as unusable. This takes effect instantly — no GPU reset or node reboot is required. The offlined page is removed from the allocatable address space: neither your currently running workloads nor any newly launched workloads will be assigned that page.
This means that by the time Crusoe notifies you of a pending row remap, the affected page is already offlined. Your workloads will not encounter that specific bad memory cell again.
This protection is durable. The NVIDIA driver records the offlined page in the GPU's InfoROM (see Step 4) and re-applies the page offlining automatically on every subsequent driver initialization — including after a VM restart. You do not need to reboot to maintain this protection.
Phase 2 — Hardware Row Remap (Permanent, Requires Full Node Power Cycle)
While Dynamic Page Offlining provides immediate protection, it is a software-level measure — the underlying hardware cell is still degraded. The permanent fix is the hardware row remap, which physically replaces the degraded memory row with a spare hardware row. This repair:
- Requires a full node power cycle (a bare-metal reboot of the underlying host) to take effect.
- Is permanent and persistent for the life of the GPU once committed.
- Reclaims the previously offlined page — after the power cycle, the memory is fully repaired and the page is available again.
Until the power cycle occurs, the node is in a pending state: the repair has been scheduled and durably recorded, but not yet committed in hardware.
⚠️ Warning: A VM reset or guest-level GPU reset will not commit the hardware row remap. Only a full node power cycle at the bare-metal level triggers the hardware repair. Do not attempt a VM reset as a remediation step for a pending row remap.
Step 4: Understand the InfoROM
The InfoROM is a small, persistent storage component physically on the GPU board itself. It is completely independent of the host OS, the NVIDIA driver, and any software running in your VM. The InfoROM stores GPU configuration and health state data across the lifetime of the GPU, including ECC error records and row remap entries.
When an uncorrectable ECC error triggers a row remap, the GPU writes the remap entry to the InfoROM. This write is confirmed by XID 63 in the system logs — a log entry indicating that the repair has been durably scheduled. Because the entry lives in the InfoROM on the GPU board itself, it survives driver reloads, VM resets, and hypervisor reboots. The repair instruction is not lost under any circumstance short of a hardware failure of the InfoROM itself.
This is also why the NVIDIA driver can re-apply Dynamic Page Offlining automatically after a driver reload or VM restart — it reads the InfoROM on initialization and reconstructs the list of offlined pages from the durable record stored there.
ℹ️ Note: The row remap counts reported by
nvidia-smiand Crusoe's monitoring systems reflect the number of remap entries recorded in the InfoROM — not the number of rows physically remapped in hardware yet. All pending entries are committed together on the next full node power cycle.
Step 5: Understand What "Pending" Means for Your Workload
When Crusoe notifies you of a GPURowRemapPending event, here is the state of your GPU at that moment:
- The bad memory cell has been detected.
- The affected page has already been dynamically offlined — your workloads will not encounter that specific cell.
- A hardware repair entry has been written to the InfoROM and is waiting to be committed on the next power cycle.
- The GPU is operational. There is no current impact to your running workloads from the specific cell that has been detected.
The remaining risk is not from the already-offlined page — it is that additional HBM cells on the same GPU may degrade before the hardware repair is committed. Each new bad cell would cause the workload that encounters it to crash before the driver can offline the new page. It is worth noting that the presence of one degraded memory cell can be an indicator that other cells on the same GPU may be similarly susceptible — making it difficult to predict whether this will remain an isolated event or develop further.
This is why Crusoe recommends migrating to a spare node when one is available — not because the current condition is immediately dangerous, but because it eliminates the risk of additional cell degradation impacting your workloads before the repair can be performed.
Step 6: Understand When Escalation Is Required
Most pending row remap events are routine maintenance conditions that resolve cleanly on the next node power cycle. However, there are scenarios that require immediate attention:
Remapping Failure (XID 64)
If the GPU fails to write the remap entry to the InfoROM, XID 64 is emitted and the Remapping Failure Occurred flag is set. This means the hardware repair could not be scheduled. Contact Crusoe Support immediately if you encounter this — do not continue running workloads on the affected GPU.
Multiple Remaps on the Same HBM Bank
NVIDIA's RMA policy considers a GPU eligible for replacement when a memory bank has accumulated a threshold of uncorrectable error remaps, or when the same row is remapped more than once. If Crusoe notifies you of multiple remap events on the same GPU in a short period, this may indicate accelerating HBM degradation and will be escalated to Crusoe's infrastructure team for evaluation.
Repeated Workload Crashes
If you are experiencing repeated, unexplained workload crashes — particularly accompanied by CUDA errors such as CUDA_ERROR_ECC_UNCORRECTABLE, or Xid 94 or Xid 48 in your system logs — this may indicate that additional HBM cells are degrading faster than Dynamic Page Offlining can protect against. Open a support ticket immediately and reference any prior ticket related to GPU health on that node.
Example
During a multi-day training run, your workload crashes unexpectedly with a CUDA ECC error. You check your system logs and find an Xid 94 entry around the time of the crash. Shortly after, you receive a notification from Crusoe Support indicating that a GPURowRemapPending event has been detected on GPU_SXM_5 on your node.
This is the expected sequence of events. The Xid 94 indicates that your workload encountered the bad memory cell directly, causing it to crash. By the time Crusoe's notification reaches you, the affected page has already been dynamically offlined — restarting your workload will not encounter that same cell again. The hardware repair has been scheduled in the InfoROM and will be committed on the next full node power cycle.
Crusoe Support will offer you the option to migrate to a spare node (recommended) or continue on your current node with the understanding that additional cell degradation is possible. Your restarted workload can resume in the meantime — the offlined page will not be reallocated.