How-To Respond to Uncorrectable ECC Errors

Last Updated: Jan 15, 2026

Introduction

High-performance GPUs rely on Error Correction Code (ECC) memory to automatically fix minor, transient memory errors. However, when a workload fails with a message like 'Cuda failure: uncorrectable ECC error encountered', it signifies a physical fault in the GPU's memory that the hardware cannot fix on its own.

This is a strong indicator of a potential hardware issue that requires investigation. These physical memory faults can also cause secondary symptoms, such as the GPU becoming unresponsive or the NVIDIA driver logging specific Xid errors to the kernel log.

This guide outlines the critical steps to collect the diagnostic information our team needs to verify the hardware issue and arrange a swift resolution.

Prerequisites

• SSH access to the affected VM.
• Root or sudo privileges are required to run diagnostic commands.

Step-by-Step Instructions

1. Check GPU Status with nvidia-smi

   • The most direct way to confirm the issue is to query the NVIDIA System Management Interface (nvidia-smi) and check the ECC error counters.

     # Run the query command with administrative privileges
     $ sudo nvidia-smi -q
     

   • In the output for each GPU, look for the Ecc Errors section. A non-zero count for DRAM Uncorrectable or a high number of SRAM Correctable errors under the Aggregate section is a primary symptom.

     ...
         ECC Errors
             ...
             Aggregate
                 SRAM Correctable              : 26
                 ...
                 DRAM Uncorrectable            : 1

2. Inspect Kernel Messages for Corroborating Errors

   • Next, check the kernel's message buffer for related errors. Uncorrectable ECC faults often cause the NVIDIA driver to log a Xid (Exception ID) error.

     # View the system's message log and filter for NVIDIA Xid errors
     $ sudo dmesg | grep "NVRM: Xid"

   • Look for the following Xid errors, which are direct symptoms of ECC-related hardware faults:
     • Xid 48: This is a definitive indicator. The log message will explicitly state 'An uncorrectable double bit error (DBE) has been detected'.
     • Xid 63: This indicates the GPU's Row Remapper has identified a faulty memory row and marked it for retirement, which is a direct response to a detected hardware error.
     • Xid 43: This indicates a general GPU memory page fault, which is often a consequence of an underlying ECC error.
3. Generate a Full NVIDIA Bug Report

   • To give our team a complete picture of the hardware and driver state, please generate a comprehensive NVIDIA bug report.

     # Run the bug report script. This will create a file in your current directory.
     $ sudo nvidia-bug-report.sh

   • This command will create a compressed log file named nvidia-bug-report.log.gz.

Resolution

1. Open a Support Ticket for Escalation
   • Uncorrectable ECC errors are not user-resolvable and indicate a potential physical hardware issue that requires Crusoe to intervene.
   • Please open a support ticket immediately. To make this process as efficient as possible, you can use our pre-filled link, which prompts you for all the necessary information: Report Uncorrectable ECC Errors.
2. Attach All Diagnostic Logs
   • When you create the ticket, please attach the three pieces of information you gathered in the steps above:
     • The complete text output of the nvidia-smi -q command.
     • The complete text output of the dmesg command.
     • The nvidia-bug-report.log.gz file.
3. Expect a Host Migration
   • Once our team receives the logs and confirms the hardware fault, the diagnostics will point toward the need for a host migration. Our standard procedure is to coordinate with you to perform a 'STOP' and 'START' operation on your instance, which will move it to a healthy, stable physical server and resolve the issue.