How-To Resolve NVLink Communication Errors (Xid 149, 154) on Crusoe Cloud VMs

Last Updated: Jan 15, 2026

Introduction

Crusoe Cloud's high-performance compute instances use NVIDIA NVLink, a high-speed interconnect that allows the GPUs within a server to communicate directly with each other at extremely high bandwidths. This technology is essential for the performance of many large-scale AI and HPC workloads.

When you see NVIDIA Xid 149 or Xid 154 errors in your VM's kernel logs (dmesg), the logs are pointing toward a potential issue with this critical NVLink communication path. This guide explains what these errors mean and outlines the best steps to take for a fast resolution.

Prerequisites

• SSH access to the affected Crusoe Cloud virtual machine.

Step-by-Step Instructions

1. Identify the Error in Your Logs

   • The first step is to confirm the observable data in your dmesg logs. The messages will look similar to this, explicitly mentioning a link down or degraded state:

     NVRM: Xid (PCI:....): 149, NETIR_LINK_DOWN Fatal [...]
     NVRM: Xid (PCI:....): 154, GPU recovery action changed [...]
     NVRM: knvlinkSetDegradedMode_IMPL: GPU... marked Degraded.

2. Generate an NVIDIA Bug Report

   • To give our team the diagnostic information we need, it’s very helpful to generate a bug report from inside the affected VM. You can do this by running the following command with sudo privileges:

     sudo nvidia-bug-report.sh

   • This script will run for about a minute and will create a single, comprehensive log file in your current directory. The output will tell you the exact filename, which will look something like <timestamp>.<vm-name>.crusoe-bug-report.tar.gz.
3. Submit a Support Ticket
   • The best way to get this sorted out is to click here to open a pre-filled support ticket.
   • This link will automatically set the ticket subject, priority, and provide a template in the description. Please be sure to fill in your VM name and attach the bug report file from Step 2.

Resolution

Here’s what you can expect after you submit the ticket with the diagnostic logs:

1. Team Review: Our support team will promptly review the logs you provided to analyze the situation.
2. Infrastructure Action: Based on the data, our team will prepare a new, healthy host for your virtual machine. The underlying physical host is typically scheduled for diagnostics and maintenance to ensure platform stability.
3. Final Instructions: We will follow up with you in the support ticket to coordinate the next steps, which usually involve a simple instance restart to complete the migration to the new hardware.

Additional Resources

• How-To Capture NVIDIA Logs
• NVIDIA Documentation: Xid Errors