Last Updated: Oct 28, 2025
Introduction
If you've run the command nvidia-smi and it appears your instance is missing a GPU, this typically indicates an underlying hardware issue, such as a dropped PCIe link, a failing GPU, or a driver-related problem. These issues can result in degraded performance or, in some cases, cause operations to hang. This article outlines the troubleshooting steps to follow when your instance appears to be missing a GPU.
Prerequisites
- Crusoe Cloud account
- Access to the Crusoe Cloud UI or Crusoe CLI
Step-by-Step Instructions
1. As a first step, it is recommended to immediately take a backup of any important data you don't wish to lose as the instance can become unresponsive.
2. Then, run the following command to clear any ECC errors that could be affecting the instance followed by a reboot.
$ nvidia-smi -r
$ reboot now
3. If you're still seeing a missing GPU in the instance, generate the Nvidia bug report:
$ sudo nvidia-bug-report.sh- Refer to this article for further assistance on pulling this report: How-To Capture NVIDIA Logs
- For CMK, follow How-To Capture NVIDIA Bug Report in Kubernetes
- Transfer this file to your host machine using tools such as scp or rsync. This will be needed when creating a ticket with Crusoe Support
4. The next step would be to perform a VM reset or a STOP / START operation.
- Please replace
<vm-name>with the name of your VM. - Note: A VM reset will preserve any ephemeral data stored on your instance, however, it is best practice to backup any important data in case the operation hangs. A STOP operation on the VM will however delete all ephemeral data stored in the instance, so a backup is always recommended
- Note: If the underlying host is experiencing any hardware related issues, the following operations may hang resulting in the instance becoming inaccessible.
$ crusoe compute vms reset <vm-name>$ crusoe compute vms stop <vm-name>$ crusoe compute vms start <vm-name>
5. If the above steps do not resolve the issue, please reach out to Crusoe Support and provide the bug report.
Additional Resources:
FAQ
1. Why is my instance missing a GPU?
Answer: This event is logged when the GPU driver attempts to access the GPU but cannot detect it due to a hardware or driver issue
2. Why was I not informed by Crusoe that the GPU is missing?
Answer: In most cases, if an instance is experiencing a hardware-related issue, Crusoe’s monitoring and alerting systems will detect it, and a Cloud Support Engineer will proactively reach out to notify you. However, some issues may only be observable from within the virtual machine (VM) itself. Since Crusoe does not have visibility inside the VM, they are not automatically alerted to these types of scenarios.