NVRM: Xid (PCI:0003:00:04): 13, pid='<unknown>', name=<unknown>, Graphics SM Warp Exception on (GPC 0, TPC 7, SM 1): Out Of Range Address

1. Stop and Start the VM to see if the issue gets resolved.
2. Debug the application using cuda-gdb or the Compute Sanitizer memcheck tool.
3. Run the application with CUDA_DEVICE_WAITS_ON_EXCEPTION=1 and then attach later with cuda-gdb .
4. Run the application again. If you're still noticing XID 13 errors, generate an NVIDIA bug report.
5. Reach out to Crusoe Support and provide the bug report.

XID

48

- dmesg logs Error Message:

NVRM: Xid (PCI:0003:00:03): 48, pid='<unknown>', name=<unknown>, An uncorrectable double bit error (DBE) has been detected on GPU in the L2 cache at cache 0, slice 2.

According to NVIDIA's documentation: "This event is logged when the GPU detects that an uncorrectable error occurs on the GPU. This is also reported back to the user application. A GPU reset or node reboot is needed to clear this error."

1. Perform a GPU reset using the following command: # nvidia-smi -r
2. If the issue persists, perform a VM reset using the following command: # crusoe compute vms reset <vm-name>
3. If the issue persists, STOP and Start the VM 
4. If the issue persists after following the above steps, generate an NVIDIA bug report
5. Reach out to Crusoe Support and provide the bug report

- dmesg logs Error Message:

NVRM: Xid (PCI:0000:14:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.

1. Stop and Start the VM to see if the issue gets resolved.
2. Generate an NVIDIA bug report
3. Reach out to Crusoe Support and provide the bug report.
4. If you have spare host availability, proceed to stop and start the VM.
5. If you do not have any spare capacity, please proceed to shut down the instance for maintenance and let us know in the support ticket.

XID

95

dmesg logs Error Message:

NVRM: Xid (PCI:0002:00:04): 95, pid='<unknown>', name=<unknown>, Uncontained: FBHUB. RST: Yes, D-RST: No

1. Perform a GPU reset using the following command: # nvidia-smi -r
2. If the issue persists, perform a VM reset using the following command: # crusoe compute vms reset <vm-name>
3. If the issue persists, STOP and Start the VM 
4. If the issue persists after following the above steps, generate an NVIDIA bug report
5. Reach out to Crusoe Support and provide the bug report

XID 119

- dmesg logs Error Message:

NVRM: Xid (PCI:0003:00:04): 119, pid=2009566, name=nvidia-smi, Timeout after 6s of waiting for RPC response from GPU7 GSP! Expected function 76 (GSP_RM_CONTROL) (0x2080014b 0x5).

1. Generate an NVIDIA bug report

2. Create the file /etc/modprobe.d/nvidia.conf

3. Add options nvidia NVreg_EnableGpuFirmware=0 to the file

4.  Update the kernel images update-initramfs -u -k all