How-To Verify GPU and InfiniBand Device Details Connected on Your Instance

Introduction

This article explains how to verify GPU and InfiniBand-related device details on your instance. Because GPUs and networking devices are passed through to the virtual machine (VM) rather than exposed as full physical devices, some traditional tools may not behave as expected.

In particular, utilities such as ibstatus may report errors even when InfiniBand-capable hardware is present and functioning correctly. This guide outlines the supported and reliable commands you can use to confirm that GPUs, Mellanox devices, and their associated drivers are properly attached and recognized by the operating system.

Prerequisites

• Crusoe Cloud account
• Access to the Crusoe Cloud UI or Crusoe CLI
• A GPU enabled instance running on Crusoe Cloud 
  • For more information on what Crusoe offers: Virtual Machines Overview

Step-by-Step Instructions

1. To get started, connect to your GPU compute instance using SSH, or open a terminal session via JupyterLab, depending on how your environment is configured.

2. Lets understand why running ibstatus may result in an error:

   $ ibstatus
   Fatal error:  No devices

   1. This behavior is expected. Since the instance is a virtual machine with PCIe device passthrough, InfiniBand devices are exposed as virtual functions, not as full host adapters. As a result, tools that expect direct access to physical InfiniBand hardware (such as ibstatus) cannot detect the devices.

3. Instead, we can confirm that Mellanox networking devices are attached to the instance by running lspci | grep -i mellanox. Here is a sample output: 

   $ lspci | grep -i mellanox
   0000:00:07.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
   0002:00:09.0 Infiniband controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
   0002:00:0a.0 Infiniband controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
   0002:00:0b.0 Infiniband controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
   0002:00:0c.0 Infiniband controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
   0003:00:09.0 Infiniband controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
   0003:00:0a.0 Infiniband controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
   0003:00:0b.0 Infiniband controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
   0003:00:0c.0 Infiniband controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function

   1. This confirms that Mellanox PCIe devices are present and visible to the operating system.

4. We can also verify that the appropriate kernel drivers are loaded by running lsmod | grep mlx. Here is a sample output:

   $ lsmod | grep mlx
   mlx5_core            1593344  0
   mlxfw                  32768  1 mlx5_core
   pci_hyperv_intf        16384  1 mlx5_core
   tls                   114688  1 mlx5_core
   psample                20480  1 mlx5_core

   1. The presence of mlx5_core and related modules indicates that the Mellanox drivers are loaded and active.

5. To confirm that GPUs are available and recognized, run nvidia-smi:

   $ nvidia-smi
   +-----------------------------------------------------------------------------------------+
   | NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
   |-----------------------------------------+------------------------+----------------------+
   | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
   |                                         |                        |               MIG M. |
   |=========================================+========================+======================|
   |   0  NVIDIA H100 80GB HBM3          On  |   00000002:00:01.0 Off |                    0 |
   | N/A   31C    P0             68W /  700W |       1MiB /  81559MiB |      0%      Default |
   |                                         |                        |             Disabled |
   +-----------------------------------------+------------------------+----------------------+
   |   1  NVIDIA H100 80GB HBM3          On  |   00000002:00:02.0 Off |                    0 |
   | N/A   25C    P0             79W /  700W |       1MiB /  81559MiB |      0%      Default |
   |                                         |                        |             Disabled |
   +-----------------------------------------+------------------------+----------------------+
   |   2  NVIDIA H100 80GB HBM3          On  |   00000002:00:03.0 Off |                    0 |
   | N/A   26C    P0             82W /  700W |       1MiB /  81559MiB |      0%      Default |
   |                                         |                        |             Disabled |
   +-----------------------------------------+------------------------+----------------------+
   |   3  NVIDIA H100 80GB HBM3          On  |   00000002:00:04.0 Off |                    0 |
   | N/A   30C    P0             69W /  700W |       1MiB /  81559MiB |      0%      Default |
   |                                         |                        |             Disabled |
   +-----------------------------------------+------------------------+----------------------+
   |   4  NVIDIA H100 80GB HBM3          On  |   00000003:00:01.0 Off |                    0 |
   | N/A   30C    P0             79W /  700W |       1MiB /  81559MiB |      0%      Default |
   |                                         |                        |             Disabled |
   +-----------------------------------------+------------------------+----------------------+
   |   5  NVIDIA H100 80GB HBM3          On  |   00000003:00:02.0 Off |                    0 |
   | N/A   26C    P0             77W /  700W |       1MiB /  81559MiB |      0%      Default |
   |                                         |                        |             Disabled |
   +-----------------------------------------+------------------------+----------------------+
   |   6  NVIDIA H100 80GB HBM3          On  |   00000003:00:03.0 Off |                    0 |
   | N/A   25C    P0             71W /  700W |       1MiB /  81559MiB |      0%      Default |
   |                                         |                        |             Disabled |
   +-----------------------------------------+------------------------+----------------------+
   |   7  NVIDIA H100 80GB HBM3          On  |   00000003:00:04.0 Off |                    0 |
   | N/A   31C    P0             78W /  700W |       1MiB /  81559MiB |      0%      Default |
   |                                         |                        |             Disabled |
   +-----------------------------------------+------------------------+----------------------+
   
   +-----------------------------------------------------------------------------------------+
   | Processes:                                                                              |
   |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
   |        ID   ID                                                               Usage      |
   |=========================================================================================|
   |  No running processes found                                                             |
   +-----------------------------------------------------------------------------------------+

   1. This command provides a high-level overview, including:
      1. GPU model(s)
      2. Driver version
      3. CUDA version
      4. Current GPU utilization
      5. Memory usage
6. For more detailed information, run: nvidia-smi -q
   1. This command displays extended details such as:
      1. PCIe information
      2. GPU clocks and power usage
      3. ECC status
      4. Temperature and thermal limits
      5. Per-GPU configuration and capabilities

Additional Resources:

• NVIDIA System Management Interface (nvidia-smi) Documentation

• Mellanox / NVIDIA Networking Documentation

• Linux PCI Utilities (lspci) Reference