How-To Run nvidia-smi Commands on CMK

Last Updated: Dec 10, 2025

Introduction

Nvidia  nvidia-smi toolkit is a command line utility that can be used to manage, monitor, and debug Nvidia GPUs.

Among other things, it can be used for retrieving Nvidia GPUs general information and status, drivers and CUDA versions, and ECC errors.

More information on all the options available for nvidia-smi can be found in the References below.

On a CMK cluster with nvidia-gpu-operator add-on installed, it is possible to run nvidia-smi by executing into the nvidia-gpu-driver-ubuntu<version>-<id> pods.

For additional Nvidia GPU debugging on CMK, refer to How-To Capture NVIDIA Bug Report in CMK.

Prerequisites

• CMK cluster deployed with nvidia-gpu-operator add-on - Manage your CMK Clusters 

Step-by-Step Instructions

1. Find the nvidia-gpu-driver-ubuntu<version>-<id> pod running on the affected node

   • # kubectl get pods -n nvidia-gpu-operator -o wide | grep <node_name> | grep nvidia-gpu-driver-ubuntu

2. Run the nvidia-smi command within the pod 

   • # kubectl -n nvidia-gpu-operator exec -it nvidia-gpu-driver-ubuntu<version>-<id> -- nvidia-smi

Example 

To get GPU general information and to find the Nvidia drivers and CUDA versions:

# kubectl -n nvidia-gpu-operator exec -it nvidia-gpu-driver-ubuntu22.04-6865f88d94-l95x5 -- nvidia-smi
Thu Oct  2 17:57:29 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20             Driver Version: 570.133.20     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000002:00:01.0 Off |                    0 |
| N/A   54C    P0            370W /  700W |   72031MiB /  81559MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

To retrieve ECC errors:

# kubectl -n nvidia-gpu-operator exec -it nvidia-gpu-driver-ubuntu22.04-6865f88d94-l95x5 -- nvidia-smi -q -d ECC

==============NVSMI LOG==============

Timestamp                                 : Thu Oct  2 18:12:28 2025
Driver Version                            : 570.133.20
CUDA Version                              : 12.8

Attached GPUs                             : 8
GPU 00000002:00:01.0
    ECC Mode
        Current                           : Enabled
        Pending                           : Enabled
    ECC Errors
        Volatile
            SRAM Correctable              : 0
            SRAM Uncorrectable Parity     : 0
            SRAM Uncorrectable SEC-DED    : 0
            DRAM Correctable              : 0
            DRAM Uncorrectable            : 0
        Aggregate
            SRAM Correctable              : 0
            SRAM Uncorrectable Parity     : 0
            SRAM Uncorrectable SEC-DED    : 0
            DRAM Correctable              : 0
            DRAM Uncorrectable            : 0
            SRAM Threshold Exceeded       : No
        Aggregate Uncorrectable SRAM Sources
            SRAM L2                       : 0
            SRAM SM                       : 0
            SRAM Microcontroller          : 0
            SRAM PCIE                     : 0
            SRAM Other                    : 0

References

• Manage your CMK Clusters
• nvidia-smi Docs
• How-To Capture NVIDIA Bug Report in CMK