Last Updated: Sept 24, 2025
Introduction
The nvbandwidth tool is a key utility for measuring the bandwidth and latency of GPU-to-GPU communication, which is critical for high-performance AI workloads. When validating a new GB200 rack or troubleshooting internode performance, it's often necessary to build the tool directly from the source to ensure you are using the latest version.
This guide provides the step-by-step process for compiling the nvbandwidth utility and executing its test suite on a Crusoe Cloud GB200 instance.
Prerequisites
- SSH access to the GB200
- NVIDIA IMEX setup
- openmpi
Step-by-Step Instructions
-
Verify imex status
Verify the imex status by running the domain query commandnvidia-imex-ctl -N$ nvidia-imex-ctl -N Connectivity Table Legend: I - Invalid - Node wasn't reachable, no connection status available N - Never Connected R - Recovering - Connection was lost, but clean up has not yet been triggered. D - Disconnected - Connection was lost, and clean up has been triggreed. A - Authenticating - If GSSAPI enabled, client has initiated mutual authentication. !V! - Version mismatch, communication disabled. !M! - Node map mismatch, communication disabled. C - Connected - Ready for operation 9/17/2025 21:47:27.023 Nodes: Node #0 - 172.27.48.24 - READY - Version: 570.158.08 Node #1 - 172.27.51.33 - READY - Version: 570.158.08 Node #2 - 172.27.51.58 - READY - Version: 570.158.08 Node #3 - 172.27.52.123 - READY - Version: 570.158.08 Node #4 - 172.27.53.2 - READY - Version: 570.158.08 Node #5 - 172.27.53.4 - READY - Version: 570.158.08 Node #6 - 172.27.54.82 - READY - Version: 570.158.08 Node #7 - 172.27.56.48 - READY - Version: 570.158.08 Node #8 - 172.27.57.69 - READY - Version: 570.158.08 Node #9 - 172.27.57.214 - READY - Version: 570.158.08 Node #10 - 172.27.58.6 - READY - Version: 570.158.08 Node #11 - 172.27.58.58 - READY - Version: 570.158.08 Node #12 - 172.27.58.251 - READY - Version: 570.158.08 Node #13 - 172.27.60.152 - READY - Version: 570.158.08 Node #14 - 172.27.61.37 - READY - Version: 570.158.08 Node #15 - 172.27.61.168 - READY - Version: 570.158.08 Node #16 - 172.27.62.107 - READY - Version: 570.158.08 Node #17 - 172.27.63.240 - READY - Version: 570.158.08 Nodes From\To 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 0 C C C C C C C C C C C C C C C C C C 1 C C C C C C C C C C C C C C C C C C 2 C C C C C C C C C C C C C C C C C C 3 C C C C C C C C C C C C C C C C C C 4 C C C C C C C C C C C C C C C C C C 5 C C C C C C C C C C C C C C C C C C 6 C C C C C C C C C C C C C C C C C C 7 C C C C C C C C C C C C C C C C C C 8 C C C C C C C C C C C C C C C C C C 9 C C C C C C C C C C C C C C C C C C 10 C C C C C C C C C C C C C C C C C C 11 C C C C C C C C C C C C C C C C C C 12 C C C C C C C C C C C C C C C C C C 13 C C C C C C C C C C C C C C C C C C 14 C C C C C C C C C C C C C C C C C C 15 C C C C C C C C C C C C C C C C C C 16 C C C C C C C C C C C C C C C C C C 17 C C C C C C C C C C C C C C C C C C Domain State: UP
-
Build
nvbandwidth
After confirming connectivity between all nodes from the matrix, buildnvbandwidthon one of the GB200 nodes-
Clone the repository
$ git clone https://github.com/NVIDIA/nvbandwidth.git $ cd nvbandwidth -
Set CUDA environment
$ export CUDA_HOME=/usr/local/cuda-12.8 $ export PATH=$CUDA_HOME/bin:$PATH $ export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH -
Install required dependencies
$ sudo apt update $ sudo apt install libboost-program-options-dev cmake -
Build the binary (multinode version)
$ cmake -DMULTINODE=1 . $ make -j
-
-
Distribute binary to all nodes
- Once the binary
nvbandwidthis created, distribute to all the other nodes. - This can be achieved by attaching and mounting a shared disk on all the nodes at
/mnt/nvbandwidthand copying the binary from the build directory tomnt/nvbandwidth. - Since this is a shared disk, the copy operation will need to be performed only on one host and the binary will be available across all the nodes.
- Once the binary
-
Create hostfile
-
Create a
hostfileon the node you wish to runmpirunfrom. It should have the IP addresses of all the nodes, with each having assigned 4 slots.$ cat hostfile 172.27.48.24 slots=4 172.27.51.33 slots=4 172.27.51.58 slots=4 ... .. . 172.27.62.107 slots=4 172.27.63.240 slots=4
-
-
Run nvbandwidth using MPI
-
Run
nvbandwidthon the tray/node you have thehostfileon$ mpirun \ -x LD_LIBRARY_PATH \ -x UCX_CUDA_IPC_ENABLE_MNNVL=y \ -x I_MPI_DEBUG=5 \ -x UCX_TLS=tcp,self \ -x MPI_MCA_coll_ucc_priority=100 \ -mca pml ucx \ -mca btl ^openib \ --allow-run-as-root \ --map-by ppr:4:node \ --bind-to core \ -np 72 \ --report-bindings \ --hostfile hostfile \ /mnt/nvbandwidth/nvbandwidth -t multinode_device_to_device_memcpy_read_ce
-
Example
A successful run with all GPUs populated and nvbandwidth doing reads should look like this:
Wed Sep 24 21:55:08 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.158.08 Driver Version: 570.158.08 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GB200 On | 00000000:B0:00.0 Off | 0 |
| N/A 36C P0 220W / 1200W | 3019MiB / 189471MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GB200 On | 00000000:C2:00.0 Off | 0 |
| N/A 35C P0 217W / 1200W | 3021MiB / 189471MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA GB200 On | 00000000:DD:00.0 Off | 0 |
| N/A 36C P0 225W / 1200W | 3018MiB / 189471MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA GB200 On | 00000000:EF:00.0 Off | 0 |
| N/A 36C P0 230W / 1200W | 3020MiB / 189471MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 50546 C /home/ubuntu/nvbandwidth 744MiB |
| 0 N/A N/A 50547 C /home/ubuntu/nvbandwidth 744MiB |
| 0 N/A N/A 50548 C /home/ubuntu/nvbandwidth 744MiB |
| 0 N/A N/A 50550 C /home/ubuntu/nvbandwidth 744MiB |
| 1 N/A N/A 50546 C /home/ubuntu/nvbandwidth 744MiB |
| 1 N/A N/A 50547 C /home/ubuntu/nvbandwidth 744MiB |
| 1 N/A N/A 50548 C /home/ubuntu/nvbandwidth 744MiB |
| 1 N/A N/A 50550 C /home/ubuntu/nvbandwidth 744MiB |
| 2 N/A N/A 50546 C /home/ubuntu/nvbandwidth 744MiB |
| 2 N/A N/A 50547 C /home/ubuntu/nvbandwidth 744MiB |
| 2 N/A N/A 50548 C /home/ubuntu/nvbandwidth 744MiB |
| 2 N/A N/A 50550 C /home/ubuntu/nvbandwidth 744MiB |
| 3 N/A N/A 50546 C /home/ubuntu/nvbandwidth 744MiB |
| 3 N/A N/A 50547 C /home/ubuntu/nvbandwidth 744MiB |
| 3 N/A N/A 50548 C /home/ubuntu/nvbandwidth 744MiB |
| 3 N/A N/A 50550 C /home/ubuntu/nvbandwidth 744MiB |
+-----------------------------------------------------------------------------------------+Additional Resources