How-To Validate Infiniband Performance with NCCL All Reduce Test

Last Updated: Jan 8, 2026

Introduction

Crusoe Cloud VMs are equipped with high-performance NVIDIA Mellanox InfiniBand (IB) networking. To help you get started, our curated VM images come pre-loaded with all the necessary software and tools to use InfiniBand right away with popular ML and HPC frameworks. While you're welcome to use a custom image, we strongly recommend starting with ubuntu22.04-nvidia-sxm-docker:latest for the quickest and easiest setup. Learn more about Crusoe Infiniband in our docs. 

Prerequisites

• Access to a Crusoe Cloud project with appropriate permissions
• 2 or more Infiniband supported VMs

Step-by-Step Instructions

You can utilize NVIDIA's Collective Communications Library (NCCL) to validate the performance of the IB networking stack once at least 2 Infiniband supported instance types are launched within an IB partition. 

1. Setup SSH connectivity between the two VMs

   • SSH into one of the VMs and issue the following command. Accept the defaults as it.

     $ ssh-keygen -t ed25519 

   • Copy the contents of ./ssh/ed25519.pub file and add it to the .ssh/authorized_keys file on both instances.
2. Clone & compile upstream NCCL tests Git Repo

   • Git clone the following repo on both the instances

     $ git clone https://github.com/NVIDIA/nccl-tests.git

   • Compile the binaries by running the following command

     $ cd nccl-tests/ && make MPI=1 CUDA_HOME=/usr/local/cuda
     make -C src build BUILDDIR=/home/ubuntu/nccl-tests/build
     make[1]: Entering directory '/home/ubuntu/nccl-tests/src'
     Compiling  timer.cc                             /home/ubuntu/nccl-tests/build/timer.o
     Compiling /home/ubuntu/nccl-tests/build/verifiable/verifiable.o
     Compiling  all_reduce.cu                        /home/ubuntu/nccl-tests/build/all_reduce.o
     Compiling  common.cu                            /home/ubuntu/nccl-tests/build/common.o
     Linking  /home/ubuntu/nccl-tests/build/all_reduce.o  /home/ubuntu/nccl-tests/build/all_reduce_perf
     Compiling  all_gather.cu                        /home/ubuntu/nccl-tests/build/all_gather.o
     Linking  /home/ubuntu/nccl-tests/build/all_gather.o  /home/ubuntu/nccl-tests/build/all_gather_perf
     Compiling  broadcast.cu                         /home/ubuntu/nccl-tests/build/broadcast.o
     Linking  /home/ubuntu/nccl-tests/build/broadcast.o  /home/ubuntu/nccl-tests/build/broadcast_perf
     Compiling  reduce_scatter.cu                    /home/ubuntu/nccl-tests/build/reduce_scatter.o
     Linking  /home/ubuntu/nccl-tests/build/reduce_scatter.o  /home/ubuntu/nccl-tests/build/reduce_scatter_perf
     Compiling  reduce.cu                            /home/ubuntu/nccl-tests/build/reduce.o
     Linking  /home/ubuntu/nccl-tests/build/reduce.o  /home/ubuntu/nccl-tests/build/reduce_perf
     Compiling  alltoall.cu                          /home/ubuntu/nccl-tests/build/alltoall.o
     Linking  /home/ubuntu/nccl-tests/build/alltoall.o  /home/ubuntu/nccl-tests/build/alltoall_perf
     Compiling  scatter.cu                           /home/ubuntu/nccl-tests/build/scatter.o
     Linking  /home/ubuntu/nccl-tests/build/scatter.o  /home/ubuntu/nccl-tests/build/scatter_perf
     Compiling  gather.cu                            /home/ubuntu/nccl-tests/build/gather.o
     Linking  /home/ubuntu/nccl-tests/build/gather.o  /home/ubuntu/nccl-tests/build/gather_perf
     Compiling  sendrecv.cu                          /home/ubuntu/nccl-tests/build/sendrecv.o
     Linking  /home/ubuntu/nccl-tests/build/sendrecv.o  /home/ubuntu/nccl-tests/build/sendrecv_perf
     Compiling  hypercube.cu                         /home/ubuntu/nccl-tests/build/hypercube.o
     Linking  /home/ubuntu/nccl-tests/build/hypercube.o  /home/ubuntu/nccl-tests/build/hypercube_perf
     make[1]: Leaving directory '/home/ubuntu/nccl-tests/src'

3. Create hostfile 

   • On one of the instances create a file named hostfile in the home directory, referencing the private IP of each instance and slots=8 which represents that each instance supports 8 ranks (8 GPUs per instance). As an example hostfile below:

     172.27.21.150 slots=8
     172.27.22.114 slots=8

4. Update NCCL configuration file

   • Ensure your /etc/nccl.conf file contains the following config: 

     NCCL_IB_MERGE_VFS=0
     NCCL_IB_HCA=^mlx5_0:1
     NCCL_SOCKET_NTHREADS=4
     NCCL_NSOCKS_PERTHREAD=8
     NCCL_TOPO_FILE=/etc/crusoe/nccl_topo/h100-80gb-sxm-ib-cloud-hypervisor.xml

     If you are using HPCx versions older than 2.18, you will need to include 

     NCCL_IBEXT_DISABLE=1

     Note: Crusoe VMs created with curated images include the NCCL config by default. For Custom images, please follow the instructions here. 

5. Run NCCL script

   • Run the following script from one of the VMs to begin the test:

     #!/bin/bash
     
     . /opt/hpcx/hpcx-init.sh
     hpcx_load
     
     mpirun -np 16 -N 8 -hostfile ~/hostfile --bind-to none -mca btl tcp,self -mca coll_hcoll_enable 0 -x PATH -x LD_LIBRARY_PATH -x NCCL_DEBUG=INFO /opt/nccl-tests/build/all_reduce_perf -b 8M -e 2048M -f 2 -t 1 -g 1 -c 1 -n 10

     Note: Ensure that the NCCL_VERSION is consistent across all instances. One of the ways to check is by running dpkg -l | grep nccl . To achieve, version alignment across instances may require upgrading or downgrading the CUDA version to match the desired NCCL version on each instance.

6. Validate Bus bandwidth  

   • #
     #                                                              out-of-place                       in-place
     #       size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong
     #        (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)
          8388608       2097152     float     sum      -1   203.91   41.14   77.13       0   186.48   44.98   84.35       0
         16777216       4194304     float     sum      -1   238.44   70.36  131.93       0   183.67   91.35  171.27       0
         33554432       8388608     float     sum      -1   264.48  126.87  237.88       0   263.04  127.57  239.19       0
         67108864      16777216     float     sum      -1   479.19  140.05  262.59       0   474.80  141.34  265.01       0
        134217728      33554432     float     sum      -1   747.72  179.50  336.57       0   744.14  180.37  338.19       0
        268435456      67108864     float     sum      -1  1287.79  208.45  390.84       0  1286.95  208.58  391.09       0
        536870912     134217728     float     sum      -1  2359.73  227.51  426.59       0  2355.40  227.93  427.37       0
       1073741824     268435456     float     sum      -1  4471.08  240.15  450.29       0  4474.95  239.94  449.90       0
       2147483648     536870912     float     sum      -1  8723.90  246.16  461.55       0  8726.38  246.09  461.42       0
     # Out of bounds values : 0 OK
     # Avg bus bandwidth    : 311.286
     #
     # Collective test concluded: all_reduce_perf

     In the above example, the peak busbw observed was 461.55 GB/s between two H100 VMs.

Additional Resources

• NVIDIA's Collective Communications Library (NCCL)
• NVIDIA NCCL tests Git Repo
• Crusoe Documentation - IB Networking
• Crusoe NCCL Topology Files