Crusoe Cloud provides a default VM image that comes with all the software libraries and tools necessary to take advantage of Infiniband through supported ML and HPC frameworks. While you are not required to use this image to use InfiniBand networking, we strongly recommend the `ubuntu20.04-nvidia-sxm-docker:latest` or `ubuntu22.04-nvidia-sxm-docker:latest` image for the easiest possible setup. Learn more about Crusoe Infiniband in our docs. 

NVIDIA Collective Communications Library

You can utilize NVIDIA's Collective Communications Library (NCCL) to validate the performance of the IB networking stack once at least 2 Infiniband supported instance types are launched within an IB partition. To do so, please follow these steps:

1. Setup passwordless SSH between the two instances by first SSH connecting into one of the instances and issuing a ssh-keygen -t ed25519. Accept the defaults and then copy the contents of the ./ssh/ed25519.pub to both instances' .ssh/authorized_keys file.

2. On each instance git clone the NVIDIA/nccl-tests repo. And compile the binaries according to the repo's instructions. On one of the instances create a hostfile referencing the private IP of each instance and slots=8 which represents that each instance supports 8 ranks (8 GPUs per instance). As an example hostfile below:

172.27.21.150 slots=8
172.27.22.114 slots=8

3. Ensure your `/etc/nccl.conf` file contains the following configs: 

NCCL_IB_MERGE_VFS=0
NCCL_IB_HCA=^mlx5_0:1
NCCL_SOCKET_NTHREADS=4
NCCL_NSOCKS_PERTHREAD=8
NCCL_TOPO_FILE=/etc/crusoe/nccl_topo/h100-80gb-sxm-ib-cloud-hypervisor.xml

Note: If you are using HPCx versions older than 2.18, you will need to include 

NCCL_IBEXT_DISABLE=1

4. Run the following script from one of the VMs to begin the test:

#!/bin/bash


. /opt/hpcx/hpcx-init.sh
hpcx_load

mpirun -np 16 -N 8 -x NCCL_DEBUG=INFO -hostfile hostfile \
--bind-to none -mca btl tcp,self -mca coll_hcoll_enable 0 \
-x NCCL_IB_AR_THRESHOLD=0 -x NCCL_IB_PCI_RELAXED_ORDERING=1 \
-x NCCL_IB_SPLIT_DATA_ON_QPS=0 -x NCCL_IB_QPS_PER_CONNECTION=2 -x CUDA_DEVICE_ORDER=PCI_BUS_ID \
-x PATH -x LD_LIBRARY_PATH \
/home/ubuntu/nccl-tests/build/all_reduce_perf -b 8 -e 2G -f 2 -t 1 -g 1 -c 1 -n 100

5. Validate the Bus bandwidth has the expected GB/s performance. 

# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
8 2 float sum -1 31.73 0.00 0.00 0 31.70 0.00 0.00 0
16 4 float sum -1 31.58 0.00 0.00 0 31.57 0.00 0.00 0
32 8 float sum -1 31.74 0.00 0.00 0 31.66 0.00 0.00 0
64 16 float sum -1 30.27 0.00 0.00 0 30.20 0.00 0.00 0
128 32 float sum -1 30.53 0.00 0.01 0 30.45 0.00 0.01 0
256 64 float sum -1 40.39 0.01 0.01 0 30.70 0.01 0.02 0
512 128 float sum -1 32.01 0.02 0.03 0 31.41 0.02 0.03 0
1024 256 float sum -1 32.66 0.03 0.06 0 32.40 0.03 0.06 0
2048 512 float sum -1 33.82 0.06 0.11 0 33.48 0.06 0.11 0
4096 1024 float sum -1 35.14 0.12 0.22 0 34.90 0.12 0.22 0
8192 2048 float sum -1 36.06 0.23 0.43 0 35.70 0.23 0.43 0
16384 4096 float sum -1 38.19 0.43 0.80 0 37.51 0.44 0.82 0
32768 8192 float sum -1 39.62 0.83 1.55 0 38.80 0.84 1.58 0
65536 16384 float sum -1 41.42 1.58 2.97 0 40.81 1.61 3.01 0
131072 32768 float sum -1 45.61 2.87 5.39 0 45.04 2.91 5.46 0
262144 65536 float sum -1 53.40 4.91 9.20 0 53.37 4.91 9.21 0
524288 131072 float sum -1 60.98 8.60 16.12 0 60.53 8.66 16.24 0
1048576 262144 float sum -1 73.81 14.21 26.64 0 72.83 14.40 27.00 0
2097152 524288 float sum -1 103.1 20.34 38.13 0 102.7 20.42 38.28 0
4194304 1048576 float sum -1 123.0 34.11 63.96 0 119.5 35.09 65.80 0
8388608 2097152 float sum -1 137.3 61.10 114.56 0 123.4 67.98 127.45 0
16777216 4194304 float sum -1 195.6 85.78 160.83 0 192.1 87.35 163.78 0
33554432 8388608 float sum -1 544.4 61.64 115.58 0 419.7 79.95 149.90 0
67108864 16777216 float sum -1 468.6 143.22 268.53 0 469.1 143.06 268.23 0
134217728 33554432 float sum -1 779.9 172.09 322.67 0 776.6 172.83 324.05 0
268435456 67108864 float sum -1 1448.4 185.33 347.49 0 1447.9 185.40 347.63 0
536870912 134217728 float sum -1 2881.9 186.29 349.29 0 2879.8 186.43 349.56 0
1073741824 268435456 float sum -1 5674.6 189.22 354.78 0 5676.4 189.16 354.67 0
2147483648 536870912 float sum -1 11152 192.56 361.06 0 11127 193.00 361.87 0