How-To Setup GB200 NVL72 Rack on RKE2 Cluster and Run NCCL Performance Validation

Last Updated: Nov 5, 2025

Introduction

This article provides guidance on setting up GB200 NVL72 rack on RKE2 Cluster with Dynamic Resource Allocation configuration and running NCCL Performance Validation.

Prerequisites

• Access to a Crusoe Cloud project with appropriate permissions
• Terraform installed (version 1.0+ recommended)
• Crusoe Terraform Provider (version >= 0.5.27) 

Step-by-Step Instructions

1. Clone repository Crusoe-ML-RKE2.

$ git clone git@github.com:crusoecloud/crusoe-ml-rke2.git

2. Checkout to branch for gb200-support

$ git checkout gb200-support

Note: that this branch has been verified for Kubernetes v1.32 and v1.33.

3. Parameters to be set in terraform.tfvars 

• Example tfvars for using all 18 workers in the rack with 2 c1a.8x headnode.

ssh_privkey_path = <...>
ssh_pubkey = <...>
worker_instance_type = "gb200-186gb-nvl-ib.4x"
worker_image = "ubuntu24.04-nvidia-nvl-arm64-gb200:latest"
worker_count = 18
ib_partition_id = <...>
headnode_count = 2
headnode_instance_type = "c1a.8x"
deploy_location = "eu-iceland1-a"
enable_dra_feature = true
rke_version = "v1.33.5+rke2r1"

• Modify/set parameters in above as required.

4. Run terraform apply to create the RKE2 cluster.

5. Retrieve and Configure the Kubeconfig for Cluster Access

• Access the cluster by copying the kubeconfig file from the headnode. Replace the 'server' address in the Kubeconfig with that of your load balancer (or control plane node when deploying single control plane node configurations).

rke_endpoint=$(terraform output -raw rke-ingress-instance_public_ip)
headnode_endpoint=$(terraform output -raw rke-headnode-instance_public_ip)
scp -i $path_to_priv_key "root@${headnode_endpoint}:/etc/rancher/rke2/rke2.yaml" ./kubeconfig
sed -i '' "s/127.0.0.1/${rke_endpoint}/g" ./kubeconfig
sed -i '' "s/default/crusoe/g" ./kubeconfig
export KUBECONFIG="$(pwd)/kubeconfig"

6. Add and Update Nvidia Helm repo, then install the GPU Operator.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update

helm install gpu-operator nvidia/gpu-operator \
--version="v25.3.2" \
    --create-namespace \
    --namespace gpu-operator \
    --set cdi.enabled=true \
    --set driver.enabled=false \
    --set 'toolkit.env[0].name=CONTAINERD_CONFIG' \
    --set 'toolkit.env[0].value=/var/lib/rancher/rke2/agent/etc/containerd/config.toml' \
    --set 'toolkit.env[1].name=CONTAINERD_SOCKET' \
    --set 'toolkit.env[1].value=/run/k3s/containerd/containerd.sock' 

7. Install the DRA Driver for GPUs.

• NVIDIA DRA Driver is a Kubernetes extension that enables dynamic, fine-grained GPU resource allocation and scheduling using the Device Resource Assignment (DRA) framework.

helm install nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \
    --version="25.8.0" \
    --create-namespace \
    --namespace nvidia-dra-driver-gpu \
    --set nvidiaDriverRoot=/ \
    --set resources.gpus.enabled=false

8. Verify GPUs are in the same communication group (clique) and can be scheduled together by DRA-aware workloads. 

$ (echo -e "NODE\tLABEL\tCLIQUE"; kubectl get nodes -o json | \
    jq -r '.items[] | [.metadata.name, "nvidia.com/gpu.clique", .metadata.labels["nvidia.com/gpu.clique"]] | @tsv') | \
    column -t

9. DeviceClasses created by the DRA driver confirms DRA is enabled and advertising GPUs.

$ kubectl get deviceclasses
NAME                                        AGE
compute-domain-daemon.nvidia.com            5m38s
compute-domain-default-channel.nvidia.com   5m38s

10. Run a simple test to validate IMEX daemons are started and IMEX channels are injected.

• Here, IMEX (NVIDIA Internode Memory Exchange) is needed to enforce fine-grained access control over GPU memory sharing, preventing unauthorized read/write between GPUs in the same NVLink partition. Setting up the NVIDIA DRA driver automatically enables IMEX, so this access control is applied without additional configuration.
• ComputeDomains leverage IMEX to isolate jobs across namespaces, ensuring MNNVL-reachability between pods within the domain, and automatically form around scheduled pods, tearing down all resources when the workload finishes.

$ cat <<EOF > imex-channel-injection.yaml
---
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: imex-channel-injection
spec:
  numNodes: 1
  channel:
    resourceClaimTemplate:
      name: imex-channel-0
---
apiVersion: v1
kind: Pod
metadata:
  name: imex-channel-injection
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nvidia.com/gpu.clique
            operator: Exists
  containers:
  - name: ctr
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["ls -la /dev/nvidia-caps-imex-channels; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: imex-channel-0
  resourceClaims:
  - name: imex-channel-0
    resourceClaimTemplateName: imex-channel-0
EOF

• Verification of IMEX channels being injected.

$ kubectl apply -f imex-channel-injection.yaml

kubectl logs imex-channel-injection
total 0
drwxr-xr-x 2 root root     60 Sep 12 19:11 .
drwxr-xr-x 6 root root    380 Sep 12 19:11 ..
crw-rw-rw- 1 root root 234, 0 Sep 12 19:11 channel0

• The channel0 device file confirms that an IMEX channel has been successfully injected. This channel is how the GPU memory access permissions are enforced: any memory export/import operations now pass through this controlled interface, ensuring that only authorized GPUs or jobs can read/write memory.

$ kubectl delete -f imex-channel-injection.yaml 

11. Install the latest version of the MPI Operator & deploy MPIJob for NCCL Validation for full rack GB200 NVL72 rack (18 nodes = 72 GPUs).

• Here we will use nvbandwidth tool for bandwidth measurements on NVIDIA GPUs. It helps measure bandwidth for various memcpy patterns across different links using copy engine or kernel copy methods and help report measured bandwidth on the system. Since nvbandwidth requires MPI, below we also install the Kubeflow MPI Operator.

$ kubectl create -f https://github.com/kubeflow/mpi-operator/releases/download/v0.6.0/mpi-operator.yaml

• YAML to create ComputeDomain and MPIJob.

apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: nvbandwidth-test-compute-domain
spec:
  numNodes: 18
  channel:
    resourceClaimTemplate:
      name: nvbandwidth-test-compute-domain-channel

---
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nccl-tests-gb200
spec:
  slotsPerWorker: 4
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          restartPolicy: OnFailure
          initContainers:
          - image: ghcr.io/coreweave/nccl-tests:12.9.1-devel-ubuntu22.04-nccl2.28.3-1-8b67957
            imagePullPolicy: IfNotPresent
            name: init
            command: ["sh", "-c", "sleep 5"]
          containers:
          - image: ghcr.io/coreweave/nccl-tests:12.9.1-devel-ubuntu22.04-nccl2.28.3-1-8b67957
            imagePullPolicy: IfNotPresent
            name: nccl-test-launcher
            securityContext:
              capabilities:
                add: ["IPC_LOCK"]
            env:
             - name: UCX_RNDV_SCHEME
               value: "get_zcopy"   # UCX memory setting
             - name: UCX_TLS
               value: "self,sm,cuda_copy"   # UCX memory setting
            command:
            - mpirun
            - --allow-run-as-root
            - --tag-output
            - -np
            - "72"    # total GPUs = 18 nodes × 4 GPUs each
            - -N
            - "4"     # 4 processes per node (one per GPU)
            - -bind-to
            - none
            - -map-by
            - slot
            - -mca
            - coll_hcoll_enable
            - "0"
            - -x
            - NCCL_IB_PCI_RELAXED_ORDERING=1
            - -x
            - NCCL_IB_SPLIT_DATA_ON_QPS=0
            - -x
            - NCCL_IB_QPS_PER_CONNECTION=2
            - -x
            - NCCL_IB_MERGE_VFS=0
            - -x
            - NCCL_IB_HCA=ibp
            - -x
            - SHARP_COLL_ENABLE_PCI_RELAXED_ORDERING=1
            - -x
            - PATH
            - -x
            - LD_LIBRARY_PATH
            - -x
            - NCCL_DEBUG=trace
            - /opt/nccl-tests/build/all_reduce_perf
            - -b
            - "2G"
            - -e
            - "32G"
            - -f
            - "2"
            - -t
            - "1"
            - -g
            - "1"
            - -c
            - "1"
            - -n
            - "100"
    Worker:
      replicas: 18 # Specify how many worker nodes you have running in the Instances tab
      template:
        spec:
          restartPolicy: OnFailure
          runtimeClassName: nvidia
          volumes:
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 64Gi
          - name: nvidia-caps
            hostPath:
              path: /dev/nvidia-caps
              type: Directory
          resourceClaims:
          - name: compute-domain-channel
            resourceClaimTemplateName: nvbandwidth-test-compute-domain-channel
          containers:
          - image: ghcr.io/coreweave/nccl-tests:12.9.1-devel-ubuntu22.04-nccl2.28.3-1-8b67957
            imagePullPolicy: IfNotPresent
            name: nccl-worker
            securityContext:
              capabilities:
                add: ["IPC_LOCK"]
            env:
             - name: NCCL_DEBUG
               value: TRACE
             - name: UCX_RNDV_SCHEME
               value: "get_zcopy"   # UCX memory setting
             - name: UCX_TLS
               value: "self,sm,cuda_copy"   # UCX memory setting
            volumeMounts:
            - mountPath: /dev/shm
              name: dshm
            - mountPath: /dev/nvidia-caps
              name: nvidia-caps
            resources:
              limits:
                nvidia.com/gpu: 4
              requests:
                nvidia.com/gpu: 4
              claims:
                - name: compute-domain-channel

• Results

[1,0]<stdout>:  2147483648     536870912     float     sum      -1   5375.1  399.52  787.95      0   5384.9  398.79  786.51      0
[1,0]<stdout>:  4294967296    1073741824     float     sum      -1    10193  421.37  831.03      0    10205  420.87  830.05      0
[1,0]<stdout>:  8589934592    2147483648     float     sum      -1    19555  439.28  866.36      0    19598  438.31  864.45      0
[1,0]<stdout>: 17179869184    4294967296     float     sum      -1    38056  451.43  890.32      0    38088  451.06  889.59      0
[1,0]<stdout>: 34359738368    8589934592     float     sum      -1    74718  459.86  906.94      0    74772  459.53  906.29      0

[1,0]<stdout>:# Out of bounds values : 0 OK
[1,0]<stdout>:# Avg bus bandwidth    : 855.949

Additional Resources

• NVIDIA DRA Driver for GPUs
• K8s - Dynamic Resource Allocation
• nvBandwidth