How-To Setup GB200 NVL72 Rack on CMK Cluster and Run NCCL Performance Validation

Last Updated: Jan 23, 2026

Introduction

This article provides guidance on setting up GB200 NVL72 rack(s) on a Crusoe Managed Kubernetes (CMK) Cluster with managed configurations on Dynamic Resource Allocation (DRA) and GPU Operator, and running a NCCL Performance Validation Test.

Note: As of this update (Jan 23, 2026), CMK creation for GB200 racks are limited to Console UI and Crusoe CLI. This Knowledge Base will be revised with updated instructions when other dev tools are enabled (i.e. Terraform).

Prerequisites

• Access to a Crusoe Cloud project with appropriate permissions
• Ensure your org/project have GB200 enabled through customer service
• Create InfiniBand Partition in the Network where the rack is (Optional, if your rack(s) is on an InfiniBand fabric)

Step-by-Step Instructions

1. Create a CMK Cluster
   • Choose Orchestration -> Create a Cluster
   • Select eu-iceland1-a for location, as currently it is the only location that supports GB200 racks.
   • On the Add-ons section, select the following: NVIDIA GPU Operator, NVIDIA Network Operator (only if you are adding multiple IB-connected racks in the cluster) and Support for NVIDIA GB200. Toggling the Support for NVIDIA GB200 will indicate this cluster will be GB200-specific.
   • Select Container Storage Interface (CSI) & Cluster Autoscaler addon if required.
     
2. Create a Node Pool
    Note: 1) Only GB200 Node Pools can be defined in a GB200 CMK Cluster at this time. 2) If you are creating multiple racks, you must create a Node Pool per rack.
   • Select Create Node Pool once the cluster is provisioned
   • Select GB200 as Instance Type: GB200-186GB-NVL for non-IB, GB200-186GB-NVL-IB for IB-connected racks
   • (If using IB-connected racks) Select InfiniBand Network and Partition.
   • Select NVLink Domain: This indicates the instances are interconnected by NVLink fabric inside a full rack. 
     
     
3. Download Kubeconfig to gain cluster access.
4. Verify GPU add-ons, including NVIDIA GPU Operator and NVIDIA DRA Driver

kubectl get pod -n nvidia-gpu-operator -l app=gpu-operator
NAME                           READY   STATUS    RESTARTS   AGE
gpu-operator-b6d589d4c-kttjz   1/1     Running   0          2d12h


kubectl get pod -n nvidia-dra-driver-gpu -l nvidia-dra-driver-gpu-component=controller
NAME                                                READY   STATUS    RESTARTS   AGE
nvidia-dra-driver-gpu-controller-5b859bbb58-n8h68   1/1     Running   0          2d12h

NCCL Validation on CMK GB200 Cluster

NCCL Validation on CMK GB200 Cluster requires a couple of additional components. Note: this is an example testing a single rack, and variables need to be adjusted for testing across multiple GB200 NVL72 clusters.

1. Install MPI Operator: The MPI Operator makes it easy to run allreduce-style distributed training on Kubernetes. Use kubectl to install the operator on the cluster.

   kubectl apply --server-side -f https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.7.0/deploy/v2beta1/mpi-operator.yaml

   Verify install:

   kubectl get crd | grep mpijobs
   
   mpijobs.kubeflow.org                                     2025-11-15T08:54:40Z

2. MPIJob: Deploy the Compute Domain and the MPIJob that defines the NCCL all reduce test. Note: Starting with DRA driver 25.8.1 (which CMK supports), Compute Domains will auto-detect node count and therefore can be set to zero as shown below.

   apiVersion: resource.nvidia.com/v1beta1
   kind: ComputeDomain
   metadata:
     name: nccl-compute-domain
   spec:
     numNodes: 0
     channel:
       resourceClaimTemplate:
         name: nccl-compute-domain-channel
   ---
   apiVersion: kubeflow.org/v2beta1
   kind: MPIJob
   metadata:
     name: nccl-tests-gb200
   spec:
     slotsPerWorker: 4
     runPolicy:
       cleanPodPolicy: Running
     mpiReplicaSpecs:
       Launcher:
         replicas: 1
         template:
           spec:
             restartPolicy: OnFailure
             initContainers:
             - image: ghcr.io/coreweave/nccl-tests:13.0.1-devel-ubuntu22.04-nccl2.28.7-1-6b47463
               imagePullPolicy: IfNotPresent
               name: init
               command: ["sh", "-c", "sleep 5"]
             containers:
             - image: ghcr.io/coreweave/nccl-tests:13.0.1-devel-ubuntu22.04-nccl2.28.7-1-6b47463
               imagePullPolicy: IfNotPresent
               name: nccl-test-launcher
               securityContext:
                 capabilities:
                   add: ["IPC_LOCK"]
               command:
               - mpirun
               - --allow-run-as-root
               - --tag-output
               - -np
               - "72" # Total number of processes 
               - -N
               - "4"
               - -bind-to
               - none
               - -map-by
               - ppr:4:node
               - --mca
               - coll
               - ^hcoll
               - -x
               - UCX_TLS=self,tcp
               - -x
               - NCCL_IB_QPS_PER_CONNECTION=2
               - -x
               - PATH
               - -x
               - LD_LIBRARY_PATH
               - -x
               - NCCL_DEBUG=TRACE
               - /opt/nccl-tests/build/all_reduce_perf
               - -b
               - "2G"
               - -e
               - "32G"
               - -f
               - "2"
               - -g
               - "1"
   
       Worker:
         replicas: 18 # Specify how many worker nodes you have running in the Instances tab
         template:
           spec:
             restartPolicy: OnFailure
             volumes:
             - name: dshm
               emptyDir:
                 medium: Memory
                 sizeLimit: 64Gi
             - name: nvidia-caps
               hostPath:
                 path: /dev/nvidia-caps
                 type: Directory
             resourceClaims:
             - name: compute-domain-channel
               resourceClaimTemplateName: nccl-compute-domain-channel
             containers:
             - image: ghcr.io/coreweave/nccl-tests:13.0.1-devel-ubuntu22.04-nccl2.28.7-1-6b47463
               imagePullPolicy: IfNotPresent
               name: nccl-worker
               securityContext:
                 capabilities:
                   add: ["IPC_LOCK"]
               env:
                - name: NCCL_DEBUG
                  value: TRACE
                - name: UCX_RNDV_SCHEME
                  value: "get_zcopy"   # UCX memory setting
                - name: UCX_TLS
                  value: "self,sm,cuda_copy"   # UCX memory setting
               volumeMounts:
               - mountPath: /dev/shm
                 name: dshm
               - mountPath: /dev/nvidia-caps
                 name: nvidia-caps
               resources:
                 limits:
                   nvidia.com/gpu: 4
                   memory: 800Gi
                 requests:
                   nvidia.com/gpu: 4
                   memory: 400Gi
                 claims:
                   - name: compute-domain-channel

3. Results

   # Somewhere in the output
   [1,0]<stdout>:#
   [1,0]<stdout>:#                                                              out-of-place                       in-place          
   [1,0]<stdout>:#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
   [1,0]<stdout>:#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
   [1,0]<stdout>:  2147483648     536870912     float     sum      -1   5253.7  408.76  806.17      0   5300.1  405.18  799.10      0
   [1,0]<stdout>:  4294967296    1073741824     float     sum      -1    10020  428.64  845.38      0    10133  423.85  835.93      0
   [1,0]<stdout>:  8589934592    2147483648     float     sum      -1    19297  445.15  877.93      0    19309  444.86  877.37      0
   [1,0]<stdout>: 17179869184    4294967296     float     sum      -1    37153  462.40  911.96      0    37114  462.89  912.93      0
   [1,0]<stdout>: 34359738368    8589934592     float     sum      -1    73031  470.48  927.89      0    73052  470.34  927.62      0
   [1,0]<stdout>:# Out of bounds values : 0 OK
   [1,0]<stdout>:# Avg bus bandwidth    : 872.229 
   [1,0]<stdout>:#
   [1,0]<stdout>:# Collective test concluded: all_reduce_perf

Additional Resources

• NVIDIA DRA Driver for GPUs
• How-To Isolate and Schedule GPU Resources for Different AI Workloads Using Compute Domains and DRA on GB200