How-To Validate DeepEP GPU Communication Performance on Crusoe Managed Kubernetes (H200 & B200)

Introduction

DeepEP is an open-source communication library from DeepSeek that accelerates Expert Parallelism (EP) in Mixture-of-Experts (MoE) models. It provides optimized all-to-all communication kernels that dispatch tokens to expert GPUs and combine results, using NVLink for intra-node traffic and InfiniBand RDMA for inter-node traffic.

This guide walks through deploying and running the DeepEP benchmark suite on a Crusoe Managed Kubernetes (CMK) cluster with H200 SXM IB or B200 SXM IB nodes. By the end you will have validated both the intra-node NVLink path and the inter-node InfiniBand path, and confirmed that the cluster is delivering bandwidth in the expected range for production MoE training workloads.

The benchmark exercises the two communication paths that dominate MoE training: NVLink within a node, and RDMA across nodes via IBGDA (InfiniBand GPUDirect Async). The IBGDA path is the most demanding to set up because it requires the GPU to write RDMA work requests directly to the InfiniBand HCA — and that in turn requires specific NVIDIA driver kernel parameters, GPUDirect RDMA kernel modules, and per-rank NVSHMEM configuration to be in place.

The procedure is identical for H200 and B200 except for the SKU-specific parameters summarized below and referenced throughout the guide. Wherever a step uses one of these values, it is called out explicitly.

This guide was validated end to end with the following software stack. The B200 configuration has been validated in two Crusoe regions plus a third independent cluster, and the H200 configuration in two independent environments — including one configured from scratch (Steps 1–4 onward) and benchmarked with the pre-built Step 6 image and this article's manifest, end to end. Component versions outside this table are unverified, and some are known not to work: NVSHMEM releases newer than 3.4.5 (3.6.5+/3.7.x, which newer NGC bases such as 26.x require) changed the RC queue-pair structures the IBGDA path depends on and have been observed to fail inter-node on this stack even when everything else is held constant. If you deviate from these versions and see failures or results outside the reference ranges, reproduce with the validated stack before treating it as a cluster issue (see Step 9).

ℹ️ Note: All reference ranges in Step 8 were measured with this stack. The B200 figures consolidate three runs (4-node and 2-node, across independent clusters); the H200 figures consolidate two 2-node runs. Both SKUs have been validated with the pre-built Step 6 images.

⚠️ Warning: This guide is for CMK clusters only. If you are running on a bare-metal Crusoe Cloud VM, use How-To Validate DeepEP GPU Communication Performance on Crusoe Cloud instead — and conversely, do not follow that article on CMK: its driver configuration steps (host-level module loading, device node creation) do not apply on CMK, where the NVIDIA driver is managed by the GPU Operator. Attempting them inside pods leads to inter-node dispatch timeouts that no amount of NVSHMEM environment tuning can fix. On CMK, driver configuration must go through Steps 1–4 below.

Prerequisites

• CMK Cluster with an H200 SXM IB or B200 SXM IB Node Pool
• Two GPU Worker Nodes Minimum (Inter-Node Test)
• NVIDIA GPU Operator Installed in nvidia-gpu-operator Namespace
• NVIDIA Network Operator Add-On Enabled at Cluster Creation
• kubectl Access with Permissions to Patch the NVIDIADriver CR and Manage Driver Pods
• A Container Registry the Cluster Can Pull From (Pull Secret Required for Authenticated Registries)

Instructions

The workflow has three phases: configure the NVIDIA driver for IBGDA (Steps 1–4, done once per cluster), confirm the node's HCA layout and topology file (Step 5), and get the image and run the benchmark (Steps 6–8). Step 9 covers troubleshooting. All manifests are shown with B200 values; H200 customers swap six values in the Job manifest (Step 7) — the image tag and five SKU-specific settings — with exact replacements provided at each point.

Step 1: Configure NVIDIA Driver Kernel Parameters

DeepEP's IBGDA transport requires two NVIDIA driver registry parameters:

• NVreg_EnableStreamMemOPs=1 — enables CUDA stream-ordered memory operations required for GPU-initiated RDMA work submission.
• NVreg_RegistryDwords="PeerMappingOverride=1" — enables peer memory mappings between the GPU and the InfiniBand HCA's user access region.

Create a ConfigMap containing these parameters. The GPU Operator's kernelModuleConfig reference will pick them up and apply them when the driver loads.

cat > nvidia-kernel-params.conf << 'CONF'
NVreg_EnableStreamMemOPs=1
NVreg_RegistryDwords="PeerMappingOverride=1"
CONF

kubectl -n nvidia-gpu-operator create configmap nvidia-kernel-params \
  --from-file=nvidia.conf=./nvidia-kernel-params.conf

Verify the ConfigMap was created with the expected content:

kubectl -n nvidia-gpu-operator get configmap nvidia-kernel-params -o yaml

💡 Tip: If your cluster already has a ConfigMap referenced by the existing NVIDIADriver CR under a different name, reuse that name rather than creating a duplicate. Inspect the existing CR with kubectl get nvidiadriver -o yaml | grep -A2 kernelModuleConfig.

Step 2: Patch the NVIDIADriver Custom Resource

First, identify the CR name for your GPU node pool. The name matches the GPU SKU associated with the pool (h200, b200, etc.) — the CR whose STATUS shows ready is the one backing your active nodes:

kubectl get nvidiadriver

Apply a single merge patch that enables GPUDirect RDMA, enables GDRCopy, and links the kernel parameter ConfigMap. Replace <YOUR_NVIDIADRIVER_CR> with the name returned above.

kubectl patch nvidiadriver <YOUR_NVIDIADRIVER_CR> --type='merge' -p='{
  "spec": {
    "gdrcopy": {
      "enabled": true,
      "image": "gdrdrv",
      "repository": "nvcr.io/nvidia/cloud-native",
      "version": "v2.5.1"
    },
    "kernelModuleConfig": {
      "name": "nvidia-kernel-params"
    },
    "rdma": {
      "enabled": true
    }
  }
}'

⚠️ Warning: Apply all three settings (gdrcopy, kernelModuleConfig, rdma) in a single merge. Partial application leaves the driver in an inconsistent state where IBGDA initialization fails at runtime.

Step 3: Apply the New Driver Configuration

The NVIDIADriver CR uses an OnDelete update strategy. Existing driver pods continue running with their previous configuration until they are restarted, so the patch from Step 2 will not take effect on running nodes until the driver pods are recycled.

⚠️ Warning: Recycle driver pods only when no GPU workloads are running on the affected nodes. The driver pod's init process cordons each node and drains GPU pods before reloading the driver — a running GPU workload (including platform burn-in or qualification tests on newly provisioned nodes) blocks the drain and leaves the driver pod stuck in Init:CrashLoopBackOff until the workload finishes.

Restart the driver pods across the cluster:

kubectl -n nvidia-gpu-operator delete pod -l app.kubernetes.io/component=nvidia-driver

Wait for the new pods to come up. Each pod contains three containers (the main driver, the GDRCopy module loader, and the peermem module loader), and all three must reach Ready state. This takes 60–90 seconds.

kubectl -n nvidia-gpu-operator get pods -l app.kubernetes.io/component=nvidia-driver -w

Cancel the watch (Ctrl+C) when every pod shows 3/3 Ready.

ℹ️ Note: The container count is the tell that the Step 2 patch took effect, because the patch is what adds the two module-loader containers (GDRCopy and peermem) to the driver pod. Correctly reconfigured pods report 0/3 from the moment they are created and climb to 3/3. If the new pods come up as 0/1 → 1/1, the driver is still running its previous single-container configuration — most commonly because the pods were recycled before the CR patch was applied. Verify the CR spec with kubectl get nvidiadriver <YOUR_NVIDIADRIVER_CR> -o yaml and recycle the pods again after patching.

ℹ️ Note: Nodes that join the cluster after the patch (new nodes, scaled or recreated pools) start their driver pods with the new configuration automatically — the manual recycle is only required for driver pods that were already running when the patch was applied.

Step 4: Verify Driver State on Each Node

Run the following check to confirm the kernel parameters are applied and the required kernel modules are loaded on every GPU node.

for pod in $(kubectl -n nvidia-gpu-operator get pods -l app.kubernetes.io/component=nvidia-driver -o name); do
  node=$(kubectl -n nvidia-gpu-operator get $pod -o jsonpath='{.spec.nodeName}')
  ready=$(kubectl -n nvidia-gpu-operator get $pod -o jsonpath='{.status.containerStatuses[*].ready}' | tr ' ' '/')
  params=$(kubectl -n nvidia-gpu-operator exec $pod -c nvidia-driver-ctr -- \
    cat /proc/driver/nvidia/params 2>/dev/null | grep -cE 'EnableStreamMemOPs: 1|RegistryDwords: "PeerMappingOverride')
  mods=$(kubectl -n nvidia-gpu-operator exec $pod -c nvidia-driver-ctr -- \
    lsmod 2>/dev/null | grep -cE '^(nvidia_peermem|gdrdrv) ')
  printf "%-55s ready=%s params=%d/2 modules=%d/2\n" "$node" "$ready" "$params" "$mods"
done

A correctly configured node prints:

<your-node-name>                                  ready=true/true/true params=2/2 modules=2/2

If any node shows fewer than 2/2 modules on a Ready pod, restart that node's driver pod:

kubectl -n nvidia-gpu-operator delete pod <pod-name-on-affected-node>

💡 Tip: Wait at least 60 seconds after a pod restart before re-running the check. The module loader containers need time to insert the modules after pod start. Re-run this verification any time nodes are added or replaced, before benchmarking.

Step 5: Confirm the Backend HCA Mapping and Topology File

The benchmark pins each rank to a dedicated backend InfiniBand HCA, and the device numbering differs between GPU SKUs:

• H200 SXM IB: mlx5_0 is the frontend NIC; mlx5_1–mlx5_8 are the eight rail-optimized backend NICs. Mapping: GPU N → mlx5_(N+1).
• B200 SXM IB: mlx5_4 is the frontend NIC (NODE affinity, attached to the first NUMA domain only); mlx5_5–mlx5_12 are the eight backend NICs, four per NUMA domain, serving GPUs 0–3 and 4–7 respectively. Mapping: GPU N → mlx5_(N+5).

Using the wrong offset pins ranks to the frontend NIC or to nonexistent devices, and the failure is silent — the benchmark runs, but at a fraction of expected inter-node bandwidth.

Confirm the layout on your nodes before running the benchmark. The driver pods run in the host namespace, so you can check directly from one without a separate debug pod:

POD=$(kubectl -n nvidia-gpu-operator get pods -l app.kubernetes.io/component=nvidia-driver -o name | head -1)
kubectl -n nvidia-gpu-operator exec $POD -c nvidia-driver-ctr -- nvidia-smi topo -m
kubectl -n nvidia-gpu-operator exec $POD -c nvidia-driver-ctr -- ls /sys/class/infiniband/

ℹ️ Note: ibdev2netdev is not present in the driver container. The topology matrix plus the /sys/class/infiniband/ listing provide the same information.

From the output, confirm which mlx5_X devices are the backend rails (the frontend NIC shows a different affinity class, NODE, than the rails, PHB), and that the offset for your SKU matches the table above. If it differs, use the actual offset and device names in the Job manifest in Step 7.

The Job manifest also points NCCL at the per-SKU topology file that Crusoe ships on every GPU node under /etc/crusoe/nccl_topo/. Confirm the file for your SKU exists — the driver container does not mount the host's /etc, so use a node debug pod:

kubectl debug node/<your-gpu-node-name> -it --image=busybox:1.36 -- ls /host/etc/crusoe/nccl_topo/

Expect the filename from the SKU table (e.g. b200-180gb-sxm-ib-cloud-hypervisor.xml). Delete the debugger pod afterwards.

Step 6: Get the Benchmark Container Image

Crusoe publishes pre-built community benchmark images matching this guide's validated stack at ghcr.io/crusoecloud/deepep-tests. Pull the tag for your SKU — each is compiled for that GPU architecture only, so the tags are not interchangeable:

• B200: ghcr.io/crusoecloud/deepep-tests:b200-torch29-nvshmem345
• H200: ghcr.io/crusoecloud/deepep-tests:h200-torch29-nvshmem345

The tag goes on the image: line of the Step 7 manifest (the manifest ships with the B200 tag pre-filled). The registry is public, so no pull secret is needed. See the repository at github.com/crusoecloud/deepep-tests for image details and its support disclaimer.

Create the benchmark namespace:

kubectl create namespace deepep-test

Building your own image instead: to integrate this stack into your own base images or host it in your own registry, use the reference Dockerfile in the github.com/crusoecloud/deepep-tests repository — it is the recipe the pre-built images follow (NVIDIA's public NGC PyTorch CUDA 13 base image, DeepEP, NVSHMEM, and the GDRCopy userspace library; all sources are public). Three of its details are load-bearing:

• DeepEP branch: the Dockerfile clones the hybrid-ep branch — the NVSHMEM-backend line this guide validates. DeepEP's main is now V2, which moved its primary transport to an NCCL-based backend (requiring NCCL 2.28+ with NCCL-GIN support and the DOCA GPUNetIO device SDK, none present in this stack) and fails to build here with Cannot find package: nccl. V2 may retain a legacy compatibility mode, but it has not been validated on Crusoe; contact Crusoe support for its current status.
• NVSHMEM version pin: nvidia-nvshmem-cu13==3.4.5 must match the NVSHMEM integration bundled with the base image's PyTorch — a mismatched wheel fails at import with undefined symbol: nvshmem_selected_device_transport. Independently, NVSHMEM 3.6.5+/3.7.x breaks the inter-node path at runtime even when it loads cleanly (see the stack-compatibility symptom in Step 9). Stay on 3.4.5 for this guide.
• The sm_100 TMA patch (B200): DeepEP's setup.py gates its aggressive TMA PTX instructions to architecture 9.0 (Hopper); the reference Dockerfile patches the gate out and sets DISABLE_AGGRESSIVE_PTX_INSTRS=0 so B200 (sm_100) builds get the fast path — without it, B200 intra-node throughput roughly halves. TORCH_CUDA_ARCH_LIST is the one SKU-specific line: 9.0 for H200, 10.0 for B200; the patch is a no-op on a 9.0 build.

If your own image requires pull authentication, create a docker-registry pull secret in the namespace and reference it from the Job in Step 7 by adding an imagePullSecrets entry to the pod spec:

kubectl -n deepep-test create secret docker-registry deepep-pull \
  --docker-server=<your-registry-host> \
  --docker-username='<registry-username>' \
  --docker-password='<registry-password-or-token>'

💡 Tip: If using Crusoe Container Registry, note that repository URIs follow the format registry.<region>.ccr.crusoecloudcompute.com/<repository-name>.<first-8-of-project-id>/<image-name>:<tag>, and authentication uses your account email with a registry token as the password. See the Crusoe Container Registry documentation for details.

Step 7: Deploy the Benchmark Job

The benchmark script is embedded in the Job manifest, so no separate ConfigMap is needed. The script runs the intra-node (NVLink) benchmark first, then the inter-node (RDMA + IBGDA) benchmark, and prints per-node and aggregate bandwidth.

The SKU-specific HCA offset is supplied by the NVSHMEM_HCA_OFFSET environment variable; the script refuses to start without it, which prevents the silent wrong-NIC failure mode. NVSHMEM_HCA_LIST itself is set inside the script per rank — each rank initializes its own NVSHMEM context, so this value cannot be a pod-level environment variable. Apply the same per-rank pattern in any application code that constructs deep_ep.Buffer() on CMK.

Save the following as deepep-bench.yaml. Values to set:

1. image: — your image URI from Step 6.
2. image — pre-filled with the B200 tag from Step 6; for H200 use the H200 tag (see the Example callout after the manifest). For a self-built image in a private registry, set your image URI and add an imagePullSecrets entry for the deepep-pull secret per the Step 6 note.
3. NVSHMEM_HCA_OFFSET — "5" for B200 as shown; "1" for H200.
4. NCCL_IB_HCA / UCX_NET_DEVICES — the backend rails from Step 5: shown for B200 (mlx5_5–mlx5_12); for H200 use mlx5_1:1,...,mlx5_8:1.
5. NCCL_TOPO_FILE — shown for B200; for H200 use the H200 filename from the SKU table.
6. resources — set just below node allocatable (kubectl describe node <node> | grep -A8 Allocatable); the values shown are validated for typical node shapes (H200 uses memory: "1700Gi").
7. To run across more than two nodes, change completions, parallelism, --nnodes, NUM_NODES, and WORLD_SIZE (= nodes × 8) together.

apiVersion: v1
kind: Service
metadata:
  name: deepep-bench
  namespace: deepep-test
  labels:
    app: deepep-bench
spec:
  clusterIP: None
  selector:
    app: deepep-bench
  publishNotReadyAddresses: true
  ports:
  - name: torchrun
    port: 29400
    targetPort: 29400
    protocol: TCP
---
apiVersion: batch/v1
kind: Job
metadata:
  name: deepep-bench
  namespace: deepep-test
  labels:
    app: deepep-bench
spec:
  backoffLimit: 0
  completions: 2
  parallelism: 2
  ttlSecondsAfterFinished: 3600
  completionMode: Indexed
  template:
    metadata:
      labels:
        app: deepep-bench
    spec:
      subdomain: deepep-bench
      restartPolicy: Never
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nvidia.com/gpu.present
                operator: In
                values:
                - "true"
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: deepep-bench
            topologyKey: kubernetes.io/hostname
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 64Gi
      - name: crusoe-config
        hostPath:
          path: /etc/crusoe
          type: Directory
      containers:
      - name: deepep-bench
        image: ghcr.io/crusoecloud/deepep-tests:b200-torch29-nvshmem345
        imagePullPolicy: Always
        command: ["/bin/bash", "-c"]
        args:
        - |
          set -euo pipefail
          echo "=== DeepEP Benchmark ==="
          echo "Node rank: $NODE_RANK / $NUM_NODES nodes"
          export NCCL_DEBUG=WARN
          export PYTHONPATH=/tmp:${PYTHONPATH:-}
          MASTER_ADDR="deepep-bench-0.deepep-bench.deepep-test.svc.cluster.local"
          cat > /tmp/deepep_benchmark.py << 'PYTHON_SCRIPT'
          import os
          import time
          import traceback
          import importlib
          import torch
          import torch.distributed as dist
          def benchmark_deepep_internode(group=None):
              deep_ep = importlib.import_module('deep_ep')
              deep_ep.Buffer.set_num_sms(20)
              rank = dist.get_rank()
              world_size = dist.get_world_size() if group is None else dist.get_world_size(group)
              num_tokens = 8192
              hidden_dim = 3072
              num_experts = 128
              topk = 8
              warmup_iters = 10
              bench_iters = 50
              num_nvl_bytes = 1024 * 1024 * 1024
              num_rdma_bytes = 512 * 1024 * 1024
              if rank == 0:
                  print(f'  Buffer: group_size={world_size}, nvl={num_nvl_bytes}, rdma={num_rdma_bytes}', flush=True)
              buffer = deep_ep.Buffer(
                  group=group,
                  num_nvl_bytes=num_nvl_bytes,
                  num_rdma_bytes=num_rdma_bytes,
                  low_latency_mode=False,
                  num_qps_per_rank=24,
              )
              x = torch.randn(num_tokens, hidden_dim, device='cuda', dtype=torch.bfloat16)
              topk_idx = torch.randint(0, num_experts, (num_tokens, topk), device='cuda', dtype=torch.int64)
              topk_weights = torch.rand(num_tokens, topk, device='cuda', dtype=torch.float32)
              for i in range(warmup_iters):
                  if rank == 0:
                      print(f'  warmup {i+1}/{warmup_iters}', flush=True)
                  try:
                      num_tokens_per_rank, num_tokens_per_rdma_rank, num_tokens_per_expert, is_token_in_rank, _ = \
                          buffer.get_dispatch_layout(topk_idx, num_experts)
                      recv_x, _, recv_topk_weights, _, handle, _ = buffer.dispatch(
                          x, None, num_tokens_per_rank, num_tokens_per_rdma_rank,
                          is_token_in_rank, num_tokens_per_expert, topk_idx, topk_weights,
                      )
                      if recv_x is not None and recv_x.size(0) > 0:
                          combine_x = torch.randn_like(recv_x)
                          _, _, _ = buffer.combine(combine_x, handle, recv_topk_weights)
                      torch.cuda.synchronize()
                  except Exception as e:
                      print(f'  [rank {rank}] warmup {i} FAILED: {e}', flush=True)
                      traceback.print_exc()
                      raise
              if rank == 0:
                  print(f'  benchmarking {bench_iters} iters...', flush=True)
              layout_start = torch.cuda.Event(enable_timing=True)
              layout_end = torch.cuda.Event(enable_timing=True)
              dispatch_start = torch.cuda.Event(enable_timing=True)
              dispatch_end = torch.cuda.Event(enable_timing=True)
              combine_start = torch.cuda.Event(enable_timing=True)
              combine_end = torch.cuda.Event(enable_timing=True)
              total_layout_ms = 0.0
              total_dispatch_ms = 0.0
              total_combine_ms = 0.0
              total_rdma_tokens = 0
              for it in range(bench_iters):
                  layout_start.record()
                  num_tokens_per_rank, num_tokens_per_rdma_rank, num_tokens_per_expert, is_token_in_rank, _ = \
                      buffer.get_dispatch_layout(topk_idx, num_experts)
                  layout_end.record()
                  layout_end.synchronize()
                  total_layout_ms += layout_start.elapsed_time(layout_end)
                  if num_tokens_per_rdma_rank is not None:
                      total_rdma_tokens += int(num_tokens_per_rdma_rank.sum().item())
                  dispatch_start.record()
                  recv_x, _, recv_topk_weights, _, handle, _ = buffer.dispatch(
                      x, None, num_tokens_per_rank, num_tokens_per_rdma_rank,
                      is_token_in_rank, num_tokens_per_expert, topk_idx, topk_weights,
                  )
                  dispatch_end.record()
                  dispatch_end.synchronize()
                  total_dispatch_ms += dispatch_start.elapsed_time(dispatch_end)
                  if recv_x is not None and recv_x.size(0) > 0:
                      combine_x = torch.randn_like(recv_x)
                      combine_start.record()
                      _, _, _ = buffer.combine(combine_x, handle, recv_topk_weights)
                      combine_end.record()
                      combine_end.synchronize()
                      total_combine_ms += combine_start.elapsed_time(combine_end)
              bytes_per_token = hidden_dim * 2
              total_rdma_bytes = total_rdma_tokens * bytes_per_token * 2
              total_ms = total_layout_ms + total_dispatch_ms + total_combine_ms
              dispatch_bw = (total_rdma_tokens * bytes_per_token) / (total_dispatch_ms / 1000) / 1e9 if total_dispatch_ms > 0 else 0
              combine_bw = (total_rdma_tokens * bytes_per_token) / (total_combine_ms / 1000) / 1e9 if total_combine_ms > 0 else 0
              total_bw = total_rdma_bytes / (total_ms / 1000) / 1e9 if total_ms > 0 else 0
              return {
                  'dispatch_bw_gbps': dispatch_bw,
                  'combine_bw_gbps': combine_bw,
                  'total_bw_gbps': total_bw,
                  'layout_ms': total_layout_ms,
                  'dispatch_ms': total_dispatch_ms,
                  'combine_ms': total_combine_ms,
              }
          def benchmark_deepep_intranode(node_group, num_sms=40):
              deep_ep = importlib.import_module('deep_ep')
              deep_ep.Buffer.set_num_sms(num_sms)
              num_tokens = 65536
              hidden_dim = 3072
              num_experts = 128
              topk = 16
              num_channels = num_sms // 2
              num_ranks_local = dist.get_world_size(node_group)
              nvl_bytes = max(
                  256 * 1024 * 1024,
                  num_channels * num_ranks_local * 6 * hidden_dim * 2 * 4
              )
              buffer = deep_ep.Buffer(
                  group=node_group,
                  num_nvl_bytes=nvl_bytes,
                  num_rdma_bytes=0,
                  low_latency_mode=False,
              )
              x = torch.randn(num_tokens, hidden_dim, device='cuda', dtype=torch.bfloat16)
              topk_idx = torch.randint(0, num_experts, (num_tokens, topk), device='cuda', dtype=torch.int64)
              topk_weights = torch.rand(num_tokens, topk, device='cuda', dtype=torch.float32)
              local_rank = dist.get_rank(node_group)
              for _ in range(10):
                  num_tokens_per_rank, _, num_tokens_per_expert, is_token_in_rank, _ = \
                      buffer.get_dispatch_layout(topk_idx, num_experts)
                  recv_x, _, recv_topk_weights, _, handle, _ = buffer.dispatch(
                      x, None, num_tokens_per_rank, None, is_token_in_rank,
                      num_tokens_per_expert, topk_idx, topk_weights,
                  )
                  if recv_x is not None and recv_x.size(0) > 0:
                      combine_x = torch.randn_like(recv_x)
                      _, _, _ = buffer.combine(combine_x, handle, recv_topk_weights)
                  torch.cuda.synchronize()
              dispatch_start = torch.cuda.Event(enable_timing=True)
              dispatch_end = torch.cuda.Event(enable_timing=True)
              combine_start = torch.cuda.Event(enable_timing=True)
              combine_end = torch.cuda.Event(enable_timing=True)
              total_dispatch_ms = 0.0
              total_combine_ms = 0.0
              total_nvl_tokens = 0
              bench_iters = 50
              for _ in range(bench_iters):
                  num_tokens_per_rank, _, num_tokens_per_expert, is_token_in_rank, _ = \
                      buffer.get_dispatch_layout(topk_idx, num_experts)
                  tokens_total = int(num_tokens_per_rank.sum().item())
                  tokens_self = int(num_tokens_per_rank[local_rank].item())
                  total_nvl_tokens += tokens_total - tokens_self
                  dispatch_start.record()
                  recv_x, _, recv_topk_weights, _, handle, _ = buffer.dispatch(
                      x, None, num_tokens_per_rank, None, is_token_in_rank,
                      num_tokens_per_expert, topk_idx, topk_weights,
                  )
                  dispatch_end.record()
                  dispatch_end.synchronize()
                  total_dispatch_ms += dispatch_start.elapsed_time(dispatch_end)
                  if recv_x is not None and recv_x.size(0) > 0:
                      combine_x = torch.randn_like(recv_x)
                      combine_start.record()
                      _, _, _ = buffer.combine(combine_x, handle, recv_topk_weights)
                      combine_end.record()
                      combine_end.synchronize()
                      total_combine_ms += combine_start.elapsed_time(combine_end)
              bytes_per_token = hidden_dim * 2
              total_nvl_bytes = total_nvl_tokens * bytes_per_token * 2
              total_ms = total_dispatch_ms + total_combine_ms
              dispatch_bw = (total_nvl_tokens * bytes_per_token) / (total_dispatch_ms / 1000) / 1e9 if total_dispatch_ms > 0 else 0
              combine_bw = (total_nvl_tokens * bytes_per_token) / (total_combine_ms / 1000) / 1e9 if total_combine_ms > 0 else 0
              total_bw = total_nvl_bytes / (total_ms / 1000) / 1e9 if total_ms > 0 else 0
              return {
                  'dispatch_bw_gbps': dispatch_bw,
                  'combine_bw_gbps': combine_bw,
                  'total_bw_gbps': total_bw,
                  'dispatch_ms': total_dispatch_ms,
                  'combine_ms': total_combine_ms,
              }
          def main():
              local_rank = int(os.environ.get('LOCAL_RANK', '0'))
              torch.cuda.set_device(local_rank)
              hca_offset = os.environ.get('NVSHMEM_HCA_OFFSET')
              assert hca_offset is not None, \
                  'NVSHMEM_HCA_OFFSET must be set in the Job manifest (H200: "1", B200: "5")'
              os.environ["NVSHMEM_HCA_LIST"] = f"mlx5_{local_rank + int(hca_offset)}:1"
              dist.init_process_group(backend='nccl', device_id=torch.device(f'cuda:{local_rank}'))
              rank = dist.get_rank()
              world_size = dist.get_world_size()
              local_world_size = int(os.environ.get('LOCAL_WORLD_SIZE', '8'))
              num_nodes = world_size // local_world_size
              node_id = rank // local_world_size
              if rank == 0:
                  print('=' * 70)
                  print(f'DeepEP Benchmark: {num_nodes} nodes, {world_size} GPUs')
                  print(f'Driver: {torch.cuda.get_device_properties(0).name}')
                  print(f'NVSHMEM_HCA_LIST mapping: mlx5_(local_rank+{hca_offset}):1 per rank')
                  try:
                      with open('/proc/driver/nvidia/version') as f:
                          print(f'NVIDIA: {f.readline().strip()}')
                  except: pass
                  print(f'PyTorch: {torch.__version__}, CUDA: {torch.version.cuda}, NCCL: {torch.cuda.nccl.version()}')
                  print('=' * 70)
              per_node_groups = []
              for n in range(num_nodes):
                  ranks = list(range(n * local_world_size, (n + 1) * local_world_size))
                  g = dist.new_group(ranks=ranks)
                  per_node_groups.append(g)
              my_node_group = per_node_groups[node_id]
              if rank == 0:
                  print('\n--- Intra-node DeepEP (NVLink) ---')
              dist.barrier()
              try:
                  intra_results = benchmark_deepep_intranode(my_node_group)
                  all_intra = [None] * world_size
                  dist.all_gather_object(all_intra, intra_results)
                  if rank == 0:
                      for n in range(num_nodes):
                          r = all_intra[n * local_world_size]
                          print(f'  Node {n}: dispatch={r["dispatch_bw_gbps"]:.1f} GB/s, combine={r["combine_bw_gbps"]:.1f} GB/s, total={r["total_bw_gbps"]:.1f} GB/s')
              except Exception as e:
                  print(f'  [rank {rank}] INTRA ERROR: {e}', flush=True)
                  traceback.print_exc()
              dist.barrier()
              if rank == 0:
                  print('\n--- Inter-node DeepEP (RDMA + IBGDA) ---')
              dist.barrier()
              try:
                  world_group = dist.new_group(ranks=list(range(world_size)))
                  inter_results = benchmark_deepep_internode(group=world_group)
                  all_inter = [None] * world_size
                  dist.all_gather_object(all_inter, inter_results)
                  if rank == 0:
                      r = all_inter[0]
                      print(f'  dispatch={r["dispatch_bw_gbps"]:.1f} GB/s, combine={r["combine_bw_gbps"]:.1f} GB/s, total={r["total_bw_gbps"]:.1f} GB/s')
                      print(f'  layout={r["layout_ms"]:.1f}ms, dispatch={r["dispatch_ms"]:.1f}ms, combine={r["combine_ms"]:.1f}ms')
              except Exception as e:
                  print(f'  [rank {rank}] INTER ERROR: {e}', flush=True)
                  traceback.print_exc()
              dist.barrier()
              if rank == 0:
                  print('\n' + '=' * 70)
                  print('DeepEP benchmark complete.')
              dist.destroy_process_group()
          if __name__ == '__main__':
              main()
          PYTHON_SCRIPT
          echo "=== Starting torchrun ==="
          torchrun \
            --nnodes=2 \
            --nproc_per_node=8 \
            --node_rank=$NODE_RANK \
            --master_addr=$MASTER_ADDR \
            --master_port=29400 \
            --rdzv_conf=timeout=900 \
            /tmp/deepep_benchmark.py
        env:
        - name: NVSHMEM_HCA_OFFSET
          value: "5"
        - name: NVSHMEM_IB_ENABLE_IBGDA
          value: "1"
        - name: NCCL_IB_HCA
          value: "mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1,mlx5_10:1,mlx5_11:1,mlx5_12:1"
        - name: NCCL_IB_DISABLE
          value: "0"
        - name: NCCL_NVLS_ENABLE
          value: "1"
        - name: NCCL_IB_SL
          value: "1"
        - name: NCCL_IB_PCI_RELAXED_ORDERING
          value: "1"
        - name: NCCL_IB_SPLIT_DATA_ON_QPS
          value: "0"
        - name: NCCL_IB_QPS_PER_CONNECTION
          value: "2"
        - name: NCCL_IBEXT_DISABLE
          value: "1"
        - name: NCCL_IB_MERGE_VFS
          value: "0"
        - name: NCCL_MAX_CTAS
          value: "32"
        - name: NCCL_MIN_CTAS
          value: "32"
        - name: NCCL_TOPO_FILE
          value: "/etc/crusoe/nccl_topo/b200-180gb-sxm-ib-cloud-hypervisor.xml"
        - name: NCCL_SOCKET_IFNAME
          value: "eth0"
        - name: NCCL_SOCKET_FAMILY
          value: "AF_INET"
        - name: GLOO_SOCKET_IFNAME
          value: "eth0"
        - name: GLOO_SOCKET_FAMILY
          value: "AF_INET"
        - name: UCX_NET_DEVICES
          value: "mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1,mlx5_10:1,mlx5_11:1,mlx5_12:1"
        - name: UCX_MEMTYPE_CACHE
          value: "n"
        - name: NVSHMEM_BOOTSTRAP
          value: "UID"
        - name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME
          value: "eth0"
        - name: NVSHMEM_BOOTSTRAP_UID_SOCK_FAMILY
          value: "AF_INET"
        - name: NVSHMEM_IB_SL
          value: "1"
        - name: NVSHMEM_ENABLE_NIC_PE_MAPPING
          value: "1"
        - name: NODE_RANK
          valueFrom:
            fieldRef:
              fieldPath: "metadata.labels['batch.kubernetes.io/job-completion-index']"
        - name: NUM_NODES
          value: "2"
        - name: NUM_GPUS_PER_NODE
          value: "8"
        - name: WORLD_SIZE
          value: "16"
        resources:
          requests:
            cpu: "158"
            memory: "900Gi"
            nvidia.com/gpu: "8"
            nvidia.com/hostdev: "8"
          limits:
            memory: "950Gi"
            nvidia.com/gpu: "8"
            nvidia.com/hostdev: "8"
        securityContext:
          capabilities:
            add:
            - IPC_LOCK
            - SYS_PTRACE
        volumeMounts:
        - name: dshm
          mountPath: /dev/shm
        - name: crusoe-config
          mountPath: /etc/crusoe
          readOnly: true

Apply:

kubectl apply -f deepep-bench.yaml

ℹ️ Note: To re-run the benchmark (after a fix, or for repeat measurements), delete the Job first — Job specs are immutable, so re-applying over an existing Job fails: kubectl -n deepep-test delete job deepep-bench then kubectl apply -f deepep-bench.yaml. Completed pods and their logs are cleaned up automatically one hour after the Job finishes (ttlSecondsAfterFinished), so capture the log output before then. The manifest sets imagePullPolicy: Always so that if you rebuild the image under the same tag, nodes fetch the new version instead of running a stale cached copy.

📝 Example (H200): The manifest above is shown with B200 values. For H200 SXM IB, replace the six SKU-specific values with:

• image: → ghcr.io/crusoecloud/deepep-tests:h200-torch29-nvshmem345

NVSHMEM_HCA_OFFSET → "1"

NCCL_IB_HCA and UCX_NET_DEVICES → "mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1"

NCCL_TOPO_FILE → "/etc/crusoe/nccl_topo/h200-141gb-sxm-ib-cloud-hypervisor.xml"

resources → memory: "1700Gi" (requests and limits; CPU, GPU, and hostdev unchanged)

💡 Tip: The pod anti-affinity only guarantees the pods land on different GPU nodes — on clusters with multiple GPU node pools, they can land on any pool. To validate a specific pool, add a nodeSelector for that pool's label, e.g. nvidia.com/gpu.product: NVIDIA-H200 (check values with kubectl get nodes -L nvidia.com/gpu.product). On mixed H200/B200 clusters this also prevents pods from landing on a pool the image was not built for. Note the label is applied by GPU Feature Discovery and can be temporarily absent while driver pods are restarting — if the column is empty, re-check after the driver pods reach 3/3, or a nodeSelector referencing it will leave the benchmark pods Pending.

Step 8: Run the Test and Interpret Results

Wait for the pods to start. The first run pulls the image onto each node, so expect several minutes of ContainerCreating; subsequent runs start immediately from the node-local image cache. Pulls from public internet registries (such as the Step 6 images on GHCR) can take 10+ minutes, and if the two nodes' pulls diverge enough, the faster pod can exhaust its 15-minute rendezvous window and fail. If that happens, delete the Job and re-apply — the second attempt starts in seconds from the now-cached image.

kubectl -n deepep-test get pods -l app=deepep-bench -o wide -w

Once all pods show 1/1 Running, tail the rank-0 pod:

kubectl -n deepep-test logs -f $(kubectl -n deepep-test get pods -l app=deepep-bench -o name | head -1)

A healthy run completes in a few minutes. The banner confirms the stack, then intra-node results print per node, then the inter-node section (each rank prints an NVSHMEM v3.4.5 line confirming the NVSHMEM/IBGDA context initialized). Output from a validated 2× B200 SXM IB run:

======================================================================
DeepEP Benchmark: 2 nodes, 16 GPUs
Driver: NVIDIA B200
NVSHMEM_HCA_LIST mapping: mlx5_(local_rank+5):1 per rank
PyTorch: 2.9.0a0+145a3a7bda.nv25.10, CUDA: 13.0, NCCL: (2, 27, 7)
======================================================================

--- Intra-node DeepEP (NVLink) ---
  Node 0: dispatch=276.9 GB/s, combine=245.1 GB/s, total=260.0 GB/s
  Node 1: dispatch=275.4 GB/s, combine=243.4 GB/s, total=258.4 GB/s

--- Inter-node DeepEP (RDMA + IBGDA) ---
NVSHMEM v3.4.5
  dispatch=56.1 GB/s, combine=51.0 GB/s, total=51.9 GB/s

Output from a validated 2× H200 SXM IB run of the same manifest with the Step 7 H200 values applied:

======================================================================
DeepEP Benchmark: 2 nodes, 16 GPUs
Driver: NVIDIA H200
NVSHMEM_HCA_LIST mapping: mlx5_(local_rank+1):1 per rank
PyTorch: 2.9.0a0+145a3a7bda.nv25.10, CUDA: 13.0, NCCL: (2, 27, 7)
======================================================================

--- Intra-node DeepEP (NVLink) ---
  Node 0: dispatch=274.8 GB/s, combine=289.4 GB/s, total=281.9 GB/s
  Node 1: dispatch=277.5 GB/s, combine=290.3 GB/s, total=283.8 GB/s

--- Inter-node DeepEP (RDMA + IBGDA) ---
NVSHMEM v3.4.5
  dispatch=54.1 GB/s, combine=45.8 GB/s, total=48.6 GB/s

Expected reference ranges:

The B200 ranges are consolidated from validation runs in two Crusoe regions plus a third independent cluster (a 4-node run and two 2-node runs, agreeing within a few percent); the third run used the pre-built Step 6 image with this article's manifest end to end. The H200 ranges are consolidated from two 2-node runs of the same manifest, the second using the pre-built Step 6 image on a freshly configured cluster. The inter-node path is bounded by the InfiniBand NICs rather than the GPUs, so the same rail-optimized fabric design gives both SKUs similar inter-node ranges. On the intra-node path, treat each SKU's range as a configuration-health baseline rather than an architecture comparison: at these benchmark settings the DeepEP kernels, not raw NVLink bandwidth, are the limiter. The two SKUs currently measure comparable dispatch throughput, and H200 measures higher combine throughput — the sm_100 code path enabled by the TMA patch described in Step 6 is newer than the mature Hopper path and is expected to improve over time.

ℹ️ Note: B200 intra-node results around 130–150 GB/s dispatch / 80–100 GB/s combine indicate the image was built without the TMA patch (or from a pre-Blackwell DeepEP revision) — the kernels are running without the aggressive PTX fast path. This is a build issue, not a hardware or fabric problem. Rebuild per Step 6 and re-run.

ℹ️ Note: Some run-to-run variation in the 5–10% range is normal, and inter-node figures shift modestly with node count (the fraction of tokens crossing the fabric changes). Values significantly below the reference ranges indicate a setup issue — see Step 9.

Step 9: Troubleshoot a Failing Run

Symptom: Inter-node benchmark logs show WARN: device mlx5_X cannot allocate buffer on the specified memory type. Skipping... followed by init failed for transport: IBGDA.

The required kernel modules are not loaded on one or more nodes. Re-run the verification from Step 4 — at least one node will show modules=0/2 or 1/2. Restart that node's driver pod and re-run the benchmark.

Symptom: Inter-node benchmark logs show cudaHostRegister with IoMemory failed with error=800.

The driver kernel parameters are not active on at least one node. Re-run the verification from Step 4 and look for nodes showing params=0/2. Restart the driver pod on the affected node.

Symptom: Benchmark fails at the first DeepEP kernel launch with a CUDA error — either named symbol not found (e.g. at csrc/kernels/layout.cu / runtime.cu) or no kernel image is available for execution on the device, often followed by EPException, ranks aborting with SIGABRT, and pods ending in Error.

The container image was built for the wrong GPU architecture (the kernels' machine code does not exist for the GPU the pod landed on). Confirm the image tag matches your SKU (Step 6) — or, for a self-built image, that TORCH_CUDA_ARCH_LIST matches (9.0 H200, 10.0 B200) — and on clusters with multiple GPU pools, confirm the Job's nodeSelector pins pods to the pool the image was built for.

Symptom: The image build fails with AssertionError: Cannot find package: nccl.

The clone picked up DeepEP's main branch, which is now V2 (NCCL backend). Confirm your build clones --branch hybrid-ep as the reference Dockerfile does (Step 6 note).

Symptom: At runtime, importing DeepEP fails with undefined symbol: nvshmem_selected_device_transport.

The NVSHMEM wheel version does not match the NVSHMEM integration bundled with the base image's PyTorch. Pin nvidia-nvshmem-cu13==3.4.5 as the reference Dockerfile does (Step 6 note) and rebuild.

Symptom: An image built on a newer NGC base (26.x) or with NVSHMEM 3.6.5+/3.7.x imports and runs, but the inter-node benchmark fails with dispatch timeouts (timeout (dispatch CPU)) or CUDA illegal-memory-access errors — while this guide's validated image passes on the same cluster.

This is a stack-compatibility limit, not a cluster fault. NVSHMEM releases after 3.4.5 changed the RC queue-pair structures the IBGDA path uses and are not drop-in compatible with the DeepEP hybrid-ep line on this platform — and newer NGC bases force NVSHMEM 3.6.5+, so the newer base pulls the whole combination outside this guide's validated stack. To distinguish this from a genuine cluster problem, run the validated image (the pre-built Step 6 tag for your SKU) on the same nodes: if it passes, the cluster and IBGDA prerequisites are healthy and the issue is the newer component stack. For the validation status of newer NGC/NVSHMEM stacks or DeepEP V2 on Crusoe fabric, contact Crusoe support.

Symptom: The image build's DeepEP compile step is killed (cudafe++ died due to signal 9 or cannot allocate memory).

The build machine ran out of RAM during parallel nvcc compilation. Lower MAX_JOBS in your build, build for a single architecture, or build on a machine with more memory.

Symptom: All ranks fail at startup with AssertionError: NVSHMEM_HCA_OFFSET must be set.

The NVSHMEM_HCA_OFFSET environment variable is missing from the Job manifest. Add it with the value from the SKU table ("1" H200, "5" B200).

Symptom: B200 intra-node results are roughly half the reference range (~130–150 GB/s dispatch) while inter-node is normal.

The image was built without the sm_100 TMA patch — DeepEP's setup.py gated off the aggressive PTX instructions. Confirm the sed patch, DISABLE_AGGRESSIVE_PTX_INSTRS=0, and the explicit python setup.py build_ext --inplace are all present in your build, as in the reference Dockerfile (Step 6 note), and rebuild.

Symptom: Inter-node dispatch reports a small fraction of the expected bandwidth (single-digit GB/s).

The per-rank NIC pinning is mapping ranks to the wrong HCAs. Verify the HCA mapping from Step 5 matches your NVSHMEM_HCA_OFFSET value, and verify the per-rank pattern (the script setting NVSHMEM_HCA_LIST inside each rank before deep_ep.Buffer() is constructed) is intact.

Symptom: Inter-node benchmark hangs with num_recv_tokens: -1 and eventually fails with DeepEP error: timeout (dispatch CPU).

Same root cause as the first symptom — kernel modules are missing on at least one node. The IBGDA initialization failed silently and the fallback path is in an inconsistent state. Follow the same resolution.

Symptom: Dispatch timeouts persist on CMK after following the Crusoe Cloud (VM) validation article, including after tuning NVSHMEM variables (NVSHMEM_IBGDA_NIC_HANDLER, NVSHMEM_IBGDA_FORCE_NIC_BUF_MEMTYPE) or manually creating /dev/gdrdrv inside the pod.

The VM article's driver configuration does not apply on CMK — the NVIDIA driver runs under the GPU Operator, and host-level module loading or device node creation from inside a pod does not produce a working IBGDA stack. No NVSHMEM environment tuning can compensate for missing kernel parameters and modules. Remove the workarounds and configure the driver through Steps 1–4 of this guide, then verify with Step 4 before re-running.

Symptom: After recycling, driver pods are stuck in Init:CrashLoopBackOff in the k8s-driver-manager init container, and its previous-attempt log (kubectl logs <driver-pod> -c k8s-driver-manager --previous) ends with cannot delete Pods with local storage followed by cannot proceed until all GPU pods are drained from the node. Other GPU Operator pods sit at Init:0/1.

Pods on the node are blocking the drain that precedes the driver reload; the error names them. Blockers come in two kinds. Bounded workloads (including platform burn-in or qualification tests on newly provisioned nodes) clear themselves — wait for them to complete, and the crash-looping init container proceeds on a later automatic retry. Long-lived pods with emptyDir storage (for example, monitoring agents scheduled onto the GPU node) never clear on their own: cordon the node manually, delete the blocking pod so its controller reschedules it elsewhere, and uncordon after the driver pod reaches 3/3 if the init process has not already done so. When several GPU nodes are draining at once, a rescheduled blocker can land on another draining node — clear nodes sequentially: bring one node's driver to 3/3 first, then move blockers off the next node so they land on the already-recovered one. For a durable fix on clusters where system pods with emptyDir routinely schedule onto GPU nodes, the NVIDIADriver CR's manager environment supports drain overrides — kubectl patch nvidiadriver <YOUR_NVIDIADRIVER_CR> --type='merge' -p '{"spec":{"manager":{"env":[{"name":"ENABLE_GPU_POD_EVICTION","value":"true"},{"name":"DRAIN_USE_FORCE","value":"true"},{"name":"DRAIN_DELETE_EMPTYDIR_DATA","value":"true"}]}}}' — after which the driver manager evicts such pods itself; use judgment, since forced drains discard evicted pods' local scratch data. Do not force-delete qualification tests — they are the node's acceptance record.

Symptom: A pod shows Running but its log stops after === Starting torchrun === for many minutes.

torchrun is waiting at rendezvous for the other pod's ranks — check the other pod, which has typically failed, been rescheduled, or is still pulling the image. The rendezvous window is 15 minutes (--rdzv_conf=timeout=900), after which the waiting pod exits with a rendezvous error. Fix the peer pod's issue, then delete the Job and re-apply.

Symptom: One or both pods remain Pending indefinitely.

The Job requires two GPU nodes with free capacity simultaneously (the pod anti-affinity forbids co-locating them). kubectl describe pod <pending-pod> shows the scheduling reason — typically insufficient allocatable CPU/memory/GPU on a second node, or no second node matching the affinity. Free up or add a second GPU node, or lower the CPU/memory requests to fit your nodes' allocatable values.

Symptom: Pods stay in ImagePullBackOff, with events showing FailedToRetrieveImagePullSecret or authorization failed: no basic auth credentials.

The public Step 6 images need no pull secret, so with those this points to a mistyped image URI. For a self-built image in a private registry, the deepep-pull secret is missing from the namespace, expired, or contains the wrong credential type — recreate it per the Step 6 note; for CCR, the username is your account email and the password is a registry token, not your console password, and the image URI must use the <repository-name>.<first-8-of-project-id> path format.

Symptom: Pods fail to start with a hostPath error for /etc/crusoe.

The node image does not expose the topology directory at the expected path. Confirm with the node debug check in Step 5; if the file name differs, update NCCL_TOPO_FILE, and if the directory is absent entirely, remove the NCCL_TOPO_FILE variable, the crusoe-config volume, and its mount, then re-run.

Example

A typical use case for this guide is the bring-up phase of a new CMK GPU node pool intended to run MoE training workloads. Before deploying a training framework, run this benchmark on two nodes to confirm that the DeepEP kernels are correctly built for the GPU architecture (including the sm_100 TMA path on B200), that the inter-node IBGDA path initializes correctly, and that bandwidth lands in the expected range.

A second common use case is verification after changes that affect the NVIDIA driver — applying a new GPU Operator version, updating the NVIDIADriver CR, or running cluster upgrades. Re-running the benchmark catches misconfigurations that would otherwise show up as training-time crashes.

A third use case is debugging a reported issue: if DeepEP-based training jobs are crashing with CUDA errors or dispatch timeouts, run this benchmark as a reproducer. If it fails with the same symptoms, the issue is at the cluster configuration layer rather than in the training code. The verification step (Step 4) then identifies which specific nodes need attention.

Related Articles

• How-To Validate DeepEP GPU Communication Performance on Crusoe Cloud
• Resolving MOFED Storage Module Race Condition on CMK GPU Nodes
• DeepEP GitHub Repository