How-To Validate DeepEP GPU Communication Performance on Crusoe Managed Kubernetes (H200 & B200)

Introduction

DeepEP is an open-source communication library from DeepSeek that accelerates Expert Parallelism (EP) in Mixture-of-Experts (MoE) models. It provides optimized all-to-all communication kernels that dispatch tokens to expert GPUs and combine results, using NVLink for intra-node traffic and InfiniBand RDMA for inter-node traffic.

This guide walks through deploying and running the DeepEP benchmark suite on a Crusoe Managed Kubernetes (CMK) cluster with H200 SXM IB or B200 SXM IB nodes. By the end you will have validated both the intra-node NVLink path and the inter-node InfiniBand path, and confirmed that the cluster is delivering bandwidth in the expected range for production MoE training workloads.

The benchmark exercises the two communication paths that dominate MoE training: NVLink within a node, and RDMA across nodes via IBGDA (InfiniBand GPUDirect Async). The IBGDA path is the most demanding to set up because it requires the GPU to write RDMA work requests directly to the InfiniBand HCA — and that in turn requires specific NVIDIA driver kernel parameters, GPUDirect RDMA kernel modules, and per-rank NVSHMEM configuration to be in place.

The procedure is identical for H200 and B200 except for the SKU-specific parameters summarized below and referenced throughout the guide. Wherever a step uses one of these values, it is called out explicitly.

This guide was validated end to end with the following software stack. The B200 configuration has been validated in two Crusoe regions on independent clusters, and the H200 configuration on a third, independent node pool. Newer component versions may work but are outside what this guide has verified — if you deviate from these versions and see results outside the reference ranges, reproduce with the validated stack before treating it as a cluster issue.

ℹ️ Note: All reference ranges in Step 8 were measured with this stack. The B200 figures consolidate a 4-node and a 2-node run in different regions; the H200 figures are from an initial 2-node validation run and may be refined as further runs accumulate.

⚠️ Warning: This guide is for CMK clusters only. If you are running on a bare-metal Crusoe Cloud VM, use How-To Validate DeepEP GPU Communication Performance on Crusoe Cloud instead — and conversely, do not follow that article on CMK: its driver configuration steps (host-level module loading, device node creation) do not apply on CMK, where the NVIDIA driver is managed by the GPU Operator. Attempting them inside pods leads to inter-node dispatch timeouts that no amount of NVSHMEM environment tuning can fix. On CMK, driver configuration must go through Steps 1–4 below.

Prerequisites

• CMK Cluster with an H200 SXM IB or B200 SXM IB Node Pool
• Two GPU Worker Nodes Minimum (Inter-Node Test)
• NVIDIA GPU Operator Installed in nvidia-gpu-operator Namespace
• NVIDIA Network Operator Add-On Enabled at Cluster Creation
• kubectl Access with Permissions to Patch the NVIDIADriver CR and Manage Driver Pods
• A Container Registry the Cluster Can Pull From (Pull Secret Required for Authenticated Registries)

Instructions

The workflow has three phases: configure the NVIDIA driver for IBGDA (Steps 1–4, done once per cluster), confirm the node's HCA layout and topology file (Step 5), and build and run the benchmark (Steps 6–8). Step 9 covers troubleshooting. All manifests are shown with B200 values; H200 customers change one line in the Dockerfile (Step 6) and five values in the Job manifest (Step 7) — exact replacements are provided at each point.

Step 1: Configure NVIDIA Driver Kernel Parameters

DeepEP's IBGDA transport requires two NVIDIA driver registry parameters:

• NVreg_EnableStreamMemOPs=1 — enables CUDA stream-ordered memory operations required for GPU-initiated RDMA work submission.
• NVreg_RegistryDwords="PeerMappingOverride=1" — enables peer memory mappings between the GPU and the InfiniBand HCA's user access region.

Create a ConfigMap containing these parameters. The GPU Operator's kernelModuleConfig reference will pick them up and apply them when the driver loads.

cat > nvidia-kernel-params.conf << 'CONF'
NVreg_EnableStreamMemOPs=1
NVreg_RegistryDwords="PeerMappingOverride=1"
CONF

kubectl -n nvidia-gpu-operator create configmap nvidia-kernel-params \
  --from-file=nvidia.conf=./nvidia-kernel-params.conf

Verify the ConfigMap was created with the expected content:

kubectl -n nvidia-gpu-operator get configmap nvidia-kernel-params -o yaml

💡 Tip: If your cluster already has a ConfigMap referenced by the existing NVIDIADriver CR under a different name, reuse that name rather than creating a duplicate. Inspect the existing CR with kubectl get nvidiadriver -o yaml | grep -A2 kernelModuleConfig.

Step 2: Patch the NVIDIADriver Custom Resource

First, identify the CR name for your GPU node pool. The name matches the GPU SKU associated with the pool (h200, b200, etc.) — the CR whose STATUS shows ready is the one backing your active nodes:

kubectl get nvidiadriver

Apply a single merge patch that enables GPUDirect RDMA, enables GDRCopy, and links the kernel parameter ConfigMap. Replace <YOUR_NVIDIADRIVER_CR> with the name returned above.

kubectl patch nvidiadriver <YOUR_NVIDIADRIVER_CR> --type='merge' -p='{
  "spec": {
    "gdrcopy": {
      "enabled": true,
      "image": "gdrdrv",
      "repository": "nvcr.io/nvidia/cloud-native",
      "version": "v2.5.1"
    },
    "kernelModuleConfig": {
      "name": "nvidia-kernel-params"
    },
    "rdma": {
      "enabled": true
    }
  }
}'

⚠️ Warning: Apply all three settings (gdrcopy, kernelModuleConfig, rdma) in a single merge. Partial application leaves the driver in an inconsistent state where IBGDA initialization fails at runtime.

Step 3: Apply the New Driver Configuration

The NVIDIADriver CR uses an OnDelete update strategy. Existing driver pods continue running with their previous configuration until they are restarted, so the patch from Step 2 will not take effect on running nodes until the driver pods are recycled.

Restart the driver pods across the cluster:

kubectl -n nvidia-gpu-operator delete pod -l app.kubernetes.io/component=nvidia-driver

Wait for the new pods to come up. Each pod contains three containers — the main driver, the GDRCopy module loader, and the peermem module loader — and all three must reach Ready state. This takes 60–90 seconds.

kubectl -n nvidia-gpu-operator get pods -l app.kubernetes.io/component=nvidia-driver -w

Cancel the watch (Ctrl+C) when every pod shows 3/3 Ready.

ℹ️ Note: The container count is the tell that the Step 2 patch took effect. Correctly reconfigured pods report 0/3 from the moment they are created and climb to 3/3. If the new pods come up as 0/1 → 1/1, the driver is still running its previous configuration — most commonly because the pods were recycled before the CR patch was applied. Verify the CR spec with kubectl get nvidiadriver <YOUR_NVIDIADRIVER_CR> -o yaml and recycle the pods again after patching.

ℹ️ Note: Nodes that join the cluster after the patch (new nodes, scaled or recreated pools) start their driver pods with the new configuration automatically — the manual recycle is only required for driver pods that were already running when the patch was applied.

Step 4: Verify Driver State on Each Node

Run the following check to confirm the kernel parameters are applied and the required kernel modules are loaded on every GPU node.

for pod in $(kubectl -n nvidia-gpu-operator get pods -l app.kubernetes.io/component=nvidia-driver -o name); do
  node=$(kubectl -n nvidia-gpu-operator get $pod -o jsonpath='{.spec.nodeName}')
  ready=$(kubectl -n nvidia-gpu-operator get $pod -o jsonpath='{.status.containerStatuses[*].ready}' | tr ' ' '/')
  params=$(kubectl -n nvidia-gpu-operator exec $pod -c nvidia-driver-ctr -- \
    cat /proc/driver/nvidia/params 2>/dev/null | grep -cE 'EnableStreamMemOPs: 1|RegistryDwords: "PeerMappingOverride')
  mods=$(kubectl -n nvidia-gpu-operator exec $pod -c nvidia-driver-ctr -- \
    lsmod 2>/dev/null | grep -cE '^(nvidia_peermem|gdrdrv) ')
  printf "%-55s ready=%s params=%d/2 modules=%d/2\n" "$node" "$ready" "$params" "$mods"
done

A correctly configured node prints:

<your-node-name>                                  ready=true/true/true params=2/2 modules=2/2

If any node shows fewer than 2/2 modules on a Ready pod, restart that node's driver pod:

kubectl -n nvidia-gpu-operator delete pod <pod-name-on-affected-node>

💡 Tip: Wait at least 60 seconds after a pod restart before re-running the check. The module loader containers need time to insert the modules after pod start. Re-run this verification any time nodes are added or replaced, before benchmarking.

Step 5: Confirm the Backend HCA Mapping and Topology File

The benchmark pins each rank to a dedicated backend InfiniBand HCA, and the device numbering differs between GPU SKUs:

• H200 SXM IB: mlx5_0 is the frontend NIC; mlx5_1–mlx5_8 are the eight rail-optimized backend NICs. Mapping: GPU N → mlx5_(N+1).
• B200 SXM IB: mlx5_4 is the frontend NIC (NODE affinity, attached to the first NUMA domain only); mlx5_5–mlx5_12 are the eight backend NICs, four per NUMA domain, serving GPUs 0–3 and 4–7 respectively. Mapping: GPU N → mlx5_(N+5).

Using the wrong offset pins ranks to the frontend NIC or to nonexistent devices, and the failure is silent — the benchmark runs, but at a fraction of expected inter-node bandwidth.

Confirm the layout on your nodes before running the benchmark. The driver pods run in the host namespace, so you can check directly from one without a separate debug pod:

POD=$(kubectl -n nvidia-gpu-operator get pods -l app.kubernetes.io/component=nvidia-driver -o name | head -1)
kubectl -n nvidia-gpu-operator exec $POD -c nvidia-driver-ctr -- nvidia-smi topo -m
kubectl -n nvidia-gpu-operator exec $POD -c nvidia-driver-ctr -- ls /sys/class/infiniband/

ℹ️ Note: ibdev2netdev is not present in the driver container. The topology matrix plus the /sys/class/infiniband/ listing provide the same information.

From the output, confirm which mlx5_X devices are the backend rails (the frontend NIC shows a different affinity class, NODE, than the rails, PHB), and that the offset for your SKU matches the table above. If it differs, use the actual offset and device names in the Job manifest in Step 7.

The Job manifest also points NCCL at the per-SKU topology file that Crusoe ships on every GPU node under /etc/crusoe/nccl_topo/. Confirm the file for your SKU exists — the driver container does not mount the host's /etc, so use a node debug pod:

kubectl debug node/<your-gpu-node-name> -it --image=busybox:1.36 -- ls /host/etc/crusoe/nccl_topo/

Expect the filename from the SKU table (e.g. b200-180gb-sxm-ib-cloud-hypervisor.xml). Delete the debugger pod afterwards.

Step 6: Build the Benchmark Container Image

The image is based on NVIDIA's public NGC PyTorch image (CUDA 13) and installs DeepEP from its hybrid-ep branch, NVSHMEM 3.4.5 via the official PyPI wheel, and the GDRCopy userspace library. All sources are public — no Crusoe-hosted image is required. Crusoe may also provide a pre-built benchmark image (check Related Articles and release announcements); if one is available you can use it and skip the build, but the Dockerfile below remains the reference recipe, including for integrating these changes into your own images.

Three build details are load-bearing and are also explained inline in the Dockerfile:

• DeepEP branch: DeepEP's main branch is now V2, which replaced the NVSHMEM/IBGDA transport with an NCCL-based backend and changed the Python API. A main build fails with Cannot find package: nccl and would not exercise the IBGDA path this guide validates. The hybrid-ep branch is the actively developed NVSHMEM-backend line (the same branch SGLang pins for its CUDA 13 builds).
• NVSHMEM version pin: the base image's PyTorch ships its own NVSHMEM integration compiled against 3.4.5. Installing a different NVSHMEM wheel version causes a runtime ABI mismatch (undefined symbol: nvshmem_selected_device_transport). The pin keeps both consistent.
• The sm_100 TMA patch (B200): DeepEP's setup.py hard-gates its aggressive TMA PTX instructions to compute architecture 9.0 (Hopper) only. On B200 (sm_100), that gate must be patched out and DISABLE_AGGRESSIVE_PTX_INSTRS=0 set, or the intra-node kernels compile without the fast path and NVLink throughput drops to roughly half. The sed step in the Dockerfile applies this; it is a no-op on an H200 (9.0) build, so the same Dockerfile serves both SKUs.

Save the following as Dockerfile. SKU-specific: the TORCH_CUDA_ARCH_LIST value is 10.0 for B200 as shown — for H200, change it to 9.0. This is the only SKU-specific line:

FROM nvcr.io/nvidia/pytorch:25.10-py3

# InfiniBand userspace libraries
RUN apt-get update && apt-get install -y --no-install-recommends \
    libibverbs-dev \
    ibverbs-utils \
 && rm -rf /var/lib/apt/lists/*

# GDRCopy v2.5 userspace library (the kernel module comes from the host
# via the GPU Operator). Build only the library: the test binaries
# hardcode -gencode flags for architectures CUDA 13's nvcc no longer
# supports.
ADD https://github.com/NVIDIA/gdrcopy/archive/refs/tags/v2.5.tar.gz /tmp/
RUN tar -xzf /tmp/v2.5.tar.gz -C /tmp/ && \
    cd /tmp/gdrcopy-2.5/ && \
    make -j$(nproc) prefix=/opt/gdrcopy CUDA=/usr/local/cuda lib lib_install

WORKDIR /workspace
# DeepEP: main is now V2 (NCCL backend, no NVSHMEM). hybrid-ep is the
# NVSHMEM-backend line this guide validates.
RUN git clone --branch hybrid-ep --depth 1 https://github.com/deepseek-ai/DeepEP.git
# Optional but recommended for reproducible builds: pin a specific
# commit instead of the branch head, e.g.:
#   RUN cd DeepEP && git fetch --depth 1 origin <commit-sha> && git checkout <commit-sha>
# (this guide's own image builds and the H200 validation run used
#  hybrid-ep commit 94a9f8f)

# NVSHMEM via the official PyPI wheel. The version pin is required: the
# base image's PyTorch bundles its own NVSHMEM integration built against
# 3.4.5, and a mismatched wheel causes a runtime ABI error
# ("undefined symbol: nvshmem_selected_device_transport").
RUN pip install nvidia-nvshmem-cu13==3.4.5

# NVSHMEM device headers need CCCL headers not on the default include path.
ENV CPLUS_INCLUDE_PATH="/usr/local/cuda-13.0/targets/x86_64-linux/include/cccl:${CPLUS_INCLUDE_PATH:-}"

# SKU-specific: 10.0 = B200 (sm_100). For H200, use 9.0.
ENV TORCH_CUDA_ARCH_LIST="10.0"
# Enable DeepEP's aggressive TMA PTX instructions. setup.py force-disables
# them for any arch other than 9.0; the sed patch below removes that gate
# so this setting is respected on sm_100. Without this, B200 intra-node
# throughput is roughly halved. Harmless (no-op) on a 9.0 build.
ENV DISABLE_AGGRESSIVE_PTX_INSTRS=0
# Limit parallel nvcc processes to avoid exhausting build-machine RAM
# ("cudafe++ died due to signal 9"). Raise if RAM allows.
ENV MAX_JOBS=4
RUN cd DeepEP && \
    # Patch out setup.py's arch gate so aggressive PTX compiles for sm_100:
    sed -i \
      -e 's/assert int(os.getenv(.DISABLE_AGGRESSIVE_PTX_INSTRS., 1)) == 1/pass  # patched for sm_100/' \
      -e "s/if os.environ\['TORCH_CUDA_ARCH_LIST'\].strip() != '9.0':/if False:  # patched: allow aggressive PTX for sm_100/" \
      setup.py && \
    # Compile the CUDA extension at build time (editable installs defer
    # build_ext to first import, which would ship an uncompiled image):
    python setup.py build_ext --inplace && \
    pip install --no-build-isolation -e .

Build the image for the linux/amd64 platform using your standard tooling (Docker, buildx, or a CI pipeline) and push it to any container registry your cluster can pull from:

docker build -t <YOUR_IMAGE_URI> .
docker push <YOUR_IMAGE_URI>

Create the benchmark namespace:

kubectl create namespace deepep-test

If your registry requires pull authentication, also create a standard docker-registry pull secret in the namespace (referenced as deepep-pull by the Job in Step 7; omit it there if your registry allows anonymous pulls):

kubectl -n deepep-test create secret docker-registry deepep-pull \
  --docker-server=<your-registry-host> \
  --docker-username='<registry-username>' \
  --docker-password='<registry-password-or-token>'

💡 Tip: If using Crusoe Container Registry, note that repository URIs follow the format registry.<region>.ccr.crusoecloudcompute.com/<repository-name>.<first-8-of-project-id>/<image-name>:<tag>, and authentication uses your account email with a registry token as the password. See How-To: Use the Crusoe Container Registry (CCR) for details.

Step 7: Deploy the Benchmark Job

The benchmark script is embedded in the Job manifest, so no separate ConfigMap is needed. The script runs the intra-node (NVLink) benchmark first, then the inter-node (RDMA + IBGDA) benchmark, and prints per-node and aggregate bandwidth.

The SKU-specific HCA offset is supplied by the NVSHMEM_HCA_OFFSET environment variable; the script refuses to start without it, which prevents the silent wrong-NIC failure mode. NVSHMEM_HCA_LIST itself is set inside the script per rank — each rank initializes its own NVSHMEM context, so this value cannot be a pod-level environment variable. Apply the same per-rank pattern in any application code that constructs deep_ep.Buffer() on CMK.

Save the following as deepep-bench.yaml. Values to set:

1. image: — your image URI from Step 6.
2. imagePullSecrets — the deepep-pull secret from Step 6 (omit for anonymous registries).
3. NVSHMEM_HCA_OFFSET — "5" for B200 as shown; "1" for H200.
4. NCCL_IB_HCA / UCX_NET_DEVICES — the backend rails from Step 5: shown for B200 (mlx5_5–mlx5_12); for H200 use mlx5_1:1,...,mlx5_8:1.
5. NCCL_TOPO_FILE — shown for B200; for H200 use the H200 filename from the SKU table.
6. resources — set just below node allocatable (kubectl describe node <node> | grep -A8 Allocatable); the values shown are validated for typical node shapes (H200 uses memory: "1700Gi").
7. To run across more than two nodes, change completions, parallelism, --nnodes, NUM_NODES, and WORLD_SIZE (= nodes × 8) together.

apiVersion: v1
kind: Service
metadata:
  name: deepep-bench
  namespace: deepep-test
  labels:
    app: deepep-bench
spec:
  clusterIP: None
  selector:
    app: deepep-bench
  publishNotReadyAddresses: true
  ports:
  - name: torchrun
    port: 29400
    targetPort: 29400
    protocol: TCP
---
apiVersion: batch/v1
kind: Job
metadata:
  name: deepep-bench
  namespace: deepep-test
  labels:
    app: deepep-bench
spec:
  backoffLimit: 0
  completions: 2
  parallelism: 2
  ttlSecondsAfterFinished: 3600
  completionMode: Indexed
  template:
    metadata:
      labels:
        app: deepep-bench
    spec:
      subdomain: deepep-bench
      restartPolicy: Never
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nvidia.com/gpu.present
                operator: In
                values:
                - "true"
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: deepep-bench
            topologyKey: kubernetes.io/hostname
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 64Gi
      - name: crusoe-config
        hostPath:
          path: /etc/crusoe
          type: Directory
      imagePullSecrets:
      - name: deepep-pull
      containers:
      - name: deepep-bench
        image: <YOUR_IMAGE_URI>
        imagePullPolicy: Always
        command: ["/bin/bash", "-c"]
        args:
        - |
          set -euo pipefail
          echo "=== DeepEP Benchmark ==="
          echo "Node rank: $NODE_RANK / $NUM_NODES nodes"
          export NCCL_DEBUG=WARN
          export PYTHONPATH=/tmp:${PYTHONPATH:-}
          MASTER_ADDR="deepep-bench-0.deepep-bench.deepep-test.svc.cluster.local"
          cat > /tmp/deepep_benchmark.py << 'PYTHON_SCRIPT'
          import os
          import time
          import traceback
          import importlib
          import torch
          import torch.distributed as dist
          def benchmark_deepep_internode(group=None):
              deep_ep = importlib.import_module('deep_ep')
              deep_ep.Buffer.set_num_sms(20)
              rank = dist.get_rank()
              world_size = dist.get_world_size() if group is None else dist.get_world_size(group)
              num_tokens = 8192
              hidden_dim = 3072
              num_experts = 128
              topk = 8
              warmup_iters = 10
              bench_iters = 50
              num_nvl_bytes = 1024 * 1024 * 1024
              num_rdma_bytes = 512 * 1024 * 1024
              if rank == 0:
                  print(f'  Buffer: group_size={world_size}, nvl={num_nvl_bytes}, rdma={num_rdma_bytes}', flush=True)
              buffer = deep_ep.Buffer(
                  group=group,
                  num_nvl_bytes=num_nvl_bytes,
                  num_rdma_bytes=num_rdma_bytes,
                  low_latency_mode=False,
                  num_qps_per_rank=24,
              )
              x = torch.randn(num_tokens, hidden_dim, device='cuda', dtype=torch.bfloat16)
              topk_idx = torch.randint(0, num_experts, (num_tokens, topk), device='cuda', dtype=torch.int64)
              topk_weights = torch.rand(num_tokens, topk, device='cuda', dtype=torch.float32)
              for i in range(warmup_iters):
                  if rank == 0:
                      print(f'  warmup {i+1}/{warmup_iters}', flush=True)
                  try:
                      num_tokens_per_rank, num_tokens_per_rdma_rank, num_tokens_per_expert, is_token_in_rank, _ = \
                          buffer.get_dispatch_layout(topk_idx, num_experts)
                      recv_x, _, recv_topk_weights, _, handle, _ = buffer.dispatch(
                          x, None, num_tokens_per_rank, num_tokens_per_rdma_rank,
                          is_token_in_rank, num_tokens_per_expert, topk_idx, topk_weights,
                      )
                      if recv_x is not None and recv_x.size(0) > 0:
                          combine_x = torch.randn_like(recv_x)
                          _, _, _ = buffer.combine(combine_x, handle, recv_topk_weights)
                      torch.cuda.synchronize()
                  except Exception as e:
                      print(f'  [rank {rank}] warmup {i} FAILED: {e}', flush=True)
                      traceback.print_exc()
                      raise
              if rank == 0:
                  print(f'  benchmarking {bench_iters} iters...', flush=True)
              layout_start = torch.cuda.Event(enable_timing=True)
              layout_end = torch.cuda.Event(enable_timing=True)
              dispatch_start = torch.cuda.Event(enable_timing=True)
              dispatch_end = torch.cuda.Event(enable_timing=True)
              combine_start = torch.cuda.Event(enable_timing=True)
              combine_end = torch.cuda.Event(enable_timing=True)
              total_layout_ms = 0.0
              total_dispatch_ms = 0.0
              total_combine_ms = 0.0
              total_rdma_tokens = 0
              for it in range(bench_iters):
                  layout_start.record()
                  num_tokens_per_rank, num_tokens_per_rdma_rank, num_tokens_per_expert, is_token_in_rank, _ = \
                      buffer.get_dispatch_layout(topk_idx, num_experts)
                  layout_end.record()
                  layout_end.synchronize()
                  total_layout_ms += layout_start.elapsed_time(layout_end)
                  if num_tokens_per_rdma_rank is not None:
                      total_rdma_tokens += int(num_tokens_per_rdma_rank.sum().item())
                  dispatch_start.record()
                  recv_x, _, recv_topk_weights, _, handle, _ = buffer.dispatch(
                      x, None, num_tokens_per_rank, num_tokens_per_rdma_rank,
                      is_token_in_rank, num_tokens_per_expert, topk_idx, topk_weights,
                  )
                  dispatch_end.record()
                  dispatch_end.synchronize()
                  total_dispatch_ms += dispatch_start.elapsed_time(dispatch_end)
                  if recv_x is not None and recv_x.size(0) > 0:
                      combine_x = torch.randn_like(recv_x)
                      combine_start.record()
                      _, _, _ = buffer.combine(combine_x, handle, recv_topk_weights)
                      combine_end.record()
                      combine_end.synchronize()
                      total_combine_ms += combine_start.elapsed_time(combine_end)
              bytes_per_token = hidden_dim * 2
              total_rdma_bytes = total_rdma_tokens * bytes_per_token * 2
              total_ms = total_layout_ms + total_dispatch_ms + total_combine_ms
              dispatch_bw = (total_rdma_tokens * bytes_per_token) / (total_dispatch_ms / 1000) / 1e9 if total_dispatch_ms > 0 else 0
              combine_bw = (total_rdma_tokens * bytes_per_token) / (total_combine_ms / 1000) / 1e9 if total_combine_ms > 0 else 0
              total_bw = total_rdma_bytes / (total_ms / 1000) / 1e9 if total_ms > 0 else 0
              return {
                  'dispatch_bw_gbps': dispatch_bw,
                  'combine_bw_gbps': combine_bw,
                  'total_bw_gbps': total_bw,
                  'layout_ms': total_layout_ms,
                  'dispatch_ms': total_dispatch_ms,
                  'combine_ms': total_combine_ms,
              }
          def benchmark_deepep_intranode(node_group, num_sms=40):
              deep_ep = importlib.import_module('deep_ep')
              deep_ep.Buffer.set_num_sms(num_sms)
              num_tokens = 65536
              hidden_dim = 3072
              num_experts = 128
              topk = 16
              num_channels = num_sms // 2
              num_ranks_local = dist.get_world_size(node_group)
              nvl_bytes = max(
                  256 * 1024 * 1024,
                  num_channels * num_ranks_local * 6 * hidden_dim * 2 * 4
              )
              buffer = deep_ep.Buffer(
                  group=node_group,
                  num_nvl_bytes=nvl_bytes,
                  num_rdma_bytes=0,
                  low_latency_mode=False,
              )
              x = torch.randn(num_tokens, hidden_dim, device='cuda', dtype=torch.bfloat16)
              topk_idx = torch.randint(0, num_experts, (num_tokens, topk), device='cuda', dtype=torch.int64)
              topk_weights = torch.rand(num_tokens, topk, device='cuda', dtype=torch.float32)
              local_rank = dist.get_rank(node_group)
              for _ in range(10):
                  num_tokens_per_rank, _, num_tokens_per_expert, is_token_in_rank, _ = \
                      buffer.get_dispatch_layout(topk_idx, num_experts)
                  recv_x, _, recv_topk_weights, _, handle, _ = buffer.dispatch(
                      x, None, num_tokens_per_rank, None, is_token_in_rank,
                      num_tokens_per_expert, topk_idx, topk_weights,
                  )
                  if recv_x is not None and recv_x.size(0) > 0:
                      combine_x = torch.randn_like(recv_x)
                      _, _, _ = buffer.combine(combine_x, handle, recv_topk_weights)
                  torch.cuda.synchronize()
              dispatch_start = torch.cuda.Event(enable_timing=True)
              dispatch_end = torch.cuda.Event(enable_timing=True)
              combine_start = torch.cuda.Event(enable_timing=True)
              combine_end = torch.cuda.Event(enable_timing=True)
              total_dispatch_ms = 0.0
              total_combine_ms = 0.0
              total_nvl_tokens = 0
              bench_iters = 50
              for _ in range(bench_iters):
                  num_tokens_per_rank, _, num_tokens_per_expert, is_token_in_rank, _ = \
                      buffer.get_dispatch_layout(topk_idx, num_experts)
                  tokens_total = int(num_tokens_per_rank.sum().item())
                  tokens_self = int(num_tokens_per_rank[local_rank].item())
                  total_nvl_tokens += tokens_total - tokens_self
                  dispatch_start.record()
                  recv_x, _, recv_topk_weights, _, handle, _ = buffer.dispatch(
                      x, None, num_tokens_per_rank, None, is_token_in_rank,
                      num_tokens_per_expert, topk_idx, topk_weights,
                  )
                  dispatch_end.record()
                  dispatch_end.synchronize()
                  total_dispatch_ms += dispatch_start.elapsed_time(dispatch_end)
                  if recv_x is not None and recv_x.size(0) > 0:
                      combine_x = torch.randn_like(recv_x)
                      combine_start.record()
                      _, _, _ = buffer.combine(combine_x, handle, recv_topk_weights)
                      combine_end.record()
                      combine_end.synchronize()
                      total_combine_ms += combine_start.elapsed_time(combine_end)
              bytes_per_token = hidden_dim * 2
              total_nvl_bytes = total_nvl_tokens * bytes_per_token * 2
              total_ms = total_dispatch_ms + total_combine_ms
              dispatch_bw = (total_nvl_tokens * bytes_per_token) / (total_dispatch_ms / 1000) / 1e9 if total_dispatch_ms > 0 else 0
              combine_bw = (total_nvl_tokens * bytes_per_token) / (total_combine_ms / 1000) / 1e9 if total_combine_ms > 0 else 0
              total_bw = total_nvl_bytes / (total_ms / 1000) / 1e9 if total_ms > 0 else 0
              return {
                  'dispatch_bw_gbps': dispatch_bw,
                  'combine_bw_gbps': combine_bw,
                  'total_bw_gbps': total_bw,
                  'dispatch_ms': total_dispatch_ms,
                  'combine_ms': total_combine_ms,
              }
          def main():
              local_rank = int(os.environ.get('LOCAL_RANK', '0'))
              torch.cuda.set_device(local_rank)
              hca_offset = os.environ.get('NVSHMEM_HCA_OFFSET')
              assert hca_offset is not None, \
                  'NVSHMEM_HCA_OFFSET must be set in the Job manifest (H200: "1", B200: "5")'
              os.environ["NVSHMEM_HCA_LIST"] = f"mlx5_{local_rank + int(hca_offset)}:1"
              dist.init_process_group(backend='nccl', device_id=torch.device(f'cuda:{local_rank}'))
              rank = dist.get_rank()
              world_size = dist.get_world_size()
              local_world_size = int(os.environ.get('LOCAL_WORLD_SIZE', '8'))
              num_nodes = world_size // local_world_size
              node_id = rank // local_world_size
              if rank == 0:
                  print('=' * 70)
                  print(f'DeepEP Benchmark: {num_nodes} nodes, {world_size} GPUs')
                  print(f'Driver: {torch.cuda.get_device_properties(0).name}')
                  print(f'NVSHMEM_HCA_LIST mapping: mlx5_(local_rank+{hca_offset}):1 per rank')
                  try:
                      with open('/proc/driver/nvidia/version') as f:
                          print(f'NVIDIA: {f.readline().strip()}')
                  except: pass
                  print(f'PyTorch: {torch.__version__}, CUDA: {torch.version.cuda}, NCCL: {torch.cuda.nccl.version()}')
                  print('=' * 70)
              per_node_groups = []
              for n in range(num_nodes):
                  ranks = list(range(n * local_world_size, (n + 1) * local_world_size))
                  g = dist.new_group(ranks=ranks)
                  per_node_groups.append(g)
              my_node_group = per_node_groups[node_id]
              if rank == 0:
                  print('\n--- Intra-node DeepEP (NVLink) ---')
              dist.barrier()
              try:
                  intra_results = benchmark_deepep_intranode(my_node_group)
                  all_intra = [None] * world_size
                  dist.all_gather_object(all_intra, intra_results)
                  if rank == 0:
                      for n in range(num_nodes):
                          r = all_intra[n * local_world_size]
                          print(f'  Node {n}: dispatch={r["dispatch_bw_gbps"]:.1f} GB/s, combine={r["combine_bw_gbps"]:.1f} GB/s, total={r["total_bw_gbps"]:.1f} GB/s')
              except Exception as e:
                  print(f'  [rank {rank}] INTRA ERROR: {e}', flush=True)
                  traceback.print_exc()
              dist.barrier()
              if rank == 0:
                  print('\n--- Inter-node DeepEP (RDMA + IBGDA) ---')
              dist.barrier()
              try:
                  world_group = dist.new_group(ranks=list(range(world_size)))
                  inter_results = benchmark_deepep_internode(group=world_group)
                  all_inter = [None] * world_size
                  dist.all_gather_object(all_inter, inter_results)
                  if rank == 0:
                      r = all_inter[0]
                      print(f'  dispatch={r["dispatch_bw_gbps"]:.1f} GB/s, combine={r["combine_bw_gbps"]:.1f} GB/s, total={r["total_bw_gbps"]:.1f} GB/s')
                      print(f'  layout={r["layout_ms"]:.1f}ms, dispatch={r["dispatch_ms"]:.1f}ms, combine={r["combine_ms"]:.1f}ms')
              except Exception as e:
                  print(f'  [rank {rank}] INTER ERROR: {e}', flush=True)
                  traceback.print_exc()
              dist.barrier()
              if rank == 0:
                  print('\n' + '=' * 70)
                  print('DeepEP benchmark complete.')
              dist.destroy_process_group()
          if __name__ == '__main__':
              main()
          PYTHON_SCRIPT
          echo "=== Starting torchrun ==="
          torchrun \
            --nnodes=2 \
            --nproc_per_node=8 \
            --node_rank=$NODE_RANK \
            --master_addr=$MASTER_ADDR \
            --master_port=29400 \
            --rdzv_conf=timeout=900 \
            /tmp/deepep_benchmark.py
        env:
        - name: NVSHMEM_HCA_OFFSET
          value: "5"
        - name: NVSHMEM_IB_ENABLE_IBGDA
          value: "1"
        - name: NCCL_IB_HCA
          value: "mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1,mlx5_10:1,mlx5_11:1,mlx5_12:1"
        - name: NCCL_IB_DISABLE
          value: "0"
        - name: NCCL_NVLS_ENABLE
          value: "1"
        - name: NCCL_IB_SL
          value: "1"
        - name: NCCL_IB_PCI_RELAXED_ORDERING
          value: "1"
        - name: NCCL_IB_SPLIT_DATA_ON_QPS
          value: "0"
        - name: NCCL_IB_QPS_PER_CONNECTION
          value: "2"
        - name: NCCL_IBEXT_DISABLE
          value: "1"
        - name: NCCL_IB_MERGE_VFS
          value: "0"
        - name: NCCL_MAX_CTAS
          value: "32"
        - name: NCCL_MIN_CTAS
          value: "32"
        - name: NCCL_TOPO_FILE
          value: "/etc/crusoe/nccl_topo/b200-180gb-sxm-ib-cloud-hypervisor.xml"
        - name: NCCL_SOCKET_IFNAME
          value: "eth0"
        - name: NCCL_SOCKET_FAMILY
          value: "AF_INET"
        - name: GLOO_SOCKET_IFNAME
          value: "eth0"
        - name: GLOO_SOCKET_FAMILY
          value: "AF_INET"
        - name: UCX_NET_DEVICES
          value: "mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1,mlx5_10:1,mlx5_11:1,mlx5_12:1"
        - name: UCX_MEMTYPE_CACHE
          value: "n"
        - name: NVSHMEM_BOOTSTRAP
          value: "UID"
        - name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME
          value: "eth0"
        - name: NVSHMEM_BOOTSTRAP_UID_SOCK_FAMILY
          value: "AF_INET"
        - name: NVSHMEM_IB_SL
          value: "1"
        - name: NVSHMEM_ENABLE_NIC_PE_MAPPING
          value: "1"
        - name: NODE_RANK
          valueFrom:
            fieldRef:
              fieldPath: "metadata.labels['batch.kubernetes.io/job-completion-index']"
        - name: NUM_NODES
          value: "2"
        - name: NUM_GPUS_PER_NODE
          value: "8"
        - name: WORLD_SIZE
          value: "16"
        resources:
          requests:
            cpu: "158"
            memory: "900Gi"
            nvidia.com/gpu: "8"
            nvidia.com/hostdev: "8"
          limits:
            memory: "950Gi"
            nvidia.com/gpu: "8"
            nvidia.com/hostdev: "8"
        securityContext:
          capabilities:
            add:
            - IPC_LOCK
            - SYS_PTRACE
        volumeMounts:
        - name: dshm
          mountPath: /dev/shm
        - name: crusoe-config
          mountPath: /etc/crusoe
          readOnly: true

Apply:

kubectl apply -f deepep-bench.yaml

ℹ️ Note: To re-run the benchmark (after a fix, or for repeat measurements), delete the Job first — Job specs are immutable, so re-applying over an existing Job fails: kubectl -n deepep-test delete job deepep-bench then kubectl apply -f deepep-bench.yaml. Completed pods and their logs are cleaned up automatically one hour after the Job finishes (ttlSecondsAfterFinished), so capture the log output before then. The manifest sets imagePullPolicy: Always so that if you rebuild the image under the same tag, nodes fetch the new version instead of running a stale cached copy.

📝 Example (H200): The manifest above is shown with B200 values. For H200 SXM IB, replace the five SKU-specific values with:

NVSHMEM_HCA_OFFSET → "1"

NCCL_IB_HCA and UCX_NET_DEVICES → "mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1"

NCCL_TOPO_FILE → "/etc/crusoe/nccl_topo/h200-141gb-sxm-ib-cloud-hypervisor.xml"

resources → memory: "1700Gi" (requests and limits; CPU, GPU, and hostdev unchanged)

💡 Tip: The pod anti-affinity only guarantees the pods land on different GPU nodes — on clusters with multiple GPU node pools, they can land on any pool. To validate a specific pool, add a nodeSelector for that pool's label, e.g. nvidia.com/gpu.product: NVIDIA-H200 (check values with kubectl get nodes -L nvidia.com/gpu.product). On mixed H200/B200 clusters this also prevents pods from landing on a pool the image was not built for.

Step 8: Run the Test and Interpret Results

Wait for the pods to start. The first run pulls the image onto each node, so expect several minutes of ContainerCreating; subsequent runs start immediately from the node-local image cache.

kubectl -n deepep-test get pods -l app=deepep-bench -o wide -w

Once all pods show 1/1 Running, tail the rank-0 pod:

kubectl -n deepep-test logs -f $(kubectl -n deepep-test get pods -l app=deepep-bench -o name | head -1)

A healthy run completes in a few minutes. The banner confirms the stack, then intra-node results print per node, then the inter-node section (each rank prints an NVSHMEM v3.4.5 line confirming the NVSHMEM/IBGDA context initialized). Output from a validated 2× B200 SXM IB run:

======================================================================
DeepEP Benchmark: 2 nodes, 16 GPUs
Driver: NVIDIA B200
NVSHMEM_HCA_LIST mapping: mlx5_(local_rank+5):1 per rank
PyTorch: 2.9.0a0+145a3a7bda.nv25.10, CUDA: 13.0, NCCL: (2, 27, 7)
======================================================================

--- Intra-node DeepEP (NVLink) ---
  Node 0: dispatch=276.9 GB/s, combine=245.1 GB/s, total=260.0 GB/s
  Node 1: dispatch=275.4 GB/s, combine=243.4 GB/s, total=258.4 GB/s

--- Inter-node DeepEP (RDMA + IBGDA) ---
NVSHMEM v3.4.5
  dispatch=56.1 GB/s, combine=51.0 GB/s, total=51.9 GB/s

Output from a validated 2× H200 SXM IB run of the same manifest with the Step 7 H200 values applied:

======================================================================
DeepEP Benchmark: 2 nodes, 16 GPUs
Driver: NVIDIA H200
NVSHMEM_HCA_LIST mapping: mlx5_(local_rank+1):1 per rank
PyTorch: 2.9.0a0+145a3a7bda.nv25.10, CUDA: 13.0, NCCL: (2, 27, 7)
======================================================================

--- Intra-node DeepEP (NVLink) ---
  Node 0: dispatch=274.8 GB/s, combine=289.4 GB/s, total=281.9 GB/s
  Node 1: dispatch=277.5 GB/s, combine=290.3 GB/s, total=283.8 GB/s

--- Inter-node DeepEP (RDMA + IBGDA) ---
NVSHMEM v3.4.5
  dispatch=54.1 GB/s, combine=45.8 GB/s, total=48.6 GB/s

Expected reference ranges:

The B200 ranges are consolidated from validation runs in two Crusoe regions (a 4-node run and a 2-node run on independent clusters, agreeing within a few percent); the H200 ranges are from an initial 2-node validation run of the same manifest and may be refined as further runs accumulate. The inter-node path is bounded by the InfiniBand NICs rather than the GPUs, so the same rail-optimized fabric design gives both SKUs similar inter-node ranges. On the intra-node path, treat each SKU's range as a configuration-health baseline rather than an architecture comparison: at these benchmark settings the DeepEP kernels, not raw NVLink bandwidth, are the limiter. The two SKUs currently measure comparable dispatch throughput, and H200 measures higher combine throughput — the sm_100 code path enabled by the Step 6 TMA patch is newer than the mature Hopper path and is expected to improve over time.

ℹ️ Note: B200 intra-node results around 130–150 GB/s dispatch / 80–100 GB/s combine indicate the image was built without the TMA patch (or from a pre-Blackwell DeepEP revision) — the kernels are running without the aggressive PTX fast path. This is a build issue, not a hardware or fabric problem. Rebuild per Step 6 and re-run.

ℹ️ Note: Some run-to-run variation in the 5–10% range is normal, and inter-node figures shift modestly with node count (the fraction of tokens crossing the fabric changes). Values significantly below the reference ranges indicate a setup issue — see Step 9.

Step 9: Troubleshoot a Failing Run

Symptom: Inter-node benchmark logs show WARN: device mlx5_X cannot allocate buffer on the specified memory type. Skipping... followed by init failed for transport: IBGDA.

The required kernel modules are not loaded on one or more nodes. Re-run the verification from Step 4 — at least one node will show modules=0/2 or 1/2. Restart that node's driver pod and re-run the benchmark.

Symptom: Inter-node benchmark logs show cudaHostRegister with IoMemory failed with error=800.

The driver kernel parameters are not active on at least one node. Re-run the verification from Step 4 and look for nodes showing params=0/2. Restart the driver pod on the affected node.

Symptom: Benchmark fails at the first DeepEP kernel launch with a CUDA error — either named symbol not found (e.g. at csrc/kernels/layout.cu / runtime.cu) or no kernel image is available for execution on the device — often followed by EPException, ranks aborting with SIGABRT, and pods ending in Error.

The container image was built for the wrong GPU architecture (the kernels' machine code does not exist for the GPU the pod landed on). Confirm the Dockerfile sets TORCH_CUDA_ARCH_LIST correctly for your SKU (9.0 H200, 10.0 B200) and rebuild — and on clusters with multiple GPU pools, confirm the Job's nodeSelector pins pods to the pool the image was built for.

Symptom: The image build fails with AssertionError: Cannot find package: nccl.

The clone picked up DeepEP's main branch, which is now V2 (NCCL backend). Confirm the Dockerfile's clone line includes --branch hybrid-ep (see Step 6).

Symptom: At runtime, importing DeepEP fails with undefined symbol: nvshmem_selected_device_transport.

The NVSHMEM wheel version does not match the NVSHMEM integration bundled with the base image's PyTorch. Pin nvidia-nvshmem-cu13==3.4.5 as in the Step 6 Dockerfile and rebuild.

Symptom: The image build's DeepEP compile step is killed (cudafe++ died due to signal 9 or cannot allocate memory).

The build machine ran out of RAM during parallel nvcc compilation. Lower MAX_JOBS in the Dockerfile, build for a single architecture, or build on a machine with more memory.

Symptom: All ranks fail at startup with AssertionError: NVSHMEM_HCA_OFFSET must be set.

The NVSHMEM_HCA_OFFSET environment variable is missing from the Job manifest. Add it with the value from the SKU table ("1" H200, "5" B200).

Symptom: B200 intra-node results are roughly half the reference range (~130–150 GB/s dispatch) while inter-node is normal.

The image was built without the sm_100 TMA patch — DeepEP's setup.py gated off the aggressive PTX instructions. Confirm the sed patch, DISABLE_AGGRESSIVE_PTX_INSTRS=0, and the explicit python setup.py build_ext --inplace are all present in the Dockerfile (Step 6) and rebuild.

Symptom: Inter-node dispatch reports a small fraction of the expected bandwidth (single-digit GB/s).

The per-rank NIC pinning is mapping ranks to the wrong HCAs. Verify the HCA mapping from Step 5 matches your NVSHMEM_HCA_OFFSET value, and verify the per-rank pattern (the script setting NVSHMEM_HCA_LIST inside each rank before deep_ep.Buffer() is constructed) is intact.

Symptom: Inter-node benchmark hangs with num_recv_tokens: -1 and eventually fails with DeepEP error: timeout (dispatch CPU).

Same root cause as the first symptom — kernel modules are missing on at least one node. The IBGDA initialization failed silently and the fallback path is in an inconsistent state. Follow the same resolution.

Symptom: Dispatch timeouts persist on CMK after following the Crusoe Cloud (VM) validation article, including after tuning NVSHMEM variables (NVSHMEM_IBGDA_NIC_HANDLER, NVSHMEM_IBGDA_FORCE_NIC_BUF_MEMTYPE) or manually creating /dev/gdrdrv inside the pod.

The VM article's driver configuration does not apply on CMK — the NVIDIA driver runs under the GPU Operator, and host-level module loading or device node creation from inside a pod does not produce a working IBGDA stack. No NVSHMEM environment tuning can compensate for missing kernel parameters and modules. Remove the workarounds and configure the driver through Steps 1–4 of this guide, then verify with Step 4 before re-running.

Symptom: A pod shows Running but its log stops after === Starting torchrun === for many minutes.

torchrun is waiting at rendezvous for the other pod's ranks — check the other pod, which has typically failed, been rescheduled, or is still pulling the image. The rendezvous window is 15 minutes (--rdzv_conf=timeout=900), after which the waiting pod exits with a rendezvous error. Fix the peer pod's issue, then delete the Job and re-apply.

Symptom: One or both pods remain Pending indefinitely.

The Job requires two GPU nodes with free capacity simultaneously (the pod anti-affinity forbids co-locating them). kubectl describe pod <pending-pod> shows the scheduling reason — typically insufficient allocatable CPU/memory/GPU on a second node, or no second node matching the affinity. Free up or add a second GPU node, or lower the CPU/memory requests to fit your nodes' allocatable values.

Symptom: Pods stay in ImagePullBackOff, with events showing FailedToRetrieveImagePullSecret or authorization failed: no basic auth credentials.

The deepep-pull secret is missing from the namespace, expired, or contains the wrong credential type. Recreate it per Step 6 — for CCR, email as username and a registry token (not console password) as password — and confirm the image URI uses the <repository-name>.<first-8-of-project-id> path format.

Symptom: Pods fail to start with a hostPath error for /etc/crusoe.

The node image does not expose the topology directory at the expected path. Confirm with the node debug check in Step 5; if the file name differs, update NCCL_TOPO_FILE, and if the directory is absent entirely, remove the NCCL_TOPO_FILE variable, the crusoe-config volume, and its mount, then re-run.

Example

A typical use case for this guide is the bring-up phase of a new CMK GPU node pool intended to run MoE training workloads. Before deploying a training framework, run this benchmark on two nodes to confirm that the DeepEP kernels are correctly built for the GPU architecture (including the sm_100 TMA path on B200), that the inter-node IBGDA path initializes correctly, and that bandwidth lands in the expected range.

A second common use case is verification after changes that affect the NVIDIA driver — applying a new GPU Operator version, updating the NVIDIADriver CR, or running cluster upgrades. Re-running the benchmark catches misconfigurations that would otherwise show up as training-time crashes.

A third use case is debugging a reported issue: if DeepEP-based training jobs are crashing with CUDA errors or dispatch timeouts, running this benchmark functions as a reproducer. If it fails with the same symptoms, the issue is at the cluster configuration layer rather than in the training code. The verification step (Step 4) then identifies which specific nodes need attention.

Related Articles

• How-To Validate DeepEP GPU Communication Performance on Crusoe Cloud
• Resolving MOFED Storage Module Race Condition on CMK GPU Nodes
• DeepEP GitHub Repository