How-To Isolate and Schedule GPU Resources on Kubernetes Using ComputeDomain on GB200s

Last Updated: Jan 23, 2026

Introduction

This guide demonstrates how to leverage NVIDIA's Dynamic Resource Allocation (DRA) and the ComputeDomain functionality to efficiently and reliably isolate GPU-accelerated workloads on GB200 Kubernetes clusters. 

In this example, we are going to define two ComputeDomain instances - allocating 3 nodes for a high-throughput job and 1 node for a smaller job - to enforce specific resource scheduling, prevent contention, and ensure diverse training pipelines are confined to their designated hardware pools for greater stability and Resource Efficiency.

Prerequisites

Access to Kubeconfig for Crusoe Managed Kubernetes Cluster with GB200 nodepool provisioned 
OR 
Access to Kubeconfig for RKE2 Cluster - GB200 NVL72 Rack on RKE2 Cluster

Step-by-Step Instructions

1. Verify if version 25.8.1 for nvidia-dra-driver-gpu helm resource is deployed.

$ helm ls -n nvidia-dra-driver-gpu
NAME                 	NAMESPACE            	REVISION STATUS  	    CHART                       	APP VERSION
nvidia-dra-driver-gpu	nvidia-dra-driver-gpu	1        deployed	nvidia-dra-driver-gpu-25.8.1	      25.8.1

2. Define ComputeDomain for Workload Isolation

Create two separate ComputeDomain resources.

• training-alpha-domain: Targets 3 nodes for the high-throughput alpha training job (using SGD).
• training-beta-domain: Targets 1 node for the smaller beta training job (using Adam).

# training-alpha-domain
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: training-alpha-domain
spec:
  numNodes: 0
  channel:
    resourceClaimTemplate:
      name: training-alpha-domain-channel # Associates a ResourceClaimTemplate

---
# training-beta-domain
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: training-beta-domain
  namespace: default
spec:
  numNodes: 0
  channel:
    resourceClaimTemplate:
      name: training-beta-domain-channel # Associates a ResourceClaimTemplate

3. Define training scripts as Kubernetes ConfigMap objects. 

# ConfigMap for the 'alpha' training script (uses SGD)
apiVersion: v1
kind: ConfigMap
metadata:
  name: train-alpha-configmap
data:
  train-alpha.py: |
    import torch
    import torch.nn as nn
    import torch.optim as optim

    x = torch.randn(10, 1)
    y = 3*x + 2

    model = nn.Linear(1, 1)
    optimizer = optim.SGD(model.parameters(), lr=0.01)
    loss_fn = nn.MSELoss()

    for i in range(100):
        pred = model(x)
        loss = loss_fn(pred, y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print("Training Alpha model finished. Model weights:", list(model.parameters()))

---
# ConfigMap for the 'beta' training script (uses Adam)
apiVersion: v1
kind: ConfigMap
metadata:
  name: train-beta-configmap
data:
  train-beta.py: |
    import torch
    import torch.nn as nn
    import torch.optim as optim

    x = torch.randn(20, 1)
    y = -2*x + 5

    model = nn.Linear(1, 1)
    optimizer = optim.Adam(model.parameters(), lr=0.01)
    loss_fn = nn.MSELoss()

    for i in range(50):
        pred = model(x)
        loss = loss_fn(pred, y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print("Training Beta model finished. Model weights:", list(model.parameters()))

4. Define Job resource with references to ResourceClaimTemplate to bind to intended ComputeDomains

# Job for the 'alpha' model training
apiVersion: batch/v1
kind: Job
metadata:
  name: train-alpha
spec:
  completions: 3
  parallelism: 3
  template:
    spec:
      restartPolicy: Never
      resourceClaims:
      - name: training-alpha-claim
        resourceClaimTemplateName: training-alpha-domain-channel # Binds to the 3-node domain
      containers:
      - name: trainer
        image: nvcr.io/nvidia/pytorch:24.01-py3
        command: ["python", "/scripts/train-alpha.py"]
        volumeMounts:
        - name: alpha-script
          mountPath: /scripts
        resources:
          claims:
          - name: training-alpha-claim
      volumes:
      - name: alpha-script
        configMap:
          name: train-alpha-configmap

---
# Job for the 'beta' model training
apiVersion: batch/v1
kind: Job
metadata:
  name: train-beta
spec:
  completions: 1
  parallelism: 1
  template:
    spec:
      restartPolicy: Never
      resourceClaims:
      - name: training-beta-claim
        resourceClaimTemplateName: training-beta-domain-channel # Binds to the 1-node domain
      containers:
      - name: trainer
        image: nvcr.io/nvidia/pytorch:24.01-py3
        command: ["python", "/scripts/train-beta.py"]
        volumeMounts:
        - name: beta-script
          mountPath: /scripts
        resources:
          claims:
          - name: training-beta-claim
      volumes:
      - name: beta-script
        configMap:
          name: train-beta-configmap

5. Apply the YAML files and use kubectl to verify the state of the allocated resources and pod placement.

$ kubectl apply -f alpha-beta-model-training.yaml
computedomain.resource.nvidia.com/training-alpha-domain created
computedomain.resource.nvidia.com/training-beta-domain created
# [...] configmaps and jobs created

6. Verify the Compute Domains are created and ready.

$ kubectl get computeDomain -A
NAMESPACE   NAME                    AGE
default     training-alpha-domain   26s
default     training-beta-domain    26s

7. resourceClaimTemplate are automatically generated by the ComputeDomains for the Jobs to consume.

$ kubectl get resourceClaimTemplate    
NAME                            AGE
training-alpha-domain-channel   30s
training-beta-domain-channel    30s

8. Confirm that the running Jobs successfully created and bound their individual, ephemeral ResourceClaim objects to the domains in the default namespace

Note on the nvidia-dra-driver-gpu namespace: The ResourceClaim objects visible in this namespace (e.g., training-alpha-domain-[...]-compute-domain-daemon-[...]) are internal claims created by the NVIDIA DRA Driver (Compute Domain Scheduler) itself. 

These claims are used to launch the special domain-specific daemon pods that manage the lifecycle and enforce the isolation for each ComputeDomain. These claims confirm that the underlying DRA driver has successfully taken ownership and control of the nodes designated by your ComputeDomain requests.

$ kubectl get resourceClaims -A
NAMESPACE               NAME                                                            STATE                AGE
default                 train-alpha-lmnfw-training-alpha-claim-jvqdk                    allocated,reserved   9s
default                 train-alpha-pbvzr-training-alpha-claim-k9bxh                    allocated,reserved   9s
default                 train-alpha-wlt4z-training-alpha-claim-zch4q                    allocated,reserved   9s
default                 train-beta-cjm5p-training-beta-claim-h4d5f                      allocated,reserved   9s
nvidia-dra-driver-gpu   training-alpha-domain-hwvkz-ds2fj-compute-domain-daemon-l4cdt   allocated,reserved   9s
nvidia-dra-driver-gpu   training-alpha-domain-hwvkz-k9gpn-compute-domain-daemon-psc6h   allocated,reserved   9s
nvidia-dra-driver-gpu   training-alpha-domain-hwvkz-zqwlv-compute-domain-daemon-qnpmd   allocated,reserved   9s
nvidia-dra-driver-gpu   training-beta-domain-4pr4v-8bp6l-compute-domain-daemon-qmpgj    allocated,reserved   9s

9. ComputeDomain describe output confirms which nodes were successfully added to each ComputeDomain per Job specification.

$ kubectl describe computeDomain training-alpha-domain
Name:         training-alpha-domain
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  resource.nvidia.com/v1beta1
Kind:         ComputeDomain

Spec:
  Channel:
    Allocation Mode:  Single
    Resource Claim Template:
      Name:   training-alpha-domain-channel
  Num Nodes:  0
Status:
  Nodes:
    Clique ID:   e3022c5d-44ac-481a-944b-9dfe52215eaa.32766
    Index:       0
    Ip Address:  10.234.1.60
    Name:        np-a6d1ff6a-1.eu-iceland1-a.compute.internal
    Status:      Ready
    Clique ID:   e3022c5d-44ac-481a-944b-9dfe52215eaa.32766
    Index:       1
    Ip Address:  10.234.2.157
    Name:        np-a6d1ff6a-4.eu-iceland1-a.compute.internal
    Status:      Ready
    Clique ID:   e3022c5d-44ac-481a-944b-9dfe52215eaa.32766
    Index:       2
    Ip Address:  10.234.3.76
    Name:        np-a6d1ff6a-2.eu-iceland1-a.compute.internal
    Status:      Ready
  Status:        Ready
Events:          <none>

---
$ kubectl describe computeDomain training-beta-domain
Name:         training-beta-domain
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  resource.nvidia.com/v1beta1
Kind:         ComputeDomain

Spec:
  Channel:
    Allocation Mode:  Single
    Resource Claim Template:
      Name:   training-beta-domain-channel
  Num Nodes:  0
Status:
  Nodes:
    Clique ID:   e3022c5d-44ac-481a-944b-9dfe52215eaa.32766
    Index:       0
    Ip Address:  10.234.0.164
    Name:        np-a6d1ff6a-3.eu-iceland1-a.compute.internal
    Status:      Ready
  Status:        Ready
Events:          <none>

10. Logs from completed pods to confirm the training executed successfully and produced model weights.

$ kubectl logs train-alpha-lmnfw
Training Alpha model finished. Model weights: [Parameter containing:
tensor([[1.4106]], requires_grad=True), Parameter containing:
tensor([1.1219], requires_grad=True)]

$ kubectl logs train-alpha-pbvzr
Training Alpha model finished. Model weights: [Parameter containing:
tensor([[2.8770]], requires_grad=True), Parameter containing:
tensor([1.7957], requires_grad=True)]

$ kubectl logs train-alpha-wlt4z
Training Alpha model finished. Model weights: [Parameter containing:
tensor([[2.7363]], requires_grad=True), Parameter containing:
tensor([1.6193], requires_grad=True)]

$ kubectl logs train-beta-cjm5p
Training Beta model finished. Model weights: [Parameter containing:
tensor([[0.2805]], requires_grad=True), Parameter containing:
tensor([0.0184], requires_grad=True)]

11. After the job completed, the ComputeDomain automatically released its previously allocated nodes, returning them to the available pool for future workloads.

$ kubectl describe computeDomain training-alpha-domain
Name:         training-alpha-domain
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  resource.nvidia.com/v1beta1
Kind:         ComputeDomain

Spec:
  Channel:
    Allocation Mode:  Single
    Resource Claim Template:
      Name:   training-alpha-domain-channel
  Num Nodes:  0
Status:
  Status:  Ready
Events:    <none>

---
$ kubectl describe computeDomain training-beta-domain
Name:         training-beta-domain
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  resource.nvidia.com/v1beta1
Kind:         ComputeDomain

Spec:
  Channel:
    Allocation Mode:  Single
    Resource Claim Template:
      Name:   training-beta-domain-channel
  Num Nodes:  0
Status:
  Status:  Ready
Events:    <none>

Additional Resources

• NVIDIA DRA Driver for GPUs
• Dynamic Resource Allocation