How-To Use NVIDIA Multi-Instance GPU to Run Training and Inference on H100 Instance on Crusoe Managed Kubernetes

Last Updated: Jan 15, 2026

Introduction

This article provides instructions on using NVIDIA Multi-Instance GPU to run training and inference on H100 instance on Crusoe Managed Kubernetes.

Multi-Instance GPU (MIG) maximizes the utility of NVIDIA Ampere, Blackwell and Hopper GPUs by partitioning them into as many as seven fully isolated instances. Each instance retains its own dedicated high-bandwidth memory, cache, and compute cores. This hardware-level isolation provides a guaranteed quality of service (QoS), enabling administrators to support workloads of all sizes.

With a single H100, you can't run a big training job and multiple inference models efficiently at the same time without MIG. They would compete for the same resources, causing the inference requests to lag (bad QoS) and cause the training job to slow down.

MIG solves this by letting you carve the H100 into hardware-isolated "GPU Instances" (GIs). Each instance has its own dedicated compute, memory, and cache, so they cannot interfere with each other.

Prerequisites

• Access to Kubeconfig for the CMK cluster with nodepool created for MIG supported SKUs.
• Ensure that the CMK cluster was created with addons nvidia_gpu_operator & nvidia_network_operator.
• kubectl installed and configured to interact with the cluster.

Step-by-Step Instructions

1. Update the ClusterPolicy Strategy to 'mixed'

By default, the ClusterPolicy is set to single, which applies one MIG configuration across all nodes. We'll change it to mixed to allow different nodes (or node groups) to have different MIG profiles.

Check the current strategy:

$ kubectl get clusterpolicy cluster-policy -n nvidia-gpu-operator -o yaml | grep stra
tegy
    strategy: single

Patch the ClusterPolicy to set the mig.strategy to mixed:

$ kubectl patch clusterpolicy cluster-policy -n nvidia-gpu-operator --type=merge -p '
spec:
  mig:
    strategy: mixed
'
clusterpolicy.nvidia.com/cluster-policy patched

2. Apply MIG Profile

An H100 80GB GPU can be partitioned into a maximum of 7 instances.

Available H100 80GB MIG Profiles:

• 1g.10gb (Smallest slice, ~1/7th compute and 10GB memory)

• 2g.20gb (Medium slice, ~2/7th compute and 20GB memory)

• 3g.40gb (Large slice, ~3/7th compute and 40GB memory)

• 4g.40gb (Large slice, ~4/7th compute and 40GB memory)

• 7g.80gb (The full GPU, but addressed as a single MIG instance)

Note: Other SKU specific profiles can be found here MIG-Profile.

The NVIDIA GPU Operator comes with a ConfigMap (e.g., default-mig-parted-config) that contains pre-defined profiles. These profiles are referred to as all-balanced, all-disabled. They are shortcuts to apply a specific combination of the hardware profiles listed above.

Common profiles in the ConfigMap include:

• all-disabled: This is the default. It disables MIG on all GPUs, returning them to their normal, full-GPU state.

• all-1g.10gb: This profile partitions every GPU into seven 1g.10gb instances. This is ideal for maximizing concurrency for many small inference workloads.

• all-3g.40gb: This partitions every GPU into two 3g.40gb instances. This is good for running two medium-sized training or inference jobs per GPU.

• all-balanced: The profile we are using.

Note: The all-balanced profile for H100-80GB creates 1g.10gb: 2, 2g.20gb: 1, and 3g.40gb: 1 on each GPU.

This gives you on ALL 8 GPUs:

• 8× 3g.40gb slices (40GB each) - Perfect for training jobs
• 8× 2g.20gb slices (20GB each) - Good for medium inference
• 16× 1g.10gb slices (10GB each) - Perfect for small inference

We will now apply label nvidia.com/mig.config=all-balanced to the Kubernetes node object. The nvidia-mig-manager will detect this label and apply the corresponding profile.

$ kubectl label node np-80f1e779-1.us-southcentral1-a.compute.internal nvidia.com/mig.config=all-balanced --overwrite

node/np-80f1e779-1.us-southcentral1-a.compute.internal labeled

3. Monitor the MIG Reconfiguration

$ kubectl logs -n nvidia-gpu-operator -l app=nvidia-mig-manager -f                                                   

time="2025-11-11T23:16:45Z" level=debug msg="Walking MigConfig for (devices=all)"
time="2025-11-11T23:16:45Z" level=debug msg="  GPU 0: 0x233010DE"
time="2025-11-11T23:16:45Z" level=debug msg="    MIG capable: true\n"
time="2025-11-11T23:16:45Z" level=debug msg="    Updating MIG config: map[1g.10gb:2 2g.20gb:1 3g.40gb:1]"
time="2025-11-11T23:16:45Z" level=debug msg="  GPU 1: 0x233010DE"
time="2025-11-11T23:16:45Z" level=debug msg="    MIG capable: true\n"
time="2025-11-11T23:16:45Z" level=debug msg="    Updating MIG config: map[1g.10gb:2 2g.20gb:1 3g.40gb:1]"
...
(Logs repeat for GPU 2 through GPU 7)
...
time="2025-11-11T23:16:51Z" level=debug msg="Running apply-exit hook"
MIG configuration applied successfully
time="2025-11-11T23:16:51Z" level=info msg="Running nvidia-smi"

Tue Nov 11 23:16:51 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20             Driver Version: 570.133.20     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
...
(List of 8 NVIDIA H100 80GB GPUs, all with MIG M. = Enabled)
...
+-----------------------------------------------------------------------------------------+
| MIG devices:                                                                            |
+------------------+----------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                     Memory-Usage |        Vol|        Shared         |
...
|==================+==================================+===========+=======================|
|  0    1   0   0  |              44MiB / 40448MiB    | 60      0 |  3   0    3    0    3 |  <-- 3g.40gb
+------------------+----------------------------------+-----------+-----------------------+
|  0    5   0   1  |              29MiB / 20096MiB    | 32      0 |  2   0    2    0    2 |  <-- 2g.20gb
+------------------+----------------------------------+-----------+-----------------------+
|  0   13   0   2  |              15MiB /  9984MiB    | 16      0 |  1   0    1    0    1 |  <-- 1g.10gb
+------------------+----------------------------------+-----------+-----------------------+
|  0   14   0   3  |              15MiB /  9984MiB    | 16      0 |  1   0    1    0    1 |  <-- 1g.10gb
+------------------+----------------------------------+-----------+-----------------------+
|  1    2   0   0  |              44MiB / 40448MiB    | 60      0 |  3   0    3    0    3 |  <-- 3g.40gb
+------------------+----------------------------------+-----------+-----------------------+
...
(Table continues for all 32 MIG devices across 8 GPUs)
...

Workflow:

1. Label Trigger & ConfigMap Lookup: When you add the nvidia.com/mig.config=all-balanced label to the node, MIG Manager detects this change and looks up "all-balanced" in the default-mig-parted-config ConfigMap to find the profile (which GPUs to configure, what MIG profiles to create, and how many instances of each).

2. GPU Reconfiguration & Device Creation: MIG Manager then executes the configuration by: 
   (a) Shutting down all GPU client pods (device plugin, DCGM, etc.).
   (b) Use the mig-parted tool to enable MIG mode and create the GPU instances (GI) and compute instances (CI) on each physical GPU according to the profile specification.
   (c) Verifying the configuration with nvidia-smi 
   (d) Restarting the GPU client pods so the device plugin can discover and advertise the new MIG resources to Kubernetes.

Verification:

$ kubectl get node np-80f1e779-1.us-southcentral1-a.compute.internal \
-o jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}' 

success

$ kubectl describe node
...
Capacity:
  cpu:                     176
  ephemeral-storage:       129886128Ki
  hugepages-1Gi:           0
  hugepages-2Mi:           0
  memory:                  990732960Ki
  nvidia.com/gpu:          0
  nvidia.com/hostdev:      8
  nvidia.com/mig-1g.10gb:  16
  nvidia.com/mig-2g.20gb:  8
  nvidia.com/mig-3g.40gb:  8
  pods:                    110
Allocatable:
  cpu:                     176
  ephemeral-storage:       119703055367
  hugepages-1Gi:           0
  hugepages-2Mi:           0
  memory:                  990630560Ki
  nvidia.com/gpu:          0
  nvidia.com/hostdev:      8
  nvidia.com/mig-1g.10gb:  16
  nvidia.com/mig-2g.20gb:  8
  nvidia.com/mig-3g.40gb:  8
  pods:                    110

Notethat nvidia.com/gpu is now 0, and we instead have 16 1g.10gb, 8 2g.20gb, and 8 3g.40gb devices available for pods to request.

4. Deploy Mixed Workloads

YAML with training-deployment requesting a single replica requesting one large 3g.40gb slice & inference-deployment with four replicas, each requesting a small 1g.10gb slice.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-deployment
  labels:
    app: llm-inference
spec:
  replicas: 4
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      nodeSelector:
        nvidia.com/mig.capable: "true"
      containers:
      - name: inference-container
        image: vllm/vllm-openai:latest
        ports:
        - containerPort: 8000
        env:
        - name: MODEL_NAME
          value: "mistralai/Mistral-7B-Instruct-v0.1"
        resources:
          limits:
            nvidia.com/mig-1g.10gb: 1
---
apiVersion: v1
kind: Service
metadata:
  name: llm-inference-service
spec:
  selector:
    app: llm-inference
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: training-deployment
  labels:
    app: llm-training
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-training
  template:
    metadata:
      labels:
        app: llm-training
    spec:
      nodeSelector:
        nvidia.com/mig.capable: "true"
      containers:
      - name: training-container
        image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
        # This command allocates a large tensor (~12GB) on the GPU and holds it
        command: ["python", "-c"]
        args:
          - |
            import torch, time
            print('Allocating large tensor on GPU...', flush=True)
            t = torch.randn(4000, 4000, 200, device='cuda')
            print(f"Tensor allocated with size: {t.nelement() * t.element_size() / 1024**3:.2f} GB. Holding memory...", flush=True)
            while True:
              time.sleep(3600)
        resources:
          limits:
            nvidia.com/mig-3g.40gb: 1

Apply the YAML:

$ kubectl apply -f mixed-workload-deployment.yaml
deployment.apps/inference-deployment created
service/llm-inference-service created
deployment.apps/training-deployment created

$ kubectl get deploy
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
inference-deployment    4/4     4            4           26s
training-deployment     1/1     1            1           26s

5. Workloads in Action

$ kubectl logs deploy/training-deployment
Allocating large tensor on GPU...
Tensor allocated with size: 11.92 GB. Holding memory...

$ kubectl logs deploy/inference-deployment
Found 4 pods, using pod/inference-deployment-6675bd98cb-dsb72
...
(APIServer pid=1) INFO 11-11 16:34:07 [api_server.py:1912] 
Starting vLLM API server 0 on http://0.0.0.0:8000

Using nvidia-smi -L and nvidia-smi commands, we can see the five pods running concurrently. They are all on separate, hardware-isolated MIG devices.

SLICE     GPU   PID     PROCESS             MEM
3g.40gb   6     230907  python             12456MiB
1g.10gb   6     232581  VLLM::EngineCore    9952MiB
1g.10gb   7     232447  VLLM::EngineCore    9952MiB
1g.10gb   4     232442  VLLM::EngineCore    9952MiB
1g.10gb   5     232577  VLLM::EngineCore    9952MiB

6. Additional MIG Management Tasks

1. Dynamic Reconfiguration: MIG Manager supports dynamic reconfiguration of the MIG geometry. You can for example update GPUs on a node to the all-3g.40gb profile, using the single MIG strategy. This strategy applies the same named profile to all nodes that share the same mig.config label. Note that no active workloads consuming prior MIG profiles should be present.
2. Using Custom MIG Configurations: If one needs a custom profile that isn't included by default (default-mig-parted-config ConfigMap), you can provide your own configuration when installing or upgrading the GPU Operator and/or by creating a new ConfigMap object and patching the cluster-policy object with the same.
3. Disabling MIG on a Node: To remove all MIG partitions and return a node's GPUs to their default, full-GPU state, one can apply the all-disabled label. This will remove all MIG devices and re-advertise the full nvidia.com/gpu resources.

$ kubectl label nodes <node-name> nvidia.com/mig.config=all-disabled --overwrite

Additional Resources

• GPU Operator with MIG