How-To Enable NVIDIA MPS GPU Sharing on Crusoe Managed Kubernetes

Introduction

NVIDIA Multi-Process Service (MPS) lets several pods share one physical GPU with enforced memory and compute limits. The NVIDIA device plugin starts an MPS control daemon on the node. Each replica gets an equal fraction of device memory and an equal share of compute (for replicas: 4, that is about 1/4 of memory and 25% of threads).

This is different from time-slicing. Time-slicing only oversubscribes scheduling slots — pods still see the full GPU and can OOM each other. MPS space-partitions the GPU. It is also not MIG: there is no hardware fault isolation, and you cannot combine MPS with MIG.

On Crusoe Managed Kubernetes (CMK) you enable it through a ConfigMap and the GPU Operator ClusterPolicy. You do not reinstall the operator. You cannot enable MPS and time-slicing on the same node at the same time.

Verified on CMK with GPU Operator v25.10.1, device plugin v0.18.1, on h200-141gb-sxm-ib.8x (8 physical H200s and 32 allocatable GPUs with replicas: 4). The MPS daemon set a pinned-memory limit of 35942 MiB per GPU and active_thread_percentage of 25.

Prerequisites

• CMK Cluster with the NVIDIA GPU Operator Add-On Enabled
• At Least One Ready GPU Node with nvidia.com/gpu Allocatable
• kubectl Configured with the Cluster Kubeconfig
• MIG Mode Disabled on the Node (MPS Is Not Supported with MIG Enabled)
• No GPU Jobs That Require Exclusive Access to a Full Device

Instructions

Step 1: Confirm Exclusive GPUs Today

Check the node before you change anything. You should see the physical GPU count and replicas=1, or time-slicing if you already followed How-To Enable GPU Time-Slicing on Crusoe Managed Kubernetes.

kubectl get nodes

Example output:

NAME                                           STATUS   ROLES    AGE   VERSION
np-4251d8c3-1.eu-iceland1-a.compute.internal   Ready    <none>   10h   v1.35.5

kubectl get node <node-name> -o jsonpath='gpu={.status.allocatable.nvidia\.com/gpu} product={.metadata.labels.nvidia\.com/gpu\.product} replicas={.metadata.labels.nvidia\.com/gpu\.replicas} strategy={.metadata.labels.nvidia\.com/gpu\.sharing-strategy} mps={.metadata.labels.nvidia\.com/mps\.capable}{"\n"}'

Example output (no sharing):

gpu=8 product=NVIDIA-H200 replicas=1 strategy= mps=false

ℹ️ Note: If you already see strategy=mps and mps=true, MPS is already enabled. Go to Step 4: Confirm the Node Is Using MPS.

Step 2: Create the MPS ConfigMap

replicas: 4 is the oversubscribe factor. Each client is limited to about 1 / replicas of memory and compute.

apiVersion: v1
kind: ConfigMap
metadata:
  name: mps-config-all
  namespace: nvidia-gpu-operator
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      mps:
        renameByDefault: false
        resources:
          - name: nvidia.com/gpu
            replicas: 4

⚠️ Warning: Create the ConfigMap in nvidia-gpu-operator. Do not use the gpu-operator namespace from the NVIDIA Helm examples — the device plugin never picks it up there.

kubectl apply -f mps-config-all.yaml

Example output:

configmap/mps-config-all created

Step 3: Point ClusterPolicy at the ConfigMap

kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  --type merge \
  -p '{"spec": {"devicePlugin": {"config": {"name": "mps-config-all", "default": "any"}}}}'

Example output:

clusterpolicy.nvidia.com/cluster-policy patched

default: any applies the any block to every GPU node. The device plugin and GPU Feature Discovery pods restart. GPU Feature Discovery then sets nvidia.com/mps.capable=true, and the MPS control DaemonSet schedules.

kubectl get pods -n nvidia-gpu-operator -l 'app in (nvidia-device-plugin-daemonset,gpu-feature-discovery,nvidia-device-plugin-mps-control-daemon)'

Example output:

NAME                                              READY   STATUS    RESTARTS   AGE
gpu-feature-discovery-wdhjb                       2/2     Running   0          40s
nvidia-device-plugin-daemonset-t57b9              2/2     Running   0          41s
nvidia-device-plugin-mps-control-daemon-zrk6k     2/2     Running   0          6s

⚠️ Warning: Do not Helm-install a second GPU Operator. On CMK the add-on already owns ClusterPolicy.

ℹ️ Note: Allocatable nvidia.com/gpu can drop to 0 for a short time while the plugin switches from exclusive or time-slicing to MPS. Wait until it returns.

Step 4: Confirm the Node Is Using MPS

kubectl get node <node-name> -o jsonpath='gpu={.status.allocatable.nvidia\.com/gpu} product={.metadata.labels.nvidia\.com/gpu\.product} replicas={.metadata.labels.nvidia\.com/gpu\.replicas} strategy={.metadata.labels.nvidia\.com/gpu\.sharing-strategy} mps={.metadata.labels.nvidia\.com/mps\.capable} count={.metadata.labels.nvidia\.com/gpu\.count}{"\n"}'

Example output:

gpu=32 product=NVIDIA-H200-SHARED replicas=4 strategy=mps mps=true count=8

ℹ️ Note: gpu=32 is 8 physical H200s × 4 replicas. count=8 is the real card count. strategy=mps (not time-slicing) and mps=true mean the MPS control daemon is in use.

Check the daemon applied the 1/4 limits:

kubectl logs -n nvidia-gpu-operator -l app=nvidia-device-plugin-mps-control-daemon -c mps-control-daemon-ctr --tail=30

Example output:

Cmd:set_default_device_pinned_mem_limit 0 35942M
Cmd:set_default_active_thread_percentage 25

35942 MiB is about one quarter of the 143771 MiB H200 memory. 25 is one quarter of the SMs.

Step 5: Run More Pods Than Physical GPUs

A request is still nvidia.com/gpu: 1. More pods than gpu.count can be Running at once.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mps-verification
spec:
  replicas: 10
  selector:
    matchLabels:
      app: mps-verification
  template:
    metadata:
      labels:
        app: mps-verification
    spec:
      containers:
        - name: cuda
          image: nvidia/cuda:12.4.1-base-ubuntu22.04
          command: ["bash", "-c", "nvidia-smi --query-gpu=name,memory.total --format=csv; sleep 3600"]
          resources:
            limits:
              nvidia.com/gpu: 1

kubectl apply -f mps-verification.yaml
kubectl get pods -l app=mps-verification

Example output:

NAME                                READY   STATUS    RESTARTS   AGE
mps-verification-6fdc8f5fd7-49zr5   1/1     Running   0          6s
mps-verification-6fdc8f5fd7-6fvvz   1/1     Running   0          6s
mps-verification-6fdc8f5fd7-6h7qm   1/1     Running   0          6s
mps-verification-6fdc8f5fd7-7lg6b   1/1     Running   0          6s
mps-verification-6fdc8f5fd7-drj7x   1/1     Running   0          6s
mps-verification-6fdc8f5fd7-k4rmn   1/1     Running   0          6s
mps-verification-6fdc8f5fd7-phq86   1/1     Running   0          6s
mps-verification-6fdc8f5fd7-rs6zw   1/1     Running   0          6s
mps-verification-6fdc8f5fd7-svh5d   1/1     Running   0          6s
mps-verification-6fdc8f5fd7-z49rc   1/1     Running   0          6s

All 10 pods are Running on 8 physical GPUs. Without MPS (or time-slicing), two of these would stay Pending.

kubectl logs <pod-name>

Example output:

name, memory.total [MiB]
NVIDIA H200, 143771 MiB

ℹ️ Note: nvidia-smi still reports the full device size. The MPS daemon, not nvidia-smi, enforces the per-client cap (35942 MiB and 25% threads in this test).

Delete the test deployment when you are done:

kubectl delete deploy mps-verification

Production workloads request GPUs the same way: resources.limits.nvidia.com/gpu: 1.

Step 6: Disable MPS

Finish or drain GPU jobs on the node first. Then clear the device plugin config and delete the ConfigMap:

kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  --type json \
  -p '[{"op":"remove","path":"/spec/devicePlugin/config"}]'

kubectl delete configmap mps-config-all -n nvidia-gpu-operator

💡 Tip: If you still have a time-slicing ConfigMap and want that instead, point ClusterPolicy at time-slicing-config-all rather than removing the config.

Wait for the device plugin to restart. Allocatable nvidia.com/gpu returns to the physical count, sharing-strategy is no longer mps, and nvidia.com/mps.capable is false. The MPS control DaemonSet goes back to DESIRED 0.

Example

You have an 8× H200 node and several inference jobs that should share a GPU without one job taking all the memory. Time-slicing would let them schedule, but any job could allocate the full 141 GB.

After you apply the MPS ConfigMap and patch ClusterPolicy with replicas: 4:

• Allocatable nvidia.com/gpu becomes 32 (8 physical GPUs × 4 replicas)
• Labels show sharing-strategy=mps and mps.capable=true
• nvidia-device-plugin-mps-control-daemon is Running
• The daemon sets ~35942 MiB pinned memory and 25% active threads per client
• A 10-replica Deployment, each requesting nvidia.com/gpu: 1, all become Running

Additional Resources

• NVIDIA Kubernetes device plugin — CUDA MPS
• GPU Sharing in Kubernetes (NVIDIA)

Related Articles

• How-To Enable GPU Time-Slicing on Crusoe Managed Kubernetes
• How-To Use NVIDIA Multi-Instance GPU to Run Training and Inference on H100 Instance on Crusoe Managed Kubernetes