Share GPUs between workloads with GPU fractioning on Crusoe Managed Kubernetes

Run several pods on one GPU, each with a hard GPU-memory limit enforced by the NVIDIA driver, using KAI Scheduler and gpu-fractioning on Crusoe Managed Kubernetes (CMK).

Good fits: development workspaces and notebooks, lightweight or bursty inference services, CI jobs that need a little GPU. Not a fit: workloads that need a whole GPU's compute or memory.

What you get

• Pods request GPU memory (16Gi), not whole GPUs. KAI packs them onto shared GPUs.
• The driver enforces each container's limit. Going over it raises a normal CUDA out-of-memory error in that container only; neighbours are unaffected.
• Inside the container, CUDA reports the limit as the GPU's total memory, so frameworks such as PyTorch and vLLM size themselves to their share automatically.
• Two compute modes per container: time-slicing (default) or sm-sharing (concurrent, via MPS).

Note — as of 11/03/26, NVIDIA driver r615 and nvidia-smi. GPU fractioning requires driver r615 or newer. In r615, the nvidia-smi header reads KMD Version and CUDA UMD Version instead of Driver Version and CUDA Version. Update any scripts or health checks that parse that header; nvidia-smi --query-gpu=driver_version --format=csv,noheader is stable across versions.

Prerequisites

• A CMK cluster with GPU node pools on a node pool version whose image runs containerd 2.x with NRI enabled (verified: 1.35.5-cmk.42).
• Do not enable the nvidia_gpu_operator add-on on this cluster; you will install the NVIDIA GPU Operator yourself with the r615 driver.
• kubectl and helm 3.x with cluster-admin access.

Step 1 — Install the NVIDIA GPU Operator with driver r615

helm upgrade -i gpu-operator gpu-operator \
  --repo https://helm.ngc.nvidia.com/nvidia --version v26.7.1 \
  -n gpu-operator --create-namespace \
  --set driver.version=615.71.09 \
  --wait --timeout 20m
kubectl get clusterpolicies.nvidia.com cluster-policy -o jsonpath='{.status.state}'   # ready

GPU Operator v26.7.1 already ships the container toolkit (v1.20.1) and device plugin (v0.20.1) that GPU fractioning needs; only the driver version has to be set. The first install takes about 10–15 minutes while it builds and loads the driver.

Step 2 — Raise the MPS daemon memory limit (8-GPU nodes)

gpu-fractioning 0.1.8 limits its MPS daemon container to 256 MiB of memory, which is not enough for sm-sharing on 8-GPU nodes. Until this is configurable upstream, raise it with a Kyverno mutating policy (skip this step if you only use time-slicing):

helm upgrade -i kyverno kyverno --repo https://kyverno.github.io/kyverno/ \
  --version 3.9.2 -n kyverno --create-namespace --wait

kubectl apply -f - <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: gpu-fractioning-mpsd-memory
spec:
  failurePolicy: Ignore
  autogen:
    podControllers:
      controllers: []
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE"]
      resources: ["pods"]
    namespaceSelector:
      matchLabels: {kubernetes.io/metadata.name: gpu-fractioning}
    objectSelector:
      matchLabels: {app.kubernetes.io/component: mpsd}
  mutations:
  - patchType: ApplyConfiguration
    applyConfiguration:
      expression: >-
        Object{spec: Object.spec{containers: [Object.spec.containers{
          name: "mpsd",
          resources: Object.spec.containers.resources{limits: {"memory": "2Gi"}}}]}}
EOF

Install the policy before Step 3 so the daemon starts with the higher limit.

Step 3 — Install gpu-fractioning

helm upgrade -i gpu-fractioning oci://ghcr.io/kai-scheduler/gpu-fractioning \
  --version 0.1.8 -n gpu-fractioning --create-namespace --wait
kubectl wait gpufractioningconfig/default --for=condition=Ready --timeout=10m

Step 4 — Install KAI Scheduler

helm upgrade -i kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \
  --version v0.18.7 -n kai-scheduler --create-namespace \
  --set global.gpuSharingMode=NvFractions \
  --set global.nvFractions.set=false \
  --wait

Install KAI after GPU-fractioning. If you install it first, restart the KAI operator afterwards.

Step 5 — Give the default queue capacity

KAI's default queue starts with zero quota, so Deployments stay Pending with NonPreemptibleOverQuota. To get started, make it unlimited:

for q in default-parent-queue default-queue; do
  kubectl patch queue $q --type=merge \
    -p '{"spec":{"resources":{"cpu":{"quota":-1},"memory":{"quota":-1},"gpu":{"quota":-1}}}}'
done

For teams, create separate queues with quotas (CPU in millicores, memory in MB, GPU in GPUs) — see the KAI queues documentation. Edits apply live with kubectl apply.

Step 6 — Verify

kubectl get gpufractioningconfig default \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {end}'
# FractiondReady=True MpsdReady=True Ready=True ...

# sm-sharing: the shared MPS server must have started on every GPU
kubectl -n gpu-fractioning logs -l app.kubernetes.io/component=mpsd --tail=-1 \
  | grep -E "Server shared (ERROR|INFO.*Creating server context)"

Expect one Creating server context on device N line per GPU and no ERROR.

Step 7 — Run a fractional workload

apiVersion: v1
kind: Pod
metadata:
  name: fractional-demo
  labels:
    kai.scheduler/queue: default-queue
  annotations:
    nvidia.com/container.app.gpu-memory.request: 16Gi   # guaranteed, used for scheduling
    nvidia.com/container.app.gpu-memory.limit: 24Gi     # optional hard cap; defaults to request
    nvidia.com/container.app.gpu-compute.mode: time-slicing   # or sm-sharing
spec:
  schedulerName: kai-scheduler
  restartPolicy: Never
  containers:
  - name: app                  # must match the name in the annotation keys
    image: pytorch/pytorch:2.14.1-cuda13.0-cudnn9-runtime
    command: ["python", "-c", "import torch; print(torch.cuda.mem_get_info())"]

The output shows a total of 24 GiB — the container's limit. nvidia-smi inside the container also reports the limit as the GPU's total memory.

Rules:

• Do not request nvidia.com/gpu in a fractional pod.
• One fractional container per pod. All processes share the limit in that container.
• Use absolute sizes (Gi) so the same manifest behaves the same on different GPU models.
• On Blackwell GPUs (B200/B300), use images built for CUDA 12.8 or newer.

Choosing a compute mode

Containers using different modes are never placed on the same GPU.

Using GPU fractioning alongside InfiniBand workloads

GPU fractioning works on nodes that also run the NVIDIA Network Operator for InfiniBand. Multi-node training jobs that request whole GPUs (nvidia.com/gpu) and InfiniBand devices can run on the same nodes as fractional pods: KAI keeps shared GPUs out of whole-GPU allocations, and in our tests neither workload affected the other's performance. Schedule those jobs with schedulerName: kai-scheduler as well.

The NVIDIA Network Operator runs side by side with a GPU Operator on driver r615. In our validation on H100 nodes (NVIDIA GPU Operator v26.7.1 with driver 615.71.09, NVIDIA Network Operator 26.1.0 with the DOCA driver), installing the Network Operator left the GPU driver and GPU fractioning untouched. Multi-node NCCL jobs used GPUDirect RDMA over InfiniBand at full bandwidth, whether or not fractional pods were running on the same nodes. Keep the GPU Operator defaults for this setup: open kernel modules and driver.rdma.enabled=false. GPUDirect RDMA then works through DMA-BUF and does not need the nvidia-peermem module. Installing the Network Operator reloads the InfiniBand drivers on each node, which briefly interrupts node networking, so add it to a running cluster during a maintenance window.

Troubleshooting

Clean up

helm uninstall kai-scheduler -n kai-scheduler
helm uninstall gpu-fractioning -n gpu-fractioning
kubectl delete mutatingpolicy gpu-fractioning-mpsd-memory && helm uninstall kyverno -n kyverno
helm uninstall gpu-operator -n gpu-operator