Introduction
GPU time-slicing lets several pods share one physical GPU. Kubernetes advertises extra nvidia.com/gpu replicas, and the NVIDIA device plugin interleaves those pods on the same device.
This is the usual way to pack many small inference or CUDA jobs onto one GPU. It is not MIG. Replicas do not get isolated memory or fault domains. Every replica on a GPU sees the full device memory and competes for time.
On Crusoe Managed Kubernetes (CMK) you enable it through a ConfigMap and the GPU Operator ClusterPolicy. You do not reinstall the operator.
Verified on CMK with GPU Operator v25.10.1, device plugin v0.18.1, on h200-141gb-sxm-ib.8x (8 physical H200s and 32 allocatable GPUs with replicas: 4).
Prerequisites
- CMK Cluster with the NVIDIA GPU Operator Add-On Enabled
- At Least One Ready GPU Node with
nvidia.com/gpuAllocatable -
kubectlConfigured with the Cluster Kubeconfig - No GPU Jobs on the Target Nodes That Require Exclusive Access to a Full Device
⚠️ Warning: After you enable time-slicing, existing GPU jobs on affected nodes share the GPU with other replicas. Drain or finish jobs that need exclusive access first.
Instructions
Step 1: Confirm Exclusive GPUs Today
Check the node before you change anything. You should see the physical GPU count and replicas=1.
kubectl get nodes
Example output:
NAME STATUS ROLES AGE VERSION np-4251d8c3-1.eu-iceland1-a.compute.internal Ready <none> 10h v1.35.5
kubectl get node <node-name> -o jsonpath='gpu={.status.allocatable.nvidia\.com/gpu} product={.metadata.labels.nvidia\.com/gpu\.product} replicas={.metadata.labels.nvidia\.com/gpu\.replicas} strategy={.metadata.labels.nvidia\.com/gpu\.sharing-strategy} count={.metadata.labels.nvidia\.com/gpu\.count}{"\n"}'Example output:
gpu=8 product=NVIDIA-H200 replicas=1 strategy= count=8
ℹ️ Note: If you already see
gpu=32andNVIDIA-H200-SHARED, time-slicing is already enabled. Go to Step 4: Confirm the Node Is Shared.
Step 2: Create the Time-Slicing ConfigMap
replicas: 4 is the oversubscribe factor. It does not add compute — it only lets more pods schedule.
apiVersion: v1
kind: ConfigMap
metadata:
name: time-slicing-config-all
namespace: nvidia-gpu-operator
data:
any: |-
version: v1
flags:
migStrategy: none
sharing:
timeSlicing:
renameByDefault: false
failRequestsGreaterThanOne: false
resources:
- name: nvidia.com/gpu
replicas: 4⚠️ Warning: Create the ConfigMap in
nvidia-gpu-operator. Do not use thegpu-operatornamespace from the NVIDIA Helm examples — the device plugin never picks it up there.
kubectl apply -f time-slicing-config-all.yaml
Example output:
configmap/time-slicing-config-all created
Step 3: Point ClusterPolicy at the ConfigMap
kubectl patch clusterpolicies.nvidia.com/cluster-policy \
--type merge \
-p '{"spec": {"devicePlugin": {"config": {"name": "time-slicing-config-all", "default": "any"}}}}'Example output:
clusterpolicy.nvidia.com/cluster-policy patched
default: any applies the any block to every GPU node. The device plugin and GPU Feature Discovery pods restart. Wait until they are Running:
NAME READY STATUS RESTARTS AGE gpu-feature-discovery-89wdz 2/2 Running 0 7s nvidia-device-plugin-daemonset-24n27 2/2 Running 0 7s
⚠️ Warning: Do not Helm-install a second GPU Operator. On CMK the add-on already owns
ClusterPolicy.
Step 4: Confirm the Node Is Shared
Run the node query from Step 1 again. This is the after state:
gpu=32 product=NVIDIA-H200-SHARED replicas=4 strategy=time-slicing count=8
ℹ️ Note:
gpu=32is 8 physical H200s × 4 replicas.count=8is the real card count. The extra 24 slots are not extra hardware — they are extra scheduling slots on the same 8 GPUs.NVIDIA-H200-SHAREDis how GPU Feature Discovery marks a time-sliced node.
Allocatable nvidia.com/gpu can update a few seconds before the labels. Wait until both match.
Step 5: Run More Pods Than Physical GPUs
A request is still nvidia.com/gpu: 1. The difference is that more pods than gpu.count can be Running at once.
apiVersion: apps/v1
kind: Deployment
metadata:
name: time-slicing-verification
spec:
replicas: 10
selector:
matchLabels:
app: time-slicing-verification
template:
metadata:
labels:
app: time-slicing-verification
spec:
containers:
- name: cuda
image: nvidia/cuda:12.4.1-base-ubuntu22.04
command: ["bash", "-c", "nvidia-smi -L; sleep 3600"]
resources:
limits:
nvidia.com/gpu: 1kubectl apply -f time-slicing-verification.yaml kubectl get pods -l app=time-slicing-verification
Example output:
NAME READY STATUS RESTARTS AGE time-slicing-verification-649796ddd9-2qpdv 1/1 Running 0 8s time-slicing-verification-649796ddd9-5fpfw 1/1 Running 0 8s time-slicing-verification-649796ddd9-5q7hq 1/1 Running 0 8s time-slicing-verification-649796ddd9-j728s 1/1 Running 0 8s time-slicing-verification-649796ddd9-mzzpc 1/1 Running 0 8s time-slicing-verification-649796ddd9-n6gml 1/1 Running 0 8s time-slicing-verification-649796ddd9-p4b8k 1/1 Running 0 8s time-slicing-verification-649796ddd9-rhtb8 1/1 Running 0 8s time-slicing-verification-649796ddd9-sfbgj 1/1 Running 0 8s time-slicing-verification-649796ddd9-tcfms 1/1 Running 0 8s
All 10 pods are Running on 8 physical GPUs. Without time-slicing, two of these would stay Pending.
kubectl logs <pod-name>
Example output:
GPU 0: NVIDIA H200 (UUID: GPU-d3227a33-fab6-5786-e04f-61d87828c3e0)
Delete the test deployment when you are done:
kubectl delete deploy time-slicing-verification
Production workloads request GPUs the same way: resources.limits.nvidia.com/gpu: 1.
Step 6: Disable Time-Slicing
Finish or drain GPU jobs on the node first. Then clear the device plugin config and delete the ConfigMap:
kubectl patch clusterpolicies.nvidia.com/cluster-policy \
--type json \
-p '[{"op":"remove","path":"/spec/devicePlugin/config"}]'
kubectl delete configmap time-slicing-config-all -n nvidia-gpu-operatorWait for the device plugin to restart, then repeat the node query from Step 1. Allocatable nvidia.com/gpu returns to the physical count (gpu=8) and the product label loses -SHARED.
Example
You have an 8× H200 node and several small CUDA or inference jobs that each need a GPU but not the full 141 GB. Before time-slicing, Kubernetes advertises 8 GPUs, so a ninth job stays Pending.
After you apply the ConfigMap and patch ClusterPolicy with replicas: 4:
- Allocatable
nvidia.com/gpubecomes32(8 physical GPUs × 4 replicas) - Node labels show
nvidia.com/gpu.product=NVIDIA-H200-SHARED,nvidia.com/gpu.replicas=4, andnvidia.com/gpu.sharing-strategy=time-slicing - A 10-replica Deployment, each requesting
nvidia.com/gpu: 1, all become Running -
nvidia-smi -Linside each pod reports one NVIDIA H200
The same node still has only 8 physical GPUs. The extra 24 slots are shared time on those devices.
Common Issues
ConfigMap Created in the Wrong Namespace
NVIDIA docs use gpu-operator. On CMK the namespace is nvidia-gpu-operator. If the ConfigMap is in another namespace, the device plugin never picks it up and allocatable GPUs stay at the physical count.
Allocatable Is 32 but Labels Still Show replicas=1
GPU Feature Discovery can lag the device plugin by a few seconds. Re-run the label command. If sharing-strategy=time-slicing never appears, check that ClusterPolicy has spec.devicePlugin.config.default: any and that the GFD pods restarted.
Pods OOM or Interfere with Each Other
Time-slicing does not isolate memory. Two large models on the same GPU can exhaust device memory. Use MIG when you need hardware isolation, or lower replicas and keep large jobs on exclusive GPUs.
Requesting nvidia.com/gpu: 2 Does Not Double Compute
The pod is still placed on a shared GPU. It does not get twice the time. Leave failRequestsGreaterThanOne: false unless you want those requests rejected.
Only Some Nodes Should Be Shared
Omit devicePlugin.config.default from the patch and label those nodes:
kubectl label node <node-name> nvidia.com/device-plugin.config=any