This article walks you through setting up Project-HAMi on Crusoe Managed Kubernetes (CMK) for L40s GPUs which natively do not support Multi-Instance GPU (MIG) technology.
The L40S-48GB.10x GPU type refers to a configuration of 10 NVIDIA L40S GPUs, each with 48 GB of GDDR6 memory, used for high-end AI and graphics workloads.
HAMi (Heterogeneous AI Computing Virtualization Middleware) is a Cloud Native Computing Foundation (CNCF) Sandbox project designed to enhance the management and utilization of heterogeneous AI computing resources—such as GPUs, NPUs, and MLUs—within Kubernetes clusters.
This open-source middleware enables fine-grained sharing, isolation, and scheduling of these devices, facilitating efficient resource allocation for AI workloads. HAMi's architecture includes a user-space CUDA API interception library, which allows for memory and compute limiting by tracking resource usage per process and adjusting GPU utilization accordingly. This approach provides soft isolation, ensuring that long-term usage remains within assigned quotas while allowing brief overages. By offering a unified interface for diverse hardware, HAMi simplifies the deployment and management of AI applications in cloud-native environments, making it a valuable tool for organizations seeking to optimize their AI infrastructure.
Pre-requirements
Access to Crusoe CLI or console
Crusoe Managed Kubernetes cluster
kubectlinstalled with default config at~/.kube/config
Set up Project-HAMi on CMK
Step 1: Cluster & Node Pool Creation
1. Crusoe Managed Kubernetes (CMK) simplifies running Kubernetes for AI workloads. It offers fully-managed control planes, which means CMK handles the availability, scaling, and lifecycle of the control plane nodes so you don't have to.
2. First, let's create a CMK Cluster and a Nodepool using the Crusoe CLI.
$ crusoe kubernetes clusters create \ --name project-hami \ --project-id 6d012e45-f991-47f6-aace-d8a61a7c32c7 \ --subnet-id 72a12f6d-388d-40d6-a4e7-456598af18f8 \ --location us-east1-a \ --cluster-version 1.30.8-cmk.31 \ --add-ons "nvidia_gpu_operator,nvidia_network_operator,crusoe_csi"
3. You can verify your cluster details with the following command:
$ crusoe kubernetes clusters get project-hami
4. The output confirms the cluster's state and configuration:
name: project-hami id: b5e7b3a4-ae41-4a17-bbb0-829b4ad29dcf configuration: ha state: STATE_RUNNING version: 1.30.8-cmk.31 location: us-east1-a ...
5. Now, create a nodepool with L40S GPUs type:
$ crusoe kubernetes nodepools create --name l40s-np --cluster-name project-hami --type l40s-48gb.10x --count 1
6. Once the node pool is created, you can fetch its details for reference:
$ crusoe kubernetes nodepools get l40s-np
7. The output shows the node pool is ready with the L40S GPU instance:
node pool name: l40s-np id: 2a0c12b2-2c60-4a05-99e2-2470a7391527 state: STATE_RUNNING type: l40s-48gb.10x desired count: 1 running count: 1 image: 1.30.8-cmk.31 ...
8. Confirm that the node is ready within Kubernetes itself:
$ kubectl get nodes NAME STATUS ROLES AGE VERSION np-2a0c12b2-1.us-east1-a.compute.internal Ready <none> 11m v1.30.8
9. Before installing HAMi, you can check the default resource allocation. The kubectl describe node command shows that Kubernetes sees each of your L40S GPUs as a single, indivisible resource.
$ kubectl describe node np-2a0c12b2-1.us-east1-a.compute.internal
10. The output confirms this:
Capacity: nvidia.com/gpu: 10
Notice that it shows 10 GPUs, which corresponds to the number of physical GPUs on your node.
Step 2: Install Project-HAMi with Helm
1. Next, install Project-HAMi onto your Kubernetes cluster using its official Helm chart. This is the simplest and most recommended way to get started.
2. First, add the HAMi Helm repository to your local configuration:
$ helm repo add hami-charts https://project-hami.github.io/HAMi/
3. Next, install the HAMi chart. This will deploy the necessary device plugin and scheduler components to your cluster:
$ helm install hami hami-charts/hami -n kube-system
4. You can verify that the HAMi pods are running by checking the pods in the kube-system namespace. You should see both the hami-device-plugin and hami-scheduler pods in a Running state.
$ kubectl get pods -A | grep hami kube-system hami-device-plugin-2hrrr 2/2 Running 0 103s kube-system hami-scheduler-7b47c4b59d-f9q6j 2/2 Running 1 (26m ago) 32m
Step 3: Label Your Node & Verify the Device Plugin's Registration
1. Now that HAMi is installed, you need to label your Kubernetes node. This tells the HAMi device plugin which nodes have GPUs that it needs to manage.
$ kubectl label node np-2a0c12b2-1.us-east1-a.compute.internal gpu=on
2. Project-HAMi uses a ConfigMap to define how it should virtualize your GPUs. The key parameter here is devicesplitcount, which determines how many virtual GPUs (vGPUs) each physical GPU will be split into.
3. You can check this value in the hami-device-plugin ConfigMap:
$ kubectl get cm hami-device-plugin -n kube-system -o yaml | grep devicesplitcount "devicesplitcount": 10,
4. With 10 physical L40S GPUs on the node, this configuration will create a total of 100 vGPU slots, which Kubernetes can now see and schedule workloads onto.
5. Once the node is labeled, the Project-HAMi device plugin automatically detects the L40S GPUs and registers each one with Kubernetes. You can view the logs of the device plugin pod to see this process in action:
$ kubectl logs hami-device-plugin-2hrrr -n kube-system
6. The logs confirm that each GPU has been registered with its ID, memory (46 GB), compute share, and the number of vGPU splits ("count": 10).
I0903 register.go:178] nvml registered device id=9, memory=46068, type=NVIDIA L40S, numa=0
I0903 register.go:185] "start working on the devices" devices=[{"id":"GPU-e3b6ed2f-1e7f-9e25-e27e-773fb898a0c8","index":2,"count":10,"devmem":46068,"devcore":100,"type":"NVIDIA-NVIDIA L40S","mode":"hami-core","health":true},{"id":"GPU-499ce651-c728-96df-5638-ebcb6f6ff610","index":3,"count":10,"devmem":46068,"devcore":100,"type":"NVIDIA-NVIDIA L40S","mode":"hami-core","health":true},{"id":"GPU-a27a5528-dab1-1b0b-7ca2-52db6c4fe664","index":4,"count":10,"devmem":46068,"devcore":100,"type":"NVIDIA-NVIDIA L40S","mode":"hami-core","health":true},{"id":"GPU-f5427d6e-f672-823f-cd5a-90c971e8e8c0","index":8,"count":10,"devmem":46068,"devcore":100,"type":"NVIDIA-NVIDIA L40S","mode":"hami-core","health":true},{"id":"GPU-b5f7218b-5977-3748-76e3-5b552554860c","count":10,"devmem":46068,"devcore":100,"type":"NVIDIA-NVIDIA L40S","mode":"hami-core","health":true},{"id":"GPU-021d64e1-12ae-4903-c516-d35e2f75873b","index":1,"count":10,"devmem":46068,"devcore":100,"type":"NVIDIA-NVIDIA L40S","mode":"hami-core","health":true},{"id":"GPU-4c51c114-25c1-9717-c70d-1801661eac4b","index":5,"count":10,"devmem":46068,"devcore":100,"type":"NVIDIA-NVIDIA L40S","mode":"hami-core","health":true},{"id":"GPU-2642de8d-246f-b5a5-e141-28704b5513fb","index":6,"count":10,"devmem":46068,"devcore":100,"type":"NVIDIA-NVIDIA L40S","mode":"hami-core","health":true},{"id":"GPU-a37c66f2-97a0-9749-92ee-8521c8ade90d","index":7,"count":10,"devmem":46068,"devcore":100,"type":"NVIDIA-NVIDIA L40S","mode":"hami-core","health":true},{"id":"GPU-1e0b76e8-b29e-a7e6-e2d6-bc677e55612f","index":9,"count":10,"devmem":46068,"devcore":100,"type":"NVIDIA-NVIDIA L40S","mode":"hami-core","health":true}]
I0903 util.go:65] "Fetching node" nodeName="np-2a0c12b2-1.us-east1-a.compute.internal"
I0903 util.go:81] "Successfully fetched node" nodeName="np-2a0c12b2-1.us-east1-a.compute.internal"
I0903 register.go:206] "patch nvidia topo score to node" hami.io/node-nvidia-score=""
I0903 register.go:212] patch node with the following annos map[hami.io/node-handshake:Reported 2025-09-03 06:17:12.898646268 +0000 UTC m=+73.213793128 hami.io/node-nvidia-register:GPU-e3b6ed2f-1e7f-9e25-e27e-773fb898a0c8,10,46068,100,NVIDIA-NVIDIA L40S,0,true,2,hami-core:GPU-499ce651-c728-96df-5638-ebcb6f6ff610,10,46068,100,NVIDIA-NVIDIA L40S,0,true,3,hami-core:GPU-a27a5528-dab1-1b0b-7ca2-52db6c4fe664,10,46068,100,NVIDIA-NVIDIA L40S,0,true,4,hami-core:GPU-f5427d6e-f672-823f-cd5a-90c971e8e8c0,10,46068,100,NVIDIA-NVIDIA L40S,0,true,8,hami-core:GPU-b5f7218b-5977-3748-76e3-5b552554860c,10,46068,100,NVIDIA-NVIDIA L40S,0,true,0,hami-core:GPU-021d64e1-12ae-4903-c516-d35e2f75873b,10,46068,100,NVIDIA-NVIDIA L40S,0,true,1,hami-core:GPU-4c51c114-25c1-9717-c70d-1801661eac4b,10,46068,100,NVIDIA-NVIDIA L40S,0,true,5,hami-core:GPU-2642de8d-246f-b5a5-e141-28704b5513fb,10,46068,100,NVIDIA-NVIDIA L40S,0,true,6,hami-core:GPU-a37c66f2-97a0-9749-92ee-8521c8ade90d,10,46068,100,NVIDIA-NVIDIA L40S,0,true,7,hami-core:GPU-1e0b76e8-b29e-a7e6-e2d6-bc677e55612f,10,46068,100,NVIDIA-NVIDIA L40S,0,true,9,hami-core:]
I0903 register.go:232] Successfully registered annotation. Next check in 30s seconds....Step 4: Confirm GPU Allocation in Kubernetes
1. The final step is to verify that Kubernetes recognizes the newly available vGPU resources. You can check the node's status using kubectl describe node.
2. You'll now see a new Capacity and Allocatable resource: nvidia.com/gpu. The value will correspond to the total number of vGPU slots created by the device plugin.
$ kubectl describe node np-2a0c12b2-1.us-east1-a.compute.internal
3. The output confirms that each GPU has been virtualized into 10 vGPU slots:
Capacity: ... nvidia.com/gpu: 100 ... Allocatable: ... nvidia.com/gpu: 100 ...
This confirms that Project-HAMi has successfully virtualized your L40S GPUs.
Real-World Use Cases and Examples
Now that your cluster is configured, let's look at a few examples of how you can use HAMi to share your L40S GPUs.
Example 1: Sharing One Physical GPU Among Multiple Pods
1. In this scenario, we'll run two pods (pod-a and pod-b) that share a single L40S GPU, while a third pod (pod-c) gets a dedicated GPU.
Here's the YAML for the pods:
apiVersion: v1
kind: Pod
metadata:
name: pod-a
spec:
containers:
- name: container
image: ubuntu:18.04
command: ["bash", "-c", "sleep 86400"]
resources:
limits:
nvidia.com/gpu: 1
nvidia.com/gpumem-percentage: 50
nvidia.com/gpucores: 50
env:
- name: NVIDIA_VISIBLE_DEVICES
value: "0"
---
apiVersion: v1
kind: Pod
metadata:
name: pod-b
spec:
containers:
- name: container
image: ubuntu:18.04
command: ["bash", "-c", "sleep 86400"]
resources:
limits:
nvidia.com/gpu: 1
nvidia.com/gpumem-percentage: 50
nvidia.com/gpucores: 50
env:
- name: NVIDIA_VISIBLE_DEVICES
value: "0"
---
apiVersion: v1
kind: Pod
metadata:
name: pod-c
spec:
containers:
- name: container
image: ubuntu:18.04
command: ["bash", "-c", "sleep 86400"]
resources:
limits:
nvidia.com/gpu: 1
nvidia.com/gpumem-percentage: 100
nvidia.com/gpucores: 100
env:
- name: NVIDIA_VISIBLE_DEVICES
value: "1"2. After deploying the pods, you can see them all running:
$ kubectl get pods NAME READY STATUS RESTARTS AGE pod-a 1/1 Running 0 8s pod-b 1/1 Running 0 8s pod-c 1/1 Running 0 8s
3. When you describe the node, Kubernetes itself only sees “1 GPU per pod” for each pod, so kubectl describe node shows 3 GPUs allocated, even though A & B share one.
Allocated resources: ... nvidia.com/gpu 3 3 ...
4. To confirm the physical allocation, you can check the GPU status from inside each pod using nvidia-smi -L.
$ kubectl exec -it pod-a -- nvidia-smi -L $ kubectl exec -it pod-b -- nvidia-smi -L $ kubectl exec -it pod-c -- nvidia-smi -L
5. The output from the first two pods confirms they are sharing the same physical GPU, while the third pod is on a separate one:
# pod-a output GPU 0: NVIDIA L40S (UUID: GPU-b5f7218b-5977-3748-76e3-5b552554860c) # pod-b output GPU 0: NVIDIA L40S (UUID: GPU-b5f7218b-5977-3748-76e3-5b552554860c) # pod-c output GPU 0: NVIDIA L40S (UUID: GPU-021d64e1-12ae-4903-c516-d35e2f75873b)
Conclusion: HAMi successfully enforces fractional GPU usage for pod-a and pod-b, preventing one from starving the other. Pod-c gets a full, dedicated GPU, demonstrating that HAMi supports both shared and exclusive allocation models.
Example 2: Requesting Multiple vGPUs for a Single Pod
In this example, a single pod requests two vGPU slots. HAMi may assign two slices from the same physical GPU or one slice from each of two GPUs, depending on availability.
Here is the YAML for a pod requesting two vGPUs:
apiVersion: v1
kind: Pod
metadata:
name: memory-fraction-gpu-pod
spec:
containers:
- name: ubuntu-container
image: ubuntu:18.04
command: ["bash", "-c", "sleep 86400"]
resources:
limits:
nvidia.com/gpu: 2
nvidia.com/gpumem-percentage: 25
nvidia.com/gpucores: 301. After deploying the pod, you can use nvidia-smi to see which GPUs were assigned:
$ kubectl exec -it memory-fraction-gpu-pod -- nvidia-smi
2. The output confirms that the pod received two separate GPU slices:
+-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA L40S On | 00000002:00:01.0 Off | 0 | | N/A 40C P8 36W / 350W | 0MiB / 11517MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA L40S On | 00000003:00:03.0 Off | 0 | | N/A 32C P8 36W / 350W | 0MiB / 11517MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
Conclusion: In this case, the pod received two separate physical GPUs, each with a slice. As shown by the nvidia-smi output, each slice provides 25% of the GPU's memory (roughly 11.5 GB), while HAMi enforces the 30% core limit via time-slicing.
This demonstrates how Project-HAMi enables flexible, safe GPU sharing, allowing you to run multiple workloads on a single GPU while still accommodating jobs that require dedicated, multi-GPU resources.