How-To Automatically Taint CMK GPU Nodes Using a DaemonSet

Introduction

Crusoe Managed Kubernetes (CMK) does not apply the nvidia.com/gpu=present:NoSchedule taint to GPU nodes, and CMK node pools do not currently support defining taints at creation time (through the console, CLI, or Terraform provider — node pools support labels only).

In Kubernetes, a taint on a node repels pods that do not tolerate it. The nvidia.com/gpu=present:NoSchedule taint is a widely adopted convention that ensures only GPU-aware workloads land on GPU nodes. If you deploy your own NVIDIA GPU Operator, it expects this taint to be present and will not schedule its own components — or your GPU workloads — correctly without it.

This article shows how to deploy a DaemonSet that automatically taints GPU nodes, including nodes added later by scale-up or the cluster autoscaler. The DaemonSet uses an init container to apply the taint via kubectl, so no long-running process is required beyond a minimal pause container.

Prerequisites

A CMK Cluster With kubectl Access and Permissions to Create Namespaces, PriorityClasses, ClusterRoles, and DaemonSets
A Label That Identifies Your GPU Nodes

The most reliable option is to set a custom label on your GPU node pool at creation time, for example gpu-taint=true.

You can also select on a label already present on your GPU nodes, such as the nvidia.com/gpu.present=true label applied by Node Feature Discovery. However, this label only appears after NFD has run on the node, which widens the window before the taint is applied.

Instructions

Step 1: Create the Namespace, PriorityClass, and RBAC Resources

The DaemonSet runs in its own namespace. A dedicated PriorityClass ensures the tainter pod is scheduled promptly when a new node joins, minimizing the window before the taint is applied. The pod's ServiceAccount needs permission to patch its own Node object.

Save the following as gpu-taint-rbac.yaml.

apiVersion: v1
kind: Namespace
metadata:
  name: gpu-node-tainter
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-node-tainter
value: 1000000000
globalDefault: false
description: "High priority so the GPU node tainter is scheduled promptly on new nodes."
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gpu-node-tainter
  namespace: gpu-node-tainter
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: gpu-node-tainter
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gpu-node-tainter
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: gpu-node-tainter
subjects:
  - kind: ServiceAccount
    name: gpu-node-tainter
    namespace: gpu-node-tainter

Apply it:

$ kubectl apply -f gpu-taint-rbac.yaml

Step 2: Create the DaemonSet

The taint is applied by an init container, so no long-running process is needed for the actual work. The main container is the minimal pause image, which keeps the pod alive with negligible resource usage.

Save the following as gpu-taint-daemonset.yaml. Replace <YOUR_GPU_NODE_LABEL_KEY> and <YOUR_GPU_NODE_LABEL_VALUE> with the label that identifies your GPU nodes (for example gpu-taint / "true").

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: gpu-node-tainter
  namespace: gpu-node-tainter
spec:
  selector:
    matchLabels:
      app: gpu-node-tainter
  template:
    metadata:
      labels:
        app: gpu-node-tainter
    spec:
      serviceAccountName: gpu-node-tainter
      priorityClassName: gpu-node-tainter
      nodeSelector:
        <YOUR_GPU_NODE_LABEL_KEY>: "<YOUR_GPU_NODE_LABEL_VALUE>"
      tolerations:
        # Tolerate the taint this DaemonSet applies, so its own pods
        # can still be (re)scheduled on tainted nodes.
        - key: nvidia.com/gpu
          operator: Equal
          value: present
          effect: NoSchedule
      initContainers:
        - name: tainter
          image: bitnami/kubectl:1.32
          command:
            - /bin/sh
            - -c
            - |
              kubectl taint node "$NODE_NAME" nvidia.com/gpu=present:NoSchedule --overwrite
              echo "Taint applied to $NODE_NAME"
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
            limits:
              cpu: 50m
              memory: 64Mi
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
          resources:
            requests:
              cpu: 1m
              memory: 4Mi
            limits:
              cpu: 10m
              memory: 8Mi

Apply it:

$ kubectl apply -f gpu-taint-daemonset.yaml

ℹ️ Note: If the pod is ever restarted, the init container runs again. The --overwrite flag makes re-applying the taint a no-op.

💡 Tip: The bitnami/kubectl:1.32 image is pinned to a specific version. Replace 1.32 with the version of kubectl that matches your cluster. Avoid using :latest in production — it can introduce breaking changes on image pull.

Step 3: Verify the Taint

Confirm the taint is present on your GPU nodes (adjust the label selector to your GPU node label).

$ kubectl describe nodes -l <YOUR_GPU_NODE_LABEL_KEY>=<YOUR_GPU_NODE_LABEL_VALUE> | grep -E '^(Name|Taints):'

Expected output:

Name:               <Hostname>
Taints:             nvidia.com/gpu=present:NoSchedule

Example

Consider a CMK cluster with a GPU node pool labeled gpu-taint=true. You deploy the NVIDIA GPU Operator via Helm, which expects GPU nodes to carry the nvidia.com/gpu=present:NoSchedule taint. Without it, the operator's device plugin and driver containers may not schedule correctly, or non-GPU workloads may land on expensive GPU nodes.

After applying the DaemonSet from this article, every node in the gpu-taint=true pool — including nodes added later by the cluster autoscaler — receives the taint automatically. The DaemonSet schedules a pod on each matching node as soon as the node registers, the init container applies nvidia.com/gpu=present:NoSchedule, and workloads without the matching toleration are kept off the node from that point on. The GPU Operator can then manage the node as expected.

⚠️ Warning: There is a brief scheduling window after a new node registers and before the tainter pod starts. Pods without the toleration could be scheduled onto a fresh GPU node during this window. The NoSchedule effect does not evict pods that are already running.

Additional Information

Removing the Taint

Deleting the DaemonSet does not remove existing taints. To remove the taint from a node, run:

$ kubectl taint node <NODE_NAME> nvidia.com/gpu=present:NoSchedule-

Was this article helpful?

0 out of 0 found this helpful

Comments

0 comments

Article is closed for comments.