How-To Enable Node Pool Taints on CMK

Introduction

Node pool taints let you reserve specific CMK node pools for specific workloads. Pods without a matching toleration won't be scheduled onto tainted nodes, so you can dedicate a set of GPU nodes to a particular team or job type without other workloads landing on them.

Taints can be set when creating a node pool, or added to an existing node pool through an update, using the Console, CLI, or Terraform.

The first time you use taints on a given cluster, a one-time control-plane update may be required before the taints API will accept your request. Whether this applies to your cluster depends on when it was created. This guide walks through checking whether your cluster is ready, enabling it if it isn't, and setting taints once it is.

Prerequisites

• A Running CMK Cluster
• Ability to Schedule a Brief Maintenance Window on the Cluster (If Enablement Is Needed)

Instructions

Step 1: Check Whether Your Cluster Already Supports Taints

Attempt to set a taint on a node pool, either by creating a new one or updating an existing one (see Step 3 for the exact commands). This check happens before anything is created or changed, so it's a safe way to find out where your cluster stands:

• If it succeeds, your cluster is already enabled; skip to Step 3.
• If it fails with the following error, continue to Step 2:

Cluster does not yet support node taints. Please contact support to have node taints enabled for this cluster.

Step 2: Enable Taints on Your Cluster

1. Open a support ticket requesting node taints be enabled on your cluster. Include:
   • The cluster name(s). Taints need to be enabled per cluster, so list all clusters where you plan to use them.
   • Whether you're creating a new node pool with taints, or adding taints to an existing node pool.
   • Your preferred maintenance window (start time and time zone). 30 minutes per cluster is typically sufficient.
2. Crusoe Support performs a control-plane maintenance operation on your cluster. This updates the control plane so system DaemonSets (GPU/RDMA drivers, storage plugins, monitoring agents) tolerate your taints going forward.

ℹ️ Note: Only the control plane is touched during this maintenance. Your worker nodes and running workloads are not restarted or disrupted, though there is a brief control-plane availability pause as each control-plane node is updated in turn.

3. Once complete, you can create or update node pools with taints on that cluster.

Step 3: Set Taints on a Node Pool

On a new node pool:

crusoe kubernetes nodepools create --name gpu-pool --cluster-id <YOUR_CLUSTER_ID> --type <YOUR_INSTANCE_TYPE> --count <YOUR_NODE_COUNT> --node-taints "dedicated=training:NoExecute"

On an existing node pool:

crusoe kubernetes nodepools update gpu-pool --node-taints "dedicated=training:NoExecute"

Clearing all taints:

crusoe kubernetes nodepools update gpu-pool --node-taints ""

Terraform support is also available via the node_taints block on the crusoe_kubernetes_node_pool resource.

💡 Tip: A taint only takes effect on nodes created after you set it. A new node pool's nodes come up tainted immediately; an existing node pool's already-running nodes are not retroactively tainted. Only nodes added afterward, from a scale-up or replacement, pick it up. To taint an already-running node directly, run kubectl taint node <node-name> <key>=<value>:<effect>.

Example

You're running ML training workloads on a CMK cluster and want to dedicate a subset of GPU nodes to one team's jobs, keeping other workloads off them. You update your existing gpu-pool node pool:

crusoe kubernetes nodepools update gpu-pool --node-taints "dedicated=training:NoExecute"

The update succeeds immediately. Any node added to gpu-pool afterward, from a scale-up or a replacement, comes up with the taint applied, so only pods with a matching toleration land there. Nodes already running in the pool at the time of the update keep accepting general workloads until they're replaced or manually tainted with kubectl taint node.