How-To Fix CMK Nodes Missing NVMe Containerd Storage After Autoscaler Resets Node Pool Settings

Introduction

If you enabled Ephemeral Storage for Containerd on a CMK node pool and later find that some nodes have /var/lib/containerd sitting on the boot disk instead of the NVMe, this is probably why.

Older versions of the CMK Cluster Autoscaler add-on (anything before crusoe-cluster-autoscaler-1.30.3-crusoe.10) have a bug where every scale operation sends the full node pool config back to the API with the ephemeral-storage flag set to false and node_labels empty. That mints a new instance template, and any node created after that point boots without the NVMe mount and without your custom labels. Your existing nodes are untouched, so the pool ends up half right, half wrong, and nobody notices until pods start getting evicted for ephemeral-storage or a nodeSelector stops matching.

This article walks through confirming that is what happened, upgrading or pausing the autoscaler so it stops happening, fixing the pool settings, and recycling the bad nodes.

ℹ️ Note: The same setting has two names depending on where you look. Terraform and the Console call it ephemeral_storage_for_containerd; the API and crusoe kubernetes nodepools list report it as use_local_disk_for_containerd. Both appear in this article, in their own contexts.

Prerequisites

Running CMK Cluster With at Least One Node Pool
Crusoe CLI Installed and Authenticated
kubectl Access With Permission to Edit Deployments in crusoe-system (Autoscaler Add-on Only)
SSH or Privileged Pod Access to Worker Nodes for Disk Inspection

Instructions

Step 1: Confirm the Pool Is Split Across Instance Templates

List the node pool in JSON and look at two things — the storage flag, and the using_latest_config value on each instance:

crusoe kubernetes nodepools list -f json

"use_local_disk_for_containerd": false,
"node_labels": {
    "crusoe.ai/nodepool.name": "h200"
},
"instance_details": [
    "{id: <VM_ID_1>, using_latest_config: false}",
    "{id: <VM_ID_2>, using_latest_config: false}",
    "{id: <VM_ID_3>, using_latest_config: true}",
    "{id: <VM_ID_4>, using_latest_config: true}"
]

A mix of true and false means nodes were built from different templates. Which set is the bad one depends on what the pool currently says:

If use_local_disk_for_containerd reads false or your custom labels are missing, the pool itself is still wrong. Nodes with using_latest_config: true are the affected ones, and you must fix the pool in Step 4 before recycling anything.
If it reads true and your labels are present, the pool has already been corrected, or the bad template was later superseded. Nodes with using_latest_config: false are the affected ones, and only they need recycling.

Cross-check with the instance templates for the pool. Multiple hidden templates named np-<pool-id>, np-<pool-id>-v2, and so on confirm that the pool has been updated more than once:

crusoe compute instance-templates list

Step 2: Verify the Disk Layout on a Suspect Node

On a node from the affected set you identified in Step 1 — or any node showing evictions — check where containerd actually lives:

ls -la /var/lib/containerd
findmnt /mnt/nvme
lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT

Healthy node:

/var/lib/containerd -> /mnt/nvme/containerd
/mnt/nvme  /dev/nvme0n1  ext4  1.8T

Affected node:

/var/lib/containerd            <- real directory, no symlink
(findmnt returns nothing for /mnt/nvme)
nvme0n1  1.8T                  <- present, no FSTYPE, no MOUNTPOINT

If /var/lib/containerd is a plain directory and the NVMe devices show no filesystem, the node booted from a template with the storage flag disabled. Confirm from the pool side with the labels as well:

kubectl get nodes -L <YOUR_CUSTOM_LABEL>

Nodes missing the label value were built from the wiped template.

Step 3: Identify the Trigger and Stop It Recurring

If the Cluster Autoscaler add-on is installed, check its image tag:

kubectl -n crusoe-system get deployment cluster-autoscaler \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Any tag older than crusoe-cluster-autoscaler-1.30.3-crusoe.10 carries the bug. You have two options.

Option A — upgrade the autoscaler image (recommended):

kubectl -n crusoe-system set image deployment/cluster-autoscaler \
  cluster-autoscaler=ghcr.io/crusoecloud/cluster-autoscaler:crusoe-cluster-autoscaler-1.30.3-crusoe.10

ℹ️ Note: Confirm the exact image repository and tag against the release page before applying — the registry path may differ from the example above.

Option B — scale the autoscaler to zero while you fix the pool:

kubectl -n crusoe-system scale deployment cluster-autoscaler --replicas=0

Use this if you cannot upgrade immediately, or if you only need a statically sized pool. Scale it back to 1 only after upgrading.

If the autoscaler is not installed or is already on a fixed release, the template was regenerated by a manual node-pool update. Check the Console audit log for UpdateNodePool events around the time the first bad node appeared, and make sure the Ephemeral Storage for Containerd toggle is on before saving any future edit to that pool.

Step 4: Restore the Node Pool Configuration

Skip this step if Step 1 showed the flag true and your labels intact.

Otherwise, re-enable the storage flag and re-apply your labels through the Console (Kubernetes → Clusters → node pool → Edit) or Terraform:

resource "crusoe_kubernetes_node_pool" "gpu" {
  # ...
  ephemeral_storage_for_containerd = true
  node_labels = {
    "<YOUR_CUSTOM_LABEL>" = "<VALUE>"
  }
}

This mints a fresh, correct template. Re-run the nodepools list -f json command from Step 1 and confirm use_local_disk_for_containerd — the API's name for the same flag — is now true, and that node_labels contains everything you expect.

⚠️ Warning: Do this after the autoscaler is upgraded or scaled down. If a buggy autoscaler is still running, its next resize call will overwrite your fix within minutes.

Step 5: Recycle the Affected Nodes

Existing nodes never pick up a new template. A stop/start or reset does not re-run bootstrap either. The only path to the correct layout is a fresh VM:

kubectl cordon <NODE_NAME>
kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data
crusoe compute vms delete <VM_ID>

The node pool's instance group notices the shortfall and provisions a replacement from the current template. Wait for the new node to reach Ready, then repeat Step 2 on it before moving to the next batch.

Work in batches sized to your workload's disruption budget — five at a time is a reasonable default for GPU pools. When every instance shows using_latest_config: true and Step 2 passes on a sample, the pool is consistent again.

⚠️ Warning: Deleting a VM destroys anything on its local NVMe. Confirm nothing on the affected nodes depends on /mnt/nvme for data you cannot regenerate. On affected nodes the NVMe is unmounted anyway, but check that /var/lib/containerd on the boot disk is not being used for anything beyond image cache.

Example

An inference team runs a four-node H200 pool with Ephemeral Storage for Containerd enabled and the Cluster Autoscaler add-on pinned to min=max=4. Two nodes fail hardware health checks and are replaced automatically. A week later, large model images start failing to pull on exactly those two nodes with no space left on device, while the other two are fine.

crusoe kubernetes nodepools list -f json shows the pool with use_local_disk_for_containerd: false, the two new VMs with using_latest_config: true, the originals with false, and two instance templates for the pool. On one of the new nodes /var/lib/containerd is a real directory on /dev/vda1 and nvme0n1 has no filesystem. The autoscaler image is on an early 1.30.3 tag.

The team upgrades the autoscaler to crusoe.10, re-enables the storage flag on the pool and confirms it now reads true, then cordons, drains, and deletes the two bad VMs one at a time. Both replacements come back with /var/lib/containerd -> /mnt/nvme/containerd and the pool is symmetric again.

Was this article helpful?

0 out of 0 found this helpful

Comments

0 comments

Article is closed for comments.