How-To Recover a CMK Node When a Full Boot Disk Blocks Access

Introduction

Kubernetes normally gives you several ways into an unhealthy node: kubectl exec into a pod already running there, a kubectl debug node pod, or plain SSH. A completely full boot disk kills all three at once.

When the root filesystem (nodefs) reaches 100% utilization, the container runtime cannot write the small scratch files it needs to launch any new process. kubectl exec and debug pods fail before your command ever runs.

If your SSH keys are distributed by a DaemonSet, that DaemonSet's pods have likely been evicted under DiskPressure, so new keys can never land on the node and you are locked out of the host as well.

This state is also self-perpetuating. If the data filling the disk was written outside pod-attributed storage — for example, through a hostPath volume that silently landed on the root disk — the kubelet's eviction manager cannot charge that usage to the offending workload. It ranks and evicts system DaemonSets (monitoring agents, log collectors, CSI driver pods) instead of the actual consumer, and the node never self-heals.

This article explains how this condition arises, how to confirm you are in it without node access, and how to recover.

Prerequisites

CMK Cluster With kubectl Access
Crusoe CLI Installed and Authenticated
Node Pool Configured to Automatically Replace Deleted Nodes (VM Replacement Path Only)

Instructions

Step 1: Confirm the Node Is in an Unrecoverable-Access State

Check the node's conditions:

kubectl get node <NODE_NAME> \
  -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,DISKPRESSURE:.status.conditions[?(@.type=="DiskPressure")].status'

Expected output:

NAME                                     READY   DISKPRESSURE
<NODE_NAME>                              True    True

System pods on the node will also show eviction messages like:
```
Pod was rejected: The node had condition: [DiskPressure].
```

Attempt a minimal exec through any pod still scheduled on the node:

kubectl -n crusoe-system exec <POD_NAME> -c <CONTAINER_NAME> -- true

Expected output when the boot disk is fully exhausted:

OCI runtime exec failed: write /tmp/runc-process...: no space left on device

If SSH also fails with Permission denied (publickey) and your keys are distributed by a DaemonSet, check whether that DaemonSet's pods on this node have been evicted. If they have, there is no remaining interactive path onto the node — continue with the steps below.

Step 2: Diagnose the Disk Consumer Without Node Access

Even with no way onto the node, the kubelet stats summary can classify the failure. Query it through the API server:

kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary

Expected output (excerpt):

"fs": {
  "capacityBytes": 142805672755,
  "availableBytes": 0,
  "inodes": 8912896,
  "inodesFree": 8814592,
  "inodesUsed": 98304
},
"runtime": {
  "imageFs": {
    "capacityBytes": 1900000000000,
    "availableBytes": 1619000000000
  }
}

ℹ️ Note: If the stats endpoint does not respond on a fully saturated node, fall back to kubectl describe node <NODE_NAME> and the events on the node's pods.

Look at four numbers:

nodefs (boot disk) — capacity vs. available. In this failure mode, available is at or near 0 bytes.
imagefs (local NVMe, when ephemeral storage for containerd is enabled) — typically lightly used, ruling out container images as the cause.
Inode usage on nodefs — inodesUsed against inodes. A few percent rules out inode exhaustion, which is a different failure with the same no space left on device error.
Per-pod ephemeral storage totals — if these sum to only a few GB while the boot disk holds 100+ GB, the data was written outside pod-attributed storage.

That combination — full nodefs, healthy imagefs, low inode usage, small pod ephemeral totals — means the boot disk is full of host-side data on the root filesystem, outside container and pod storage.

ℹ️ Note: A confirming signature lives in the kubelet logs and events: the eviction manager's "pods ranked for eviction" list contains only system and DaemonSet pods, and the kubelet logs messages like unable to evict any pods or cannot evict a critical pod. The workload actually consuming the disk never appears in the ranking, because eviction ordering is based on per-pod ephemeral usage — which is exactly what a host-side write bypasses.

Step 3: Cordon the Node

Cordon before you touch the workload. If you delete the writer first, its controller can reschedule a replacement onto this same node and resume filling the disk:
```
kubectl cordon <NODE_NAME>
```
Expected output:
```
node/<NODE_NAME> cordoned
```

Step 4: Identify and Stop the Workload Writing to the Boot Disk

The most common cause on CMK is a hostPath volume with type: DirectoryOrCreate whose path lives under a local NVMe mount point (for example /mnt/nvme/<WORKSPACE_DIR>). If the NVMe filesystem is not mounted at the moment the container starts, the kubelet silently creates the directory on the root disk instead, and the workload's workspace, outputs, or checkpoints fill the boot disk — even though the pod's own ephemeral-storage accounting stays near zero.

This is per-pod-start behavior, not per-workload: replicas of the same job on other nodes may be healthy because their NVMe mount was present at container start. Suspect the pods that were running on the affected nodes when growth began, and check their manifests for hostPath volumes with DirectoryOrCreate.

Delete the offending workload pods so the writes stop:

kubectl delete pod <WORKLOAD_POD_NAME> -n <NAMESPACE>

Expected output:

pod "<WORKLOAD_POD_NAME>" deleted

⚠️ Warning: Do not skip this step. If the writer keeps running, any space you free will be immediately reconsumed and the node will return to the same state.

Step 5: Drain the Node

Move the remaining workloads off so the node is idle before the reset:
```
kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data
```
Expected output (drain prints one line per evicted pod, then):
```
node/<NODE_NAME> drained
```

ℹ️ Note: If drain stalls, a PodDisruptionBudget is blocking eviction. Check kubectl get pdb -A and either relax the budget or scale the owning workload down before retrying.

Step 6: Reset the VM

Reset the underlying VM with the Crusoe CLI:
```
crusoe compute vms reset <VM_NAME>
```
The reset does not delete the data on the boot disk. What it frees is transient: tmpfs-backed paths are cleared, and restarting containerd and the kubelet releases runtime scratch state and any deleted-but-still-open files whose space the kernel was holding. That is often enough headroom for kubectl exec to work again after the node rejoins.

Expected result: the node returns to Ready=True after the reset. Re-run the minimal exec from Step 1. If it now succeeds, continue to Step 7. If pods still fail to start, skip to Step 8.

Step 7: Clean Up the Boot Disk (If Exec Is Restored)

Once the node returns and exec succeeds, locate and remove the stray data from the root disk:
```
df -h /
sudo du -xh -d1 / 2>/dev/null | sort -rh | head -15
```
Expected output (example of the failure signature):
```
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       133G  133G     0 100% /

127G    /mnt
2.1G    /var
1.4G    /usr
...
```
The directory dominating the listing is your target. In the hostPath fallback case, it is the workspace path that should have been on NVMe. The -x flag keeps du on the root filesystem, so anything it reports is genuinely consuming boot disk.
Keep the node cordoned until df -h / confirms the boot disk has healthy free space, then uncordon:
```
kubectl uncordon <NODE_NAME>
```
Expected output:
```
node/<NODE_NAME> uncordoned
```

ℹ️ Note: If the NVMe filesystem is now mounted over the offending path, the runaway data is hidden underneath the mount. Unmount it first to reach and delete the files on the root disk, then remount.

Step 8: Delete the VM If the Node Does Not Recover

If the reset does not free enough space — for example, the node returns Ready=True but DiskPressure=True persists and new pods still fail with:
```
mkdir /var/lib/kubelet/pods/...: no space left on device
```
then the boot disk has no recoverable headroom and deleting the VM is the only remaining option. You can do this yourself from the Crusoe Console or CLI; no support intervention is required. Delete the VM, not just the Kubernetes node object:
```
crusoe compute vms delete <VM_NAME>
```
Expected result: the node pool provisions a replacement automatically, and a new node joins the cluster within a few minutes — visible via kubectl get nodes with a new name in the same node pool.

⚠️ Warning: Deleting the VM destroys all node-local state, including anything on the boot disk and local NVMe (such as distributed-framework worker state). Confirm nothing irreplaceable lives only on that node before deleting.

💡 Tip: To prevent recurrence, ensure hostPath volumes that target NVMe-backed paths cannot fall back to the root disk: verify the mount exists before the workload starts (for example, with an init container mount-check), or avoid DirectoryOrCreate under mount points entirely.

Example

A multi-node training job mounts /mnt/nvme/checkpoints as a hostPath volume with type: DirectoryOrCreate and writes a checkpoint every few minutes. On seven of eight nodes the NVMe filesystem was mounted before the container started, so the checkpoints land on local NVMe as intended. On the eighth, the container won the race against the mount, so the kubelet created /mnt/nvme/checkpoints on the root disk instead.

Several hours later that node reports Ready=True with DiskPressure=True, monitoring and log-collector DaemonSets have been evicted, kubectl exec fails with no space left on device, and SSH is refused because the key-distribution DaemonSet can no longer schedule there. The job's other seven replicas look completely healthy, which is what makes the failure easy to misread as a single bad node rather than a manifest problem.

The stats summary shows nodefs at 0 bytes available, imagefs on NVMe 85% free, inode usage near 1%, and per-pod ephemeral totals of a few hundred MB — the host-side-write signature from Step 2. Cordoning the node, deleting the training pod, draining, and resetting the VM restores exec, and du -xh -d1 / shows 127 GB under /mnt on the root disk. After unmounting NVMe, deleting the stray checkpoint directory, and remounting, df -h / reports healthy free space and the node is uncordoned. Adding an init container that fails unless /mnt/nvme is a mount point keeps it from happening again.

Was this article helpful?

0 out of 0 found this helpful

Comments

0 comments

Article is closed for comments.