How to Create a NodePool with Ephemeral Storage for Containerd in Crusoe Managed Kubernetes (CMK)

Last Updated: Dec 2, 2025

Introduction

When deploying containerized applications on Crusoe Managed Kubernetes (CMK), container images, logs, and ephemeral storage can quickly consume the default 128 GiB root disk. This can lead to Kubernetes declaring Node Disk Pressure, which causes pod evictions, throttling, and node instability.

This guide provides the steps to create a CMK nodepool that is configured to utilize the Local NVMe Ephemeral Disks as the Containerd root path. This greatly increases the available high-speed storage for container images and scratch space, preventing node instability due to disk pressure.

For SKU information refer to Virtual Machine - Overview.

Note: This feature is in early access and only supported via Crusoe CLI and will NOT work on existing node pools.

Prerequisites

• Access to a Crusoe Cloud project with appropriate permissions
• Access to Crusoe CLI or Console
• CMK cluster with NodePool Versions:
  • 1.30.8-cmk.11 + (Deprecated version)
  • 1.31.7-cmk.6 + 
  • 1.32.7-cmk.3 + 

Step-by-Step Instructions

1. Confirm that your Crusoe CLI configuration file, located at ~/.crusoe/config , is correctly set up with your required Access Key, Secret Key, and the target Project.

   $ crusoe --version
   crusoe version v0.31.15

2. Fetch CMK Cluster Details:

   $ crusoe kubernetes clusters list

   Example:

   $ crusoe kubernetes clusters list
   cluster name       id                                    current state    version        location              node pool count
   -------------------------------------------------------------------------------------------------------------------------------
   us-east-test       b5e7b3a4-ae41-4a17-bbb0-829b4ad29dcf  STATE_RUNNING    1.32.7-cmk.3  us-east1-a            1

3.  Fetch Infiniband (IB) Partition Details:

   $ crusoe networking ib-partitions list

   Example:

   $ crusoe networking ib-partitions list
   name            partition id                          ib network id
   ------------------------------------------------------------------------------------------
   us-east-a100    dcc11892-70aa-4dce-a578-a689e34c0ea5  36f543c2-0c3c-4c8d-b717-194b16e43fcd

4.  Fetch Crusoe VM Type Details:

   $ crusoe compute vms types

5.  Create a new nodepool with --ephemeral-storage-for-containerd flag set to true via CLI as below: 

   $ crusoe kubernetes nodepools create 
     --name <name> 
     --node-pool-version 1.32.7-cmk.3
     --ib-partition-id <ib_partition_id> 
     --ephemeral-storage-for-containerd true 
     --cluster-id <cluster_id> 
     --count <count_value> 
     --keyfile <key_file> 
     --type <type>

   Example:

   $ crusoe kubernetes nodepools create 
      --name a100-test-np 
      --node-pool-version 1.32.7-cmk.3
      --ib-partition-id dcc11892-70aa-4dce-a578-a689e34c0ea5 
      --ephemeral-storage-for-containerd true 
      --cluster-id b5e7b3a4-ae41-4a17-bbb0-829b4ad29dcf 
      --count 2 
      --keyfile "/Users/dummy/.ssh/id_ed25519.pub" 
      --type a100-80gb-sxm-ib.8x
      
   node pool name: a100-test-np
   id: 970da5d8-7426-4480-9fb7-10b1b4991733
   state: STATE_RUNNING
   type: a100-80gb-sxm-ib.8x
   desired count: 2
   running count: 2
   image: 1.32.7-cmk.3
   node labels: { crusoe.ai/accelerator: "nvidia-a100-80gb-sxm-ib", crusoe.ai/instance.class: "a100-80gb-sxm-ib", crusoe.ai/project.id: "6d012e45-f991-47f6-aace-d8a61a7c32c7"  }
   instance ids: [014f9978-16bd-4fbb-bb97-57d7b00187ff 5ac503c9-ba0f-432b-9a8d-f124abc92854]
   subnet id: 72a12f6d-388d-40d6-a4e7-456598af18f8
   cluster name: test-cluster
   cluster id: b5e7b3a4-ae41-4a17-bbb0-829b4ad29dcf
   ephemeral storage for containerd: true

   To check all parameters that can be set, run crusoe kubernetes nodepools create -h

6.  To verify, SSH into a node in the new nodepool and confirm that the Containerd path (/var/lib/containerd) is configured to use the local NVMe drive (e.g., /dev/nvme0n1) - this is 1/8 local ephemeral NVMe drive.

   ubuntu@np-970da5d8-1:~$ lsblk
   NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
   loop0     7:0    0  63.9M  1 loop /snap/core20/2318
   loop1     7:1    0    87M  1 loop /snap/lxd/29351
   loop2     7:2    0  38.8M  1 loop /snap/snapd/21759
   vda     252:0    0   128G  0 disk
   ├─vda1  252:1    0 127.9G  0 part /
   ├─vda14 252:14   0     4M  0 part
   └─vda15 252:15   0   106M  0 part /boot/efi
   nvme0n1 259:0    0 894.3G  0 disk /mnt/nvme
   nvme1n1 259:1    0 894.3G  0 disk
   nvme3n1 259:2    0 894.3G  0 disk
   nvme2n1 259:3    0 894.3G  0 disk
   nvme7n1 259:4    0 894.3G  0 disk
   nvme5n1 259:5    0 894.3G  0 disk
   nvme4n1 259:6    0 894.3G  0 disk
   nvme6n1 259:7    0 894.3G  0 disk
   
   ubuntu@np-970da5d8-1:~$ df -h /var/lib/containerd
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme0n1    880G   12G  824G   2% /mnt/nvme

Additional Resources

• CMK - Manage your Node Pools
• Kubernetes Clusters Running Out of Disk Space