Last Updated: Jan 14, 2026
Introduction
Other Kubernetes offerings, such as NVIDIA Lepton, can use Crusoe's GPU VMs as worker nodes. Currently, the Crusoe boot disk has a fixed size of 128gb, and for large scale ML jobs running on Kubernetes, it could fill up the temporary storage (in /var/lib/containerd, mounted on the boot disk) and run out of space. This solution is will allow users to not run out of space when executing long-running workloads on Kubernetes by using the local NVMe.
Note: This solution is for using other Kubernetes distributions - Crusoe Managed Kubernetes (CMK) already provides the ability to use the local NVMe natively for containerd as shown here.
Prerequisites
- Running Kubernetes cluster using Crusoe Cloud instances as worker nodes
-
kubectlconfigured to access the cluster - Containerd storage mounted to
/var/lib/containerd - Familiarity with Kubernetes DaemonSets and
hostPathvolumes.
Step-by-Step Instructions
1. Deploy the DaemonSet
-
Create a DaemonSet that: 1/ configure RAID0 on the local NVMes and mount to
/mnt/containerd-nvmeand 2/ Bind mount it to/var/lib/containerdcontainerd-nvme-bind-mount.yaml
apiVersion: apps/v1 kind: DaemonSet metadata: name: raid-setup namespace: kube-system spec: selector: matchLabels: name: raid-setup template: metadata: labels: name: raid-setup spec: hostPID: true hostNetwork: true containers: - name: raid-setup image: 'ubuntu:22.04' securityContext: privileged: true command: - /bin/bash - '-c' args: - > set -euo pipefail echo "Starting RAID setup script for containerd bind mount..." # Remove Fluent Bit repository file to avoid GPG errors echo "Removing Fluent Bit repository file..." rm -f /etc/apt/sources.list.d/fluent-bit.list 2>/dev/null || true # Update and install required packages echo "Updating apt repositories..." apt-get update -o Acquire::AllowInsecureRepositories=true -o Acquire::AllowDowngradeToInsecureRepositories=true 2>&1 || true echo "Installing nvme-cli, mdadm, gawk, xfsprogs, util-linux, and rsync..." DEBIAN_FRONTEND=noninteractive apt-get install -y nvme-cli mdadm gawk xfsprogs util-linux rsync 2>&1 echo "info: detecting NVMe drives by-id..." # Collect all nvme-* symlinks from host's /dev/disk/by-id all_symlinks=$(ls -1 /dev/disk/by-id/nvme-* 2>/dev/null | grep -vE '(-part[0-9]+$|_[0-9]+$)' || true) if [ -z "$all_symlinks" ]; then echo "WARNING: No NVMe drives detected" echo "INFO: RAID setup skipped - no NVMe devices available." echo "Entering sleep mode..." exec sleep infinity fi # Deduplicate symlinks nvme_devices="" seen_targets="" echo "info: processing symlinks..." for symlink in $all_symlinks; do target=$(readlink "$symlink" || echo "") if [ -z "$target" ]; then echo "warning: could not read symlink $symlink, skipping" continue fi if [[ "$target" =~ ^/ ]]; then abs_target="$target" else device_name=$(basename "$target") abs_target="/dev/$device_name" fi if [ ! -b "$abs_target" ]; then echo "warning: device $abs_target does not exist, skipping" continue fi if ! echo "$seen_targets" | grep -q -w "$abs_target"; then nvme_devices="$nvme_devices $abs_target" seen_targets="$seen_targets $abs_target" echo "info: found device: $abs_target (from symlink: $(basename $symlink))" fi done nvme_devices=$(echo "$nvme_devices" | xargs) num_nvme=$(echo "$nvme_devices" | wc -w) if [ "$num_nvme" -eq 0 ]; then echo "ERROR: No valid NVMe block devices found" echo "Entering sleep mode..." exec sleep infinity fi echo "info: found $num_nvme NVMe drive(s): $nvme_devices" total_size=0 for dev in $nvme_devices; do if [ ! -b "$dev" ]; then echo "ERROR: Device $dev is not a block device" exit 1 fi size=$(blockdev --getsize64 $dev 2>/dev/null || echo 0) size_gb=$(echo "$size" | awk '{printf "%.2f", $1/1024/1024/1024}') total_size=$((total_size + size)) echo "info: verified: $dev (${size_gb} GB)" done raid_size_gb=$(echo "$total_size" | awk '{printf "%.2f", $1/1024/1024/1024}') echo "info: total RAID0 capacity will be: ${raid_size_gb} GB" if [ -b /dev/md/ephemeral ]; then echo "info: md dev already exists, checking if active..." if mdadm --detail /dev/md/ephemeral >/dev/null 2>&1; then echo "info: RAID array is active and healthy" skip_raid_creation=true else echo "warning: md dev exists but not healthy, attempting reassemble..." mdadm --stop /dev/md/ephemeral 2>/dev/null || true if mdadm --assemble /dev/md/ephemeral $nvme_devices 2>/dev/null; then echo "info: RAID array reassembled successfully" skip_raid_creation=true else echo "info: reassemble failed, recreating RAID array" mdadm --zero-superblock $nvme_devices 2>/dev/null || true rm -f /dev/md/ephemeral skip_raid_creation=false fi fi else skip_raid_creation=false fi if [ "$skip_raid_creation" != "true" ]; then echo "info: creating RAID0 with $num_nvme device(s)" if ! mdadm --create /dev/md/ephemeral \ --force \ --name=ephemeral \ --level=0 \ --raid-devices=$num_nvme \ --homehost=any \ $nvme_devices; then echo "error: failed to create RAID array" exit 1 fi echo "info: RAID array created successfully" fi echo "info: waiting for device to settle..." udevadm settle 2>/dev/null || sleep 2 sleep 2 if [ ! -b /dev/md/ephemeral ]; then echo "error: /dev/md/ephemeral not created" exit 1 fi if blkid -p -u filesystem /dev/md/ephemeral 2>/dev/null | grep -q xfs; then echo "info: already formatted with XFS" else echo "info: creating XFS filesystem..." if ! mkfs.xfs -f /dev/md/ephemeral; then echo "error: failed to create XFS filesystem" exit 1 fi echo "info: XFS filesystem created" fi echo "info: creating mount point on host..." nsenter --mount=/proc/1/ns/mnt -- mkdir -p /mnt/containerd-nvme if nsenter --mount=/proc/1/ns/mnt -- mountpoint -q /mnt/containerd-nvme; then echo "info: /mnt/containerd-nvme is already mounted on host" current_mount=$(nsenter --mount=/proc/1/ns/mnt -- mount | grep "/mnt/containerd-nvme" | awk '{print $1}') echo "info: currently mounted device: $current_mount" if [ "$current_mount" != "/dev/md/ephemeral" ] && [ "$current_mount" != "/dev/md127" ]; then echo "warning: wrong device mounted, remounting..." nsenter --mount=/proc/1/ns/mnt -- umount /mnt/containerd-nvme || true echo "info: mounting RAID array on host..." nsenter --mount=/proc/1/ns/mnt -- mount /dev/md/ephemeral /mnt/containerd-nvme fi else echo "info: mounting RAID array on host..." nsenter --mount=/proc/1/ns/mnt -- mount /dev/md/ephemeral /mnt/containerd-nvme echo "info: mounted successfully on host" fi nsenter --mount=/proc/1/ns/mnt -- chmod 755 /mnt/containerd-nvme echo "info: verifying mount on host..." nsenter --mount=/proc/1/ns/mnt -- mountpoint -q /mnt/containerd-nvme nsenter --mount=/proc/1/ns/mnt -- df -h /mnt/containerd-nvme # Add to fstab for persistence across reboots echo "info: adding RAID mount to /etc/fstab..." RAID_UUID=$(blkid -s UUID -o value /dev/md/ephemeral) if ! nsenter --mount=/proc/1/ns/mnt -- grep -q "$RAID_UUID" /etc/fstab; then echo "UUID=${RAID_UUID} /mnt/containerd-nvme xfs defaults,noatime 0 2" | nsenter --mount=/proc/1/ns/mnt -- tee -a /etc/fstab echo "info: added to /etc/fstab" else echo "info: already in /etc/fstab" fi # Save RAID configuration for persistence echo "info: saving RAID configuration..." mdadm --detail --scan | nsenter --mount=/proc/1/ns/mnt -- tee /etc/mdadm/mdadm.conf echo "info: RAID configuration saved" # Setup containerd directory structure on NVMe echo "info: setting up containerd directory structure..." nsenter --mount=/proc/1/ns/mnt -- mkdir -p /mnt/containerd-nvme/containerd # Sync existing containerd data if not already done if ! nsenter --mount=/proc/1/ns/mnt -- [ -f /mnt/containerd-nvme/containerd/.sync_complete ]; then echo "info: checking for existing containerd data..." if nsenter --mount=/proc/1/ns/mnt -- [ -d /var/lib/containerd ] && \ nsenter --mount=/proc/1/ns/mnt -- [ "$(ls -A /var/lib/containerd 2>/dev/null)" ]; then echo "info: syncing existing containerd data to NVMe (this may take several minutes)..." CONTAINERD_SIZE=$(nsenter --mount=/proc/1/ns/mnt -- du -sh /var/lib/containerd 2>/dev/null | awk '{print $1}') echo "info: current containerd size: ${CONTAINERD_SIZE}" # Use rsync to copy data nsenter --mount=/proc/1/ns/mnt -- rsync -avxHAX --info=progress2 /var/lib/containerd/ /mnt/containerd-nvme/containerd/ nsenter --mount=/proc/1/ns/mnt -- touch /mnt/containerd-nvme/containerd/.sync_complete echo "info: data sync completed" else echo "info: no existing containerd data to sync" nsenter --mount=/proc/1/ns/mnt -- touch /mnt/containerd-nvme/containerd/.sync_complete fi else echo "info: containerd data already synced (found .sync_complete marker)" fi # Create bind mount to /var/lib/containerd echo "info: setting up bind mount for /var/lib/containerd..." if nsenter --mount=/proc/1/ns/mnt -- mountpoint -q /var/lib/containerd; then echo "info: /var/lib/containerd is already a bind mount" current_source=$(nsenter --mount=/proc/1/ns/mnt -- findmnt -n -o SOURCE /var/lib/containerd) echo "info: current bind mount source: $current_source" if [ "$current_source" != "/mnt/containerd-nvme/containerd" ]; then echo "warning: wrong bind mount source, fixing..." nsenter --mount=/proc/1/ns/mnt -- umount /var/lib/containerd || true nsenter --mount=/proc/1/ns/mnt -- mount --bind /mnt/containerd-nvme/containerd /var/lib/containerd echo "info: bind mount corrected" fi else echo "info: creating bind mount..." # Backup original directory if it exists and has content if nsenter --mount=/proc/1/ns/mnt -- [ -d /var/lib/containerd ] && \ nsenter --mount=/proc/1/ns/mnt -- [ "$(ls -A /var/lib/containerd 2>/dev/null)" ]; then if ! nsenter --mount=/proc/1/ns/mnt -- [ -d /var/lib/containerd.backup ]; then echo "info: backing up original /var/lib/containerd to /var/lib/containerd.backup" nsenter --mount=/proc/1/ns/mnt -- mv /var/lib/containerd /var/lib/containerd.backup nsenter --mount=/proc/1/ns/mnt -- mkdir -p /var/lib/containerd else echo "info: backup already exists, removing current directory" nsenter --mount=/proc/1/ns/mnt -- rm -rf /var/lib/containerd nsenter --mount=/proc/1/ns/mnt -- mkdir -p /var/lib/containerd fi else nsenter --mount=/proc/1/ns/mnt -- mkdir -p /var/lib/containerd fi # Create the bind mount nsenter --mount=/proc/1/ns/mnt -- mount --bind /mnt/containerd-nvme/containerd /var/lib/containerd echo "info: bind mount created successfully" fi # Add bind mount to fstab echo "info: adding bind mount to /etc/fstab..." if ! nsenter --mount=/proc/1/ns/mnt -- grep -q "/mnt/containerd-nvme/containerd /var/lib/containerd" /etc/fstab; then echo "/mnt/containerd-nvme/containerd /var/lib/containerd none bind 0 0" | nsenter --mount=/proc/1/ns/mnt -- tee -a /etc/fstab echo "info: bind mount added to /etc/fstab" else echo "info: bind mount already in /etc/fstab" fi # Verify everything echo "info: verifying bind mount..." nsenter --mount=/proc/1/ns/mnt -- mountpoint -q /var/lib/containerd nsenter --mount=/proc/1/ns/mnt -- df -h /var/lib/containerd echo "============================================" echo "info: RAID + containerd bind mount setup completed successfully!" echo "info: /var/lib/containerd is now using NVMe RAID0 storage" echo "============================================" # Monitoring loop while true; do sleep 300 # Check RAID mount if ! nsenter --mount=/proc/1/ns/mnt -- mountpoint -q /mnt/containerd-nvme; then echo "WARNING: /mnt/containerd-nvme is no longer mounted! Attempting remount..." if nsenter --mount=/proc/1/ns/mnt -- mount /dev/md/ephemeral /mnt/containerd-nvme; then echo "INFO: Successfully remounted /mnt/containerd-nvme" nsenter --mount=/proc/1/ns/mnt -- chmod 755 /mnt/containerd-nvme else echo "ERROR: Failed to remount /mnt/containerd-nvme" fi fi # Check bind mount if ! nsenter --mount=/proc/1/ns/mnt -- mountpoint -q /var/lib/containerd; then echo "WARNING: /var/lib/containerd bind mount is gone! Attempting remount..." if nsenter --mount=/proc/1/ns/mnt -- mount --bind /mnt/containerd-nvme/containerd /var/lib/containerd; then echo "INFO: Successfully remounted /var/lib/containerd" else echo "ERROR: Failed to remount /var/lib/containerd" fi fi # Log disk usage echo "INFO: Current disk usage:" nsenter --mount=/proc/1/ns/mnt -- df -h /var/lib/containerd | tail -1 done volumeMounts: - name: host-dev mountPath: /dev - name: host-disk mountPath: /host-disk-by-id readOnly: true - name: host-etc mountPath: /etc - name: containerd-mount mountPath: /var/lib/containerd mountPropagation: Bidirectional - name: nvme-mount mountPath: /mnt/containerd-nvme mountPropagation: Bidirectional volumes: - name: host-dev hostPath: path: /dev - name: host-disk hostPath: path: /dev/disk/by-id - name: host-etc hostPath: path: /etc - name: containerd-mount hostPath: path: /var/lib/containerd type: DirectoryOrCreate - name: nvme-mount hostPath: path: /mnt/containerd-nvme type: DirectoryOrCreate -
Apply the pod configuration:
kubectl apply -f containerd-nvme-bind-mount.yaml -
You can check the logs of one of the replicas from the DaemonSets to see that it is successfully configured. You should see a similar output at the end as the following:
info: setting up bind mount for /var/lib/containerd... info: creating bind mount... info: backing up original /var/lib/containerd to /var/lib/containerd.backup info: bind mount created successfully info: adding bind mount to /etc/fstab... /mnt/containerd-nvme/containerd /var/lib/containerd none bind 0 0 info: bind mount added to /etc/fstab info: verifying bind mount... Filesystem Size Used Avail Use% Mounted on /dev/md127 24T 176G 24T 1% /var/lib/containerd ============================================ info: RAID + containerd bind mount setup completed successfully! info: /var/lib/containerd is now using NVMe RAID0 storage ============================================You can also SSH into the VM directly to check the mounts and see that bind mount from the NVME (on
/mnt/containerd-nvme) to the containerd (/var/lib/containerd) is configured correctly:ssh ubuntu@<IP of the Worker Node VM> #Followed by ubuntu@node-name:~$ df -Th /mnt/containerd-nvme Filesystem Type Size Used Avail Use% Mounted on /dev/md127 xfs 24T 175G 24T 1% /mnt/containerd-nvme ubuntu@node-name:~$ df -Th /var/lib/containerd Filesystem Type Size Used Avail Use% Mounted on /dev/md127 xfs 24T 175G 24T 1% /var/lib/containerd