How-To Use Crusoe CSI Driver for Shared Disk NFS Mounts on CMK

Last Updated: January 5, 2026

Introduction

This article provides guidance on configuring NFS mounts using the Crusoe CSI Driver on Crusoe Managed Kubernetes (CMK) clusters. NFS (Network File System) mounts enable shared storage accessible by multiple pods across different nodes (ReadWriteMany), which is essential for AI/ML workloads requiring shared datasets.

By following this guide, you will learn how to:

• Verify your cluster meets the requirements for NFS mounts
• Upgrade the CSI driver to a compatible version
• Create StorageClasses, PVCs, and Deployments that use NFS storage
• Troubleshoot common NFS mount issues

Prerequisites

Before configuring NFS mounts, ensure the following requirements are met:

NFS Migration Status

• Your project must have completed (or be in the process of completing) the NFS migration. Contact Crusoe Support to confirm your project's NFS migration status if unsure

Version Compatibility

Note: The versions listed above come with the latest CSI driver and NFS drivers pre-bundled. Other worker image versions (e.g., 1.32.7-cmk.3) may also support NFS mounts when paired with CSI driver v0.10.2+. If you encounter mount failures, upgrade to the recommended versions.

We expect upcoming Kubernetes v1.34.x for both Control Plane and Worker Images to support NFS compatibility.

Cluster Requirements

• CMK cluster with crusoe_csi add-on enabled (for new clusters), or Helm 3.x installed for manual CSI installation
• kubectl configured to access your CMK cluster
• Sufficient CPU/memory resources on nodes for CSI controller pods

Step-by-Step Instructions

For New CMK Clusters

If you are creating a new CMK cluster, select one of the compatible control plane images listed in the Prerequisites. When you enable the crusoe_csi add-on during cluster creation, the CSI driver with NFS support (v0.10.2+) will be automatically installed. Ensure your node pools use compatible worker images.

Once the cluster is created, proceed to Step 7 to create your StorageClass and begin using NFS mounts.

For Existing CMK Clusters

If you have an existing CMK cluster that needs NFS support, follow the steps below to upgrade your CSI driver.

1. Check Existing CSI Driver Version

Determine if your cluster already has the CSI driver installed and what version it is running.

helm list -n crusoe-system

Example output (older version requiring upgrade):

NAME              NAMESPACE      REVISION  STATUS    CHART                    APP VERSION
crusoe-csi-driver crusoe-system  1         deployed  crusoe-csi-driver-0.6.2  v0.2.1

If a Helm release exists or CSI pods are running, proceed to Step 2.

2. Uninstall Existing CSI Driver

If a Helm release exists, uninstall it:

# If it shows crusoe-csi-driver, uninstall it
helm uninstall crusoe-csi-driver -n crusoe-system

3. Remove Remaining CSI Resources

Even after Helm uninstall, some resources may remain. Always run these commands to ensure a clean state before installing the new CSI driver:

# Delete Deployments
kubectl delete deployment -n crusoe-system crusoe-csi-driver-fs-controller
kubectl delete deployment -n crusoe-system crusoe-csi-driver-ssd-controller

# Delete DaemonSets
kubectl delete daemonset -n crusoe-system crusoe-csi-driver-fs-node
kubectl delete daemonset -n crusoe-system crusoe-csi-driver-ssd-node

# Delete CSI Driver registrations
kubectl delete csidriver fs.csi.crusoe.ai
kubectl delete csidriver ssd.csi.crusoe.ai

# Delete ClusterRoles
kubectl delete clusterrole crusoe-csi-driver-external-attacher-runner
kubectl delete clusterrole crusoe-csi-driver-external-provisioner-runner
kubectl delete clusterrole crusoe-csi-driver-external-resizer-runner
kubectl delete clusterrole crusoe-csi-driver-node-read-labels

# Delete ClusterRoleBindings
kubectl delete clusterrolebinding crusoe-csi-driver-csi-attacher-role
kubectl delete clusterrolebinding crusoe-csi-driver-csi-provisioner-role
kubectl delete clusterrolebinding crusoe-csi-driver-csi-resizer-role
kubectl delete clusterrolebinding crusoe-csi-driver-node-read-labels

# Delete ServiceAccounts
kubectl delete serviceaccount -n crusoe-system crusoe-csi-driver-controller
kubectl delete serviceaccount -n crusoe-system crusoe-csi-driver-node

Verify cleanup:

kubectl get all -n crusoe-system | grep csi
kubectl get csidriver

All commands should return empty/no resources found.

4. Install CSI Driver via Helm

Add the Crusoe Helm repository and install the latest CSI driver:

helm repo add crusoe https://crusoecloud.github.io/crusoe-csi-driver-helm-charts
helm repo update

helm install crusoe-csi-driver crusoe/crusoe-csi-driver \
  --namespace crusoe-system \
  --create-namespace \
  --version v0.10.4

To list all available CSI driver versions:

helm search repo crusoe/crusoe-csi-driver --versions

5. Create API Keys Secret (If Needed)

The CSI driver requires a secret named crusoe-api-keys. If your cluster has credentials in a differently named secret (e.g., crusoe-secrets), create the expected secret:

# Check if crusoe-api-keys already exists
kubectl get secret crusoe-api-keys -n crusoe-system

# If not found, create it from existing crusoe-secrets
kubectl create secret generic crusoe-api-keys \
  --namespace crusoe-system \
  --from-literal=CRUSOE_ACCESS_KEY="$(kubectl get secret crusoe-secrets -n crusoe-system -o jsonpath='{.data.CRUSOE_ACCESS_KEY}' | base64 -d)" \
  --from-literal=CRUSOE_SECRET_KEY="$(kubectl get secret crusoe-secrets -n crusoe-system -o jsonpath='{.data.CRUSOE_SECRET_KEY}' | base64 -d)"

6. Verify CSI Driver Installation

# Check CSI pods are running
kubectl get pods -n crusoe-system | grep csi

# Check registered CSI drivers
kubectl get csidriver

Expected csidriver output:

NAME                ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   AGE
fs.csi.crusoe.ai    true             false            false             1m
ssd.csi.crusoe.ai   true             false            false             1m

Important: Note the exact provisioner names from kubectl get csidriver. You will need these when creating StorageClasses.

For All Clusters (New and Existing)

7. Create StorageClass for SharedFS (NFS)

Create a file named storageclass-sharedfs.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: crusoe-sharedfs-sc
provisioner: fs.csi.crusoe.ai
volumeBindingMode: Immediate
allowVolumeExpansion: true
reclaimPolicy: Delete

Apply it:

kubectl apply -f storageclass-sharedfs.yaml

8. Create PersistentVolumeClaim

Create a file named pvc-sharedfs.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sharedfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: crusoe-sharedfs-sc
  resources:
    requests:
      storage: 1Ti

Note: Kubernetes uses Ti, Gi, Mi — not TiB, GiB, MiB.

Apply it:

kubectl apply -f pvc-sharedfs.yaml

Verify the PVC is bound:

kubectl get pvc sharedfs-pvc

9. Create Deployment Using NFS Mount

Create a file named deployment-nfs.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-nfs-app
  labels:
    app: example-nfs-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-nfs-app
  template:
    metadata:
      labels:
        app: example-nfs-app
    spec:
      containers:
      - name: app
        image: ubuntu:24.04
        command: ["sleep", "infinity"]
        volumeMounts:
        - name: shared-data
          mountPath: /data
      volumes:
      - name: shared-data
        persistentVolumeClaim:
          claimName: sharedfs-pvc

Apply it:

kubectl apply -f deployment-nfs.yaml

Verify pods are running:

kubectl get pods -l app=example-nfs-app

Example

A successful NFS mount deployment shows all pods in Running status with the shared volume mounted:

$ kubectl get pods -l app=example-nfs-app
NAME                               READY   STATUS    RESTARTS   AGE
example-nfs-app-59fb96cd7f-xxxxx   1/1     Running   0          52s
example-nfs-app-59fb96cd7f-xxxxx   1/1     Running   0          52s
example-nfs-app-59fb96cd7f-xxxxx   1/1     Running   0          52s

$ kubectl exec example-nfs-app-59fb96cd7f-64prv -- df -h /data
Filesystem                                                Size  Used Avail Use% Mounted on
100.64.0.2:/volumes/866993a6-xxxx-xxxx-xxxx-xxxxxxxxxxxx  1.0T     0  1.0T   0% /data

Troubleshooting

Issue: Mount Failed - No Such Device

MountVolume.SetUp failed for volume "pvc-xxxxx": mount failed: exit status 255
mount: mounting 100.64.0.2:/volumes/xxxxx failed: No such device

Root Cause: The worker node image does not have NFS kernel modules pre-installed, or the CSI driver version is incompatible.

Solution: Upgrade your node pool to use a compatible worker image (see Version Compatibility table) and ensure CSI driver is v0.10.2+.

Issue: Secret Not Found

Error: secret "crusoe-api-keys" not found

Root Cause: The CSI driver expects a secret named crusoe-api-keys but it doesn't exist or has a different name.

Solution: Create the secret from existing credentials as shown in Step 5.

Issue: CSI Controller Pod Pending - Insufficient CPU

0/2 nodes are available: 2 Insufficient cpu

Root Cause: Not enough CPU resources available on nodes to schedule the CSI controller pod.

Solution: Scale up your node pool (add more nodes or use larger instance types). Note: Node DaemonSet pods may still work; only the controller needs additional scheduling capacity.

Issue: PVC Stuck in Pending

Waiting for a volume to be created either by the external provisioner...

Root Cause: StorageClass provisioner name doesn't match registered CSI driver, or CSI controller is not running.

Solution:

• Check registered CSI drivers: kubectl get csidriver
• Verify CSI controller pod is running: kubectl get pods -n crusoe-system | grep controller
• Update StorageClass to use correct provisioner name

Additional Resources

• Crusoe CSI Driver Helm Charts GitHub Repository
• CSI Driver Changelog