Pods stuck in ContainerCreating or Terminating on NFS-backed PVCs

Introduction

If a pod using a PVC on Crusoe's Shared Filesystem storage class (fs.csi.crusoe.ai) is stuck in ContainerCreating or Terminating, this article will help you recognize the error signatures below and gather the right diagnostics before contacting Crusoe Support.

Prerequisites

• kubectl Access to the Affected Cluster
• SSH Access to the Affected Node (or kubectl debug node Access as a Fallback)
• procps Installed on the Node (For ps, Install With apt-get install -y procps if Missing)

Symptoms

• Pods using a PVC on the Shared Filesystem storage class remain in ContainerCreating for an extended period, or a terminating pod never fully clears.
• kubectl describe pod <pod-name> shows repeated FailedMount events.
• The stall is intermittent and often node-specific, some nodes never see it, others hit it repeatedly.
• Once resolved, the issue tends to self-heal on a later retry rather than requiring manual intervention.

You may see one or more of the following in pod events or CSI driver logs:

MountVolume.SetUp failed for volume "<pvc-name>" : rpc error: code = DeadlineExceeded desc = context deadline exceeded

MountVolume.SetUp failed for volume "<pvc-name>" : rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL

MountVolume.SetUp failed for volume "<pvc-name>" : rpc error: code = Internal desc = failed to publish volume <volume-id>: failed to mount volume at target path <target-path>: mount failed: exit status 255
Mounting command: mount
Mounting arguments: -t nfs -o vers=3,nconnect=16,spread_reads,spread_writes,remoteports=<vip-range> <vip>:/volumes/<volume-id> <target-path>
Output: mount: mounting <vip>:/volumes/<volume-id> on <target-path> failed: Resource busy

ℹ️ Note: The Resource busy (EBUSY) error can appear even when the underlying volume mounted successfully. It's a symptom of a retry racing the original mount attempt, not necessarily a failed mount. See Cause below.

Instructions

If you're actively seeing a pod stuck in this state, the most useful thing you can do is capture diagnostics while it's still stuck — several of these commands only return meaningful output during an active stall, not after it clears.

Step 1: Identify the Affected Node

kubectl get pod <pod-name> -o wide

Step 2: Get a Shell on the Node

Via SSH, if available:

ssh <node-ip>

Or via kubectl if SSH isn't an option:

kubectl debug node/<node-name> -it --image=ubuntu

ℹ️ Note: kubectl debug node shares the node's process namespace by default, so the commands below will see the real host processes from either path.

Step 3: Capture Mount and Driver State

nfsstat -m
vastnfs-ctl status
dmesg -T | tail -100

nfsstat -m shows the mount point, the actual server endpoint in use, NFS version, mount options, and per-mount RTT/retransmit/ops stats in one shot. vastnfs-ctl status reports the VAST NFS client driver's status and version.

Step 4: Find the Stuck Mount Process

ps -eo pid,ppid,stat,wchan:32,cmd | grep -E 'mount\.nfs|[m]ount -t nfs'

This returns the PID (first column) of the blocked mount process. The result also tells you which phase is stuck: a process named mount.nfs means the mount is still in its initial handshake with the mount service (port 111). A process blocked inside the mount call itself, with no separate mount.nfs process, means it has moved past the handshake into establishing the NFS data connection (port 2049). The wchan column shows what kernel function it's parked in, capture whatever value you see.

Step 5: Capture the Stuck Process's State

cat /proc/<PID>/stack
cat /proc/<PID>/status
strace -f -tt -p <PID>

Let strace run for 10–15 seconds, then Ctrl-C. Since it's attaching to an already-running process, it'll show what the process is currently doing rather than a full trace from the start.

ℹ️ Note: strace only observes userspace system calls. It will show the process is blocked inside the mount(2) call, but not the underlying kernel-level network activity behind it. The next step covers that gap.

Step 6: Capture Outstanding Connections

ss -tanp '( dport = :2049 or dport = :111 )'

getent hosts <nfs-hostname-from-nfsstat-output>

This shows any connections or connection attempts on either port, along with the owning process. Port 111 traffic confirms the mount handshake phase; port 2049 confirms the NFS data connection phase, consistent with what Step 4 already indicated.

If you open a support ticket for this issue, attaching the output of all of the above will significantly speed up diagnosis.