Introduction
When multiple processes on a VM write concurrently to the same file on a Crusoe Shared Disk (NFS), the VM may become unresponsive with a continuously repeating kernel soft lockup. One CPU becomes pinned in the kernel's NFS write path, and any process that subsequently writes to the affected file enters uninterruptible I/O ("D" state) and cannot be killed or timed out.
On Slurm clusters, this typically surfaces as slurmd becoming unresponsive and the controller marking the node down, with in-flight jobs lost.
The cause is a defect in versions of the VAST NFS client driver (vastnfs) earlier than 4.5.7. When a write error occurs, an internal page-group lock is not released on the error path. The next process contending for that lock spins on it indefinitely, because the owner has already exited the code path and will never release it. The condition does not self-resolve and persists until the VM is reset.
This is a driver-level defect. It does not indicate a problem with the shared disk itself, the storage backend, or your workload — although the trigger pattern, several workers appending to one shared output or checkpoint file, is common in multiprocessing and Slurm training jobs.
Prerequisites
- Crusoe Shared Disk (NFS) Mounted on One or More VMs
- VAST NFS Client Driver (
vastnfsDKMS Module) Installed, as Included in Curated Images Such asubuntu24.04-nvidia-slurm - Root or
sudoAccess on the Affected VM - Maintenance Window With the NFS Volume Unmounted and No Jobs Running (Driver Upgrade Only)
Symptoms
- The kernel log shows repeating soft lockup messages with a trace through the NFS write path, for example a CPU stuck in
nfs_page_set_headlockornfs_lock_and_join_requests. - Multiple worker processes are stuck in uninterruptible I/O ("D" state) writing to the same file on the NFS mount, and cannot be killed with
kill -9. - On Slurm clusters,
slurmdstops responding, the controller marks the node down, and jobs on the node are marked failed. -
lsmodshows the NFS stack as out-of-tree modules —nfs(OE),sunrpc(OE),rpcrdma(OE)— confirming thevastnfsdriver is in use rather than the in-tree client.
Instructions
Step 1: Confirm the Driver Version
Run the following on the affected VM:
vastnfs-ctl status
If the version shown is earlier than 4.5.7, this defect applies. Versions 4.5.7 and later contain the fix.
Step 2: Check for the Lockup Signature
Review the kernel log:
sudo dmesg | grep -i "soft lockup" sudo journalctl -k --since "-24h" | grep -A 20 "soft lockup"
A trace referencing nfs_page_set_headlock or nfs_lock_and_join_requests, with processes in "D" state, matches this issue.
Step 3: Capture Diagnostics Before Resetting
⚠️ Warning: Resetting the VM destroys the evidence. Capture the diagnostics below before you reset — without them the root cause cannot be confirmed after the fact.
Save the kernel journal and the process state:
sudo journalctl -k --since "-48h" > /tmp/kernel-log.txt ps -eo pid,tgid,ppid,stat,wchan:32,cmd | awk '$4 ~ /D/' > /tmp/dstate.txt
Attach both files to your support ticket. This allows Crusoe Support to confirm the root cause rather than infer it.
Step 4: Reset the VM
Processes in uninterruptible I/O cannot be killed, so a reset is the only way to clear the condition. Reset the VM from the Crusoe Cloud console or with the Crusoe CLI.
Step 5: Upgrade the Driver
A curated image containing the fixed driver is in progress. Until it ships, install the fixed driver manually. The current published package is 4.5.8, which includes the 4.5.7 fix along with later changes.
⚠️ Warning: This upgrade must be applied per node, and it needs a quiet maintenance window — the NFS volume must be unmounted and no jobs may be running on the node. Reloading the driver under active I/O can hang the node again.
Confirm the current version, then drain and unmount before touching the driver:
vastnfs-ctl status
Drain jobs from the node and unmount the NFS volume, then install and reload:
sudo dpkg -i vastnfs-dkms_4.5.8-vastdata_all.deb sudo vastnfs-ctl reload
Remount the NFS volume and confirm the new version is active:
vastnfs-ctl status
The package is available from the crusoe-nfs-support repository.
Resolution
After upgrading to vastnfs 4.5.7 or later, the error path releases the page-group lock correctly, and concurrent writes to a shared file no longer trigger the lockup. The node recovers normally and stays available under the same workload that previously hung it.
Until the upgrade is applied, you can avoid the trigger by having each worker process write to its own file rather than several workers appending to one shared file.
On Slurm clusters, it is also worth confirming requeue behavior so a future node failure does not silently kill jobs:
scontrol show config | grep Requeue
Check whether jobs are submitted with --no-requeue, or via srun, which does not requeue. Requeue does not preserve completed compute, but it does restart the work automatically instead of losing it.
ℹ️ Note: An updated curated image with the fixed driver is planned. This article will be updated when it is available.
Example
A Slurm training job runs 16 data-loader workers on one node, each appending progress records to a single log file on a shared NFS mount. Partway through the run the node stops responding to scontrol, and the controller marks it down. The job is lost.
On the serial console, dmesg shows a soft lockup repeating every few seconds with nfs_page_set_headlock in the trace, and ps shows all 16 workers in "D" state on the same file. kill -9 does nothing. vastnfs-ctl status reports version 4.5.6.
The team captures the journal and the D-state process list, resets the VM, and upgrades the driver to 4.5.8 during their next maintenance window. As an immediate mitigation before the window, they change the job to give each worker its own log file, and the lockup does not recur.
Related Articles
- High I/O Wait and Pressure Stall on a VM Accessing a Shared Disk
- FAQ: Slow Performance and Hanging on Shared Storage
- How-To Setup Serial-Console to Access Your VM