How-To Fix Shared Disks Mounting Read-Only Despite the rw Option

Introduction

If your shared disk mounts read-only even though your mount command requests rw — and writes fail with Read-only file system — this guide explains why it happens and how to fix it.

Crusoe shared disks are mounted over NFSv3 using the vastnfs client driver, which provides the multi-connection performance options (nconnect, remoteports, spread_reads, spread_writes) recommended for shared disks.

A regression in vastnfs driver versions 4.5.x prior to 4.5.8 — including 4.5.1, bundled with Ubuntu 24.04 curated images — causes new mounts to silently inherit the read-only flag from any NFS mount from the same storage endpoint that is already active on the VM. This happens even for a completely different volume, and regardless of the rw option.

The failure mode is confusing to diagnose because everything else looks healthy: the same volume mounts read-write on any other VM, the storage backend reports no issues, and nothing is logged to dmesg or journalctl. Unmounting and remounting can appear to fix it temporarily, but the problem returns whenever a read-only mount is active first — typically after a reboot where /etc/fstab mounts a read-only volume early.

This guide shows how to confirm you are hitting this issue, then gives two solutions: upgrading the driver (permanent fix) or adding the nosharecache mount option (immediate workaround).

Prerequisites

• Shared Disk Attached to the VM
• Sudo Access on the VM
• Ubuntu 24.04 Crusoe Curated Image, or Any vastnfs Driver Version Below 4.5.8

Instructions

Work through the two diagnostic steps first, then apply either Solution 1 or Solution 2 — they are alternatives, not sequential steps.

Step 1: Confirm the Symptom

Mount the shared disk requesting read-write, then inspect the negotiated options and attempt a write:

sudo mount -t nfs -o rw,vers=3,nconnect=16,spread_reads,spread_writes,remoteports=<NFS_SERVER_IP_RANGE> <NFS_SERVER_IP>:/volumes/<VOLUME_UUID> /mnt/data
findmnt -T /mnt/data -o OPTIONS
sudo touch /mnt/data/writetest

Affected output — the kernel negotiated ro even though the command requested rw:

OPTIONS
ro,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,rdirplus=force,forcerdirplus,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,...
touch: cannot touch '/mnt/data/writetest': Read-only file system

ℹ️ Note: The ro in the client mount options is the distinguishing signature of this issue. If the mount shows rw but writes fail with Permission denied, the cause is server-side export permissions — a different problem not covered by this guide.

Step 2: Confirm the Trigger and Driver Version

Check for an existing read-only NFS mount from the same storage endpoint, and check the driver version:

findmnt -t nfs
vastnfs-ctl status

You are hitting this issue if both of the following are true:

• findmnt -t nfs shows at least one active mount from the same NFS server IP with ro in its options — for example, a reference dataset deliberately mounted read-only via /etc/fstab.
• vastnfs-ctl status reports a version below 4.5.8-vastdata, for example:

version: 4.5.1-vastdata
kernel modules: sunrpc rpcrdma compat_nfs_ssc lockd nfs_acl nfs nfsv3
services: rpcbind.socket rpcbind
rpc_pipefs: /run/rpc_pipefs

Solution 1: Upgrade the vastnfs Driver (Permanent Fix)

The regression is fixed in vastnfs 4.5.8. Upgrading requires briefly unmounting all NFS volumes or rebooting the VM, so plan a short maintenance window.

Follow How-To Safely Upgrade the VAST NFS Driver on a Crusoe VM for the full procedure. After the upgrade, verify with the read-only mount still active:

vastnfs-ctl status
sudo mount -t nfs -o rw,vers=3,nconnect=16,spread_reads,spread_writes,remoteports=<NFS_SERVER_IP_RANGE> <NFS_SERVER_IP>:/volumes/<VOLUME_UUID> /mnt/data
findmnt -T /mnt/data -o OPTIONS
sudo touch /mnt/data/writetest

The mount options should show rw and the write should succeed, with no special mount options required.

Solution 2: Add the nosharecache Mount Option (Workaround)

If you cannot schedule a driver upgrade immediately, add nosharecache to the mount options of every volume that must be read-write. This forces the kernel to create an independent superblock for the mount instead of reusing shared state, so the requested rw takes effect:

sudo mount -t nfs -o rw,nosharecache,vers=3,nconnect=16,spread_reads,spread_writes,remoteports=<NFS_SERVER_IP_RANGE> <NFS_SERVER_IP>:/volumes/<VOLUME_UUID> /mnt/data

To persist across reboots, add it to the corresponding /etc/fstab entry:

<NFS_SERVER_IP>:/volumes/<VOLUME_UUID>  /mnt/data  nfs  rw,nosharecache,vers=3,nconnect=16,spread_reads,spread_writes,remoteports=<NFS_SERVER_IP_RANGE>,_netdev,nofail  0 0

💡 Tip: Leaving nosharecache in place after upgrading the driver causes no harm, so you do not need to revert your fstab changes once you upgrade.

Example

A team runs CPU VMs whose /etc/fstab mounts a shared reference dataset read-only at boot, alongside read-write volumes from the same storage endpoint. On these VMs, every newly created shared disk mounts read-only despite rw in the mount command, while identical disks mount read-write on VMs without the read-only fstab entry.

Following Step 2, the engineer finds the read-only reference mount active and vastnfs-ctl status reporting 4.5.1-vastdata. Solution 2 restores write access immediately, and the driver upgrade in Solution 1 is rolled out at the next maintenance window as the permanent fix.

VMs created from updated Crusoe curated images include the fixed driver and are not affected.

Related Articles

• How-To Safely Upgrade the VAST NFS Driver on a Crusoe VM
• VM Hangs With Kernel Soft Lockup When Multiple Processes Write to Shared Storage
• FAQ: Slow Performance and Hanging on Shared Storage

Additional Resources

• Setting Up the VAST NFS Driver