How-To: Benchmark and Isolate Shared Disk (NFS) Read/Write Performance

Introduction

If a Shared Disk (NFS) feels slower than expected, this guide helps you measure its actual read and write performance and narrow down where the slowness is coming from — the disk, the client configuration, or the individual VM.

Shared Disk performance depends on a few client-side settings being in place. Following these steps and recording the results gives Crusoe Support the data needed to investigate quickly if you open a ticket.

Prerequisites

• The VAST NFS driver installed. Shared Disks mount over NFS using the VAST NFS driver. Check it with vastnfs-ctl status. If the command is not found, install it — see Setting up the VAST NFS driver.
• Recommended network and cache optimizations applied. The driver installer's --apply-network-optimizations flag sets the MTU to 9000 and applies ring-buffer tuning, and --apply-read-ahead-cache raises the NFS readahead cache from the Linux default of 128 KB to 16 MB. Both are recommended for best performance and should be applied before benchmarking.
• The Shared Disk mounted correctly. Confirm with findmnt -t nfs, and check provisioned capacity with df -h.

Instructions

Step 1: Know Your Expected Performance

Shared Disk performance scales with the disk's provisioned capacity, so compare your measured numbers against the target for your disk size rather than an absolute figure. The scaling rates are up to 200 MB/s read and 40 MB/s write per TiB, and up to 1.2k IOPS per TiB. Per-VM throughput is also bounded by the VPC network bandwidth allocated to that VM, so a single VM will not necessarily reach the disk's full aggregate target on its own. See Managing Shared Disks — Performance Profile.

Step 2: Benchmark Read and Write Throughput and IOPS

From within the mounted Shared Disk path, run the fio benchmarks documented for Shared Disks. Run each on the affected VM and record the results.

# write bandwidth
fio --name=my-job --group_reporting --time_based=1 --runtime=10s --ramp_time=5s --size 20G --numjobs=32 --ioengine=libaio --direct=1 --iodepth 8 --rw write --bs 1m

# read bandwidth
fio --name=my-job --group_reporting --time_based=1 --runtime=10s --ramp_time=5s --size 20G --numjobs=32 --ioengine=libaio --direct=1 --iodepth 8 --rw read --bs 1m

# write IOPS
fio --name=my-job --group_reporting --time_based=1 --runtime=10s --ramp_time=5s --size 20G --numjobs=32 --ioengine=libaio --direct=1 --iodepth 8 --rw write --bs 4k

# read IOPS
fio --name=my-job --group_reporting --time_based=1 --runtime=10s --ramp_time=5s --size 20G --numjobs=32 --ioengine=libaio --direct=1 --iodepth 8 --rw read --bs 4k

Step 3: Isolate the Pattern

The shape of the slowdown points to where the problem is. Use the following comparisons to narrow it down.

3a. Compare Reads Against Writes

If writes meet the expected target but reads are significantly slower on the same disk and VM, the issue is specific to the read path rather than the disk or your mount configuration. A symmetric slowdown (both reads and writes low) more often points to the client setup in Prerequisites — recheck the driver, MTU, and readahead.

3b. Compare a Single Stream Against Many Streams

Re-run the read bandwidth test with --numjobs=1, then with the default --numjobs=32 (and optionally higher). If aggregate throughput rises as you add streams, the path is healthy and you were simply measuring a single-stream rate. If aggregate throughput does not improve as you add streams, the VM is hitting an aggregate ceiling — note this, as it is a useful diagnostic signal.

3c. Compare an Affected VM Against a Healthy VM

Run the same benchmark on a VM that performs well and one that does not, both against the same Shared Disk. If only specific VMs are slow while others reading the same disk are fast, the slowness follows the VM rather than the disk or the storage backend.

3d. Check Whether a Remount Changes Anything

Unmount and remount the Shared Disk on the affected VM, then re-run the read benchmark:

sudo umount <path-to-mount>
sudo mount -va

mount -va remounts entries defined in /etc/fstab. If the Shared Disk is not in /etc/fstab, remount it with the explicit mount command shown for the disk in the Crusoe Cloud Console instead. Note whether throughput changes after the remount and whether it gradually degrades again over time. This behavior is a useful data point for Support.

Step 4: (Optional) Check VM-to-VM Network Throughput

Because per-VM Shared Disk throughput is bounded by VPC network bandwidth, you can sanity-check the network independently of NFS by running iperf3 between two of your own VMs in the same VPC (one as server, one as client), including a parallel-stream run (-P). This confirms whether the VM is achieving expected network throughput outside of the storage path.

Step 5: What to Share When Opening a Ticket

If reads underperform the expected target after the checks above, include the following so Support can investigate without a round-trip:

• The Shared Disk's provisioned capacity and the expected target for that size.
• The fio read and write bandwidth and IOPS results (single-stream and multi-stream).
• Whether the slowdown is read-only or affects writes too.
• Whether it affects specific VMs or all VMs mounting the disk.
• Whether a remount changed throughput, and whether it degraded again over time.
• Output of vastnfs-ctl status, findmnt -t nfs, and df -h from an affected VM.

Additional Resources

• Managing Shared Disks
• Setting up the VAST NFS driver
• Shared Disks Metrics
• Shared Disks Troubleshooting