How-To: Debug SSH Access Issues Caused by Stale virtiofs Mount Entries

Introduction

Deprecated virtiofs entries left in /etc/fstab are one possible cause of SSH access failures after a VM reboot or stop/start. This guide walks through confirming the symptom, determining whether stale virtiofs entries are responsible, and recovering the VM if they are. If the checks below do not point to virtiofs, the SSH failure has a different cause and should be investigated separately.

How Stale virtiofs Entries Break SSH

virtiofs has been deprecated. When a virtiofs entry remains in /etc/fstab with default options (no nofail), it fails to mount at boot, which fails local-fs.target. The system then never reaches multi-user.target — the state where sshd starts — so the VM lands in emergency (maintenance) mode and SSH is refused.

If sshd is later started by hand in emergency mode, a second symptom appears: the port answers, but the NFS-backed /home is not mounted. Because a user's key lives in /home/<user>/.ssh/authorized_keys, key authentication for that user fails ("port open, but login fails"). Root login is unaffected, since root's key sits on the local root filesystem rather than on /home.

Prerequisites

• Serial Console Access to the Affected VM
• The VAST NFS Driver Available for Remounting Shared Disks (Required for Step 5)

Instructions

Step 1: Confirm the VM Is in Emergency Mode

Open the VM's serial console. A VM in emergency mode drops straight to a root shell with no login: prompt. A healthy VM instead shows a normal login: prompt. If you reach a normal login prompt and SSH still fails, the issue is not emergency mode — stop here and investigate other causes (network rules, key configuration, sshd config).

Step 2: Determine Whether Stale virtiofs Entries Are the Cause

From the serial-console root shell, check /etc/fstab for deprecated entries:

grep virtiofs /etc/fstab

Stale entries typically look like this — note the virtiofs filesystem type and the absence of nofail:

home /home virtiofs defaults 0 0
data /data virtiofs defaults 0 0
work /work virtiofs defaults 0 0

• If one or more virtiofs lines are returned, they are the likely cause of the failed boot — continue to Step 3.
• If no virtiofs lines are returned, virtiofs is not responsible. Review the rest of /etc/fstab for other failing mounts (run systemctl --failed and journalctl -b to identify what blocked local-fs.target), and investigate accordingly.

Step 3: Remediate the Stale virtiofs Entries

Once you have confirmed virtiofs entries are present, remove them and reboot cleanly:

1. Back up fstab: cp /etc/fstab /etc/fstab.bak
2. Comment out the virtiofs lines: sed -i '/ virtiofs /s/^/#/' /etc/fstab
3. Confirm the lines are commented: cat /etc/fstab
4. Reload systemd: systemctl daemon-reload
5. Reboot gracefully: systemctl reboot

⚠️ Use systemctl reboot rather than reboot -f. The graceful reboot flushes your /etc/fstab edit to disk; a forced reboot may not, leaving the VM to boot the old fstab and stay in emergency mode. Only fall back to reboot -f if systemctl reboot hangs — and run sync first.

Step 4: Verify Recovery

After the VM comes back, confirm:

• systemctl is-system-running returns running (not maintenance).
• systemctl status ssh shows active.
• The SSH port is reachable.

Step 5: Restore User Login

Removing the virtiofs lines is not enough on its own — if a mount such as /home holds the user's SSH keys, leaving it unmounted can still cost the user SSH access to the machine. After removing the stale entries, remount the affected paths using the supported VAST NFS driver so user access is preserved on the next boot. The driver must be installed first (confirm with vastnfs-ctl status); see Setting up the VAST NFS driver. Use the exact mount command shown for the disk in the Crusoe Cloud Console, and add a persistent entry to /etc/fstab (which already includes _netdev and nofail), for example:

# eu-iceland1-a
nfs.crusoecloudcompute.com:/volumes/<volume_uuid> <path-to-mount> nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,forcerdirplus,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,local_lock=none,remoteports=dns,spread_reads,spread_writes,_netdev,nofail,x-systemd.automount,x-systemd.idle-timeout=30 0 0

# all other regions
172.27.255.2:/volumes/<volume_uuid> <path-to-mount> nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,forcerdirplus,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,local_lock=none,spread_reads,spread_writes,_netdev,nofail,x-systemd.automount,x-systemd.idle-timeout=30,remoteports=172.27.255.2-172.27.255.17 0 0

For full steps on persisting shared-disk mounts, see Managing Shared Disks.

Prevention

A VM with stale virtiofs entries looks healthy until its next reboot or stop/start, then fails as described above. Migrate any remaining virtiofs entries to the supported VAST NFS driver proactively, and always include nofail (and _netdev) on remote/network filesystem entries in /etc/fstab so a future mount failure cannot block local-fs.target and drop the VM into emergency mode.

💡 Tip: Before editing /etc/fstab, always back up the file and ensure serial console access is enabled so the VM can be recovered if a bad entry breaks the boot.

Additional Resources

• Managing Shared Disks
• Setting up the VAST NFS driver
• Shared Disks Troubleshooting
• Accessing VMs — Serial Console Access
• How-To: Persist Shared Disk Mount Across Reboots