Introduction
When something goes wrong on a running VM, your first instinct is to SSH in and investigate. But SSH isn't always available — and in some failure scenarios, it never will be. Serial console provides an out-of-band connection to your VM through the Crusoe CLI, bypassing the network stack entirely. It's the recovery path of last resort when the normal management plane is gone.
There are two distinct situations where you'll reach for serial console, and they behave differently:
Emergency/Single-User Mode — the guest OS failed to complete a normal boot. A filesystem mount failure, a broken systemd unit, or a kernel panic can all cause the system to drop into an emergency shell before multi-user mode is reached. In this state, sshd never started, so the VM is completely unreachable over the network. Serial console drops you directly into a root shell — no password required — where you can inspect logs, fix the underlying issue, and boot normally.
Normal Boot, SSH Inaccessible — the VM booted successfully into multi-user mode, but SSH isn't reachable. A misconfigured network interface, a firewall rule blocking port 22, or a crashed sshd are common causes. The VM is running fine; you just can't get to it over the network. In this case you'll be presented with a normal OS login prompt over serial, and you'll need a local password to authenticate.
The second scenario is why you need to act now, before you're locked out. You cannot set a local password after SSH is gone. This article walks through how to set one while you still have access.
Prerequisites
- Crusoe CLI Installed and Configured
- Active SSH Access to the VM
Instructions
Step 1: Set a Local Console Password
SSH into your VM:
ssh ubuntu@<VM_PUBLIC_IP>
Set a password for the current user:
sudo passwd $(whoami)
You'll be prompted to enter and confirm the password. This credential is stored as a hashed value in /etc/shadow and persists across reboots.
New password: Retype new password: passwd: password updated successfully
Store the password somewhere secure — a shared team vault like 1Password is appropriate if multiple people may need console access.
Step 2: Understand the SSH Security Implications
Setting a local password has security implications beyond console access that you should understand before proceeding.
Password authentication is disabled by default. Crusoe VM images explicitly set PasswordAuthentication no in /etc/ssh/sshd_config.d/, so this password cannot be used for SSH logins out of the box. As long as you haven't changed this, your SSH attack surface is unchanged.
If you have enabled password authentication, this password is also valid for SSH. Check your sshd config if you're unsure:
sudo sshd -T | grep passwordauthentication
If the output is passwordauthentication yes, anyone who obtains this password can authenticate over SSH. Beyond credential theft, password authentication also exposes the VM to brute-force and dictionary attacks — this is a reality of operating any internet-facing infrastructure today, not specific to Crusoe. Automated scanners probe public IPs continuously for open SSH ports and attempt common and weak passwords at scale.
⚠️ If you must enable password authentication, use a long, randomly generated password. Weak or predictable passwords are routinely compromised by automated attacks within hours of a VM becoming internet-accessible. Additionally, consider restricting SSH access by source IP via your VPC security group rules, or eliminating public SSH exposure entirely by placing the VM behind a VPN such as Tailscale or WireGuard — both limit exposure regardless of authentication method.
Setting a password for the root user requires extra caution. If you are setting a root password for console access, we strongly recommend explicitly hardening your sshd config as a precaution:
⚠️ Add the following to
/etc/ssh/sshd_configif setting a root password. This ensures root SSH logins always require a key, regardless of whether password authentication is enabled now or in the future.
PermitRootLogin prohibit-password
Reload sshd to apply:
sudo systemctl reload ssh
💡 Note: on Ubuntu, the sshd service unit is named
ssh— bothsystemctl reload sshandsystemctl reload sshdwill work, butsshis the canonical name.
Step 3: Connect via Serial Console
From your local terminal, connect to the serial console using the Crusoe CLI:
crusoe compute vms serial-console \ --name <VM_NAME> \ --port-num <PORT>
ℹ️ Note:
--port-numspecifies which virtual serial port to connect to and accepts values 1–4, defaulting to 1 if not specified. Port 1 maps tottyS0— the primary console device that the kernel and systemd write boot output to, and where the login prompt appears. This is the right port for the vast majority of use cases. Ports 2–4 (ttyS1–ttyS3) support additional concurrent sessions on the same VM — useful when multiple engineers need simultaneous console access during a collaborative incident response without interrupting each other's session.
Once connected, what you see depends entirely on the state of the VM — if it has dropped into emergency mode you'll see kernel and systemd output, if it's fully booted you'll be presented with a login prompt. See the Example section below for what each scenario looks like in practice.
Example
Scenario 1: VM in Emergency Mode (Guest OS Failed to Boot)
If the guest OS failed to complete a normal boot — for example, due to a filesystem mount failure — the VM drops into an emergency shell. The serial console output will show the kernel boot log followed by a maintenance prompt:
Connecting to serial console... Press ENTER to continue and ~. to exit. [FAILED] Failed to mount /data. See 'systemctl status data.mount' for details. [DEPEND] Dependency failed for Local File Systems. [FAILED] Failed to mount /home. See 'systemctl status home.mount' for details. You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" to boot into default mode. Press Enter for maintenance (or press Control-D to continue): root@my-vm:~#
Press Enter and you'll be dropped into a root shell with no password prompt. From here you can inspect the failure with journalctl -xb, fix the underlying issue, and either exit to attempt a normal boot or systemctl reboot to restart the VM.
Scenario 2: VM Booted Normally, SSH Inaccessible
If the VM booted successfully but SSH is unreachable — a misconfigured network interface, a broken sshd config, or a firewall rule blocking port 22 — you'll see a normal login prompt:
Connecting to serial console... Press ENTER to continue and ~. to exit. Ubuntu 22.04.5 LTS my-vm.cloud.crusoe.ai ttyS0 my-vm login:
Log in with the username and password you set in Step 1. From here you can investigate and fix whatever is blocking SSH — check sshd status, review network interface configuration, or inspect firewall rules — without needing to stop or redeploy the VM.
Type ~. to exit the serial console session when done.
Next Steps
Once connected via serial console, common recovery actions include:
- Check sshd status:
sudo systemctl status ssh - Restart sshd after fixing a config error:
sudo systemctl restart ssh - Review sshd configuration:
sudo cat /etc/ssh/sshd_config - Check network interface state:
ip addr showandip route show - Inspect boot logs in emergency mode:
journalctl -xb - Attempt normal boot from emergency shell:
exitorsystemctl default - Reboot the VM:
sudo reboot