Introduction
This article is a companion to How-To Rescue a VM That Doesn't Have Serial Console Setup, which describes the procedure for attaching a broken VM's boot disk to a healthy "rescue VM" to recover access or data. Follow that article for the procedure itself; it is not repeated here.
This article covers one specific hazard the rescue guide does not: what happens when the rescue VM and the broken VM were created from the same image. That is a common situation, since teams tend to standardize on one image.
Crusoe Support has seen real recovery cases where this turned a healthy rescue VM into a second broken VM.
ℹ️ Note: If your rescue VM uses a different image than the broken VM, this hazard does not apply and the rescue guide alone is sufficient.
Prerequisites
- A Broken VM Whose Boot Disk Needs Recovering
- A Separate Rescue VM, Running and Reachable Over SSH
- Console or Crusoe CLI Access to Attach and Detach Disks
- Any Automated Stop/Start Tooling Paused for the Duration of the Recovery
How the Conflict Happens
Disks created from the same image are clones and carry identical internal identifiers: the same partition GUIDs, filesystem UUIDs, and filesystem labels (for example, cloudimg-rootfs on Ubuntu images). During boot, the bootloader and OS locate the root filesystem by searching for these identifiers. With two disks answering to the same identity, the OS may select either one.
The consequence: if the rescue VM is restarted while the broken VM's boot disk is attached, it may boot from the broken disk instead of its own. It will then show as Running in the console while being unreachable over SSH and silent on the serial console — the same symptoms as the VM you were rescuing.
ℹ️ Note: This selection happens only at boot time. A running VM is not affected by having such a disk attached. Every rule below follows from that one fact.
Instructions
Apply these rules in addition to the steps in the rescue guide.
- Confirm the rescue VM is running and reachable over SSH before the disk is attached. Never have the disk attached across a start — the boot with two identical disks present is where the wrong one can be picked. Verifying SSH first also ensures any pre-existing issue on the rescue VM isn't later mistaken for a boot conflict.
-
Do not stop, restart, reset, or reboot the rescue VM while the disk is attached. This includes reboots from inside the OS (
sudo reboot) and, critically, any automated stop/start tooling or orchestration managing the VM. -
Mount by device node or device path, never by label or UUID. The rescue guide's mount examples (
/dev/vdb1, or/dev/disk/by-id/virtio-<DISK_SERIAL>-part1) are safe.LABEL=andUUID=mounts are not, since both disks share them. Mount read-only (-o ro) for recovery work. - Keep the attachment short. Attach, copy what you need, unmount, detach — in one session where possible. Every hour the disk stays attached is an hour in which an unplanned restart can trigger the problem.
- Detach the disk, per the rescue guide, before any restart. After the detach, the VM can be stopped and started freely again.
⚠️ Warning: Rule 2 is the one that actually bites. A scheduled or orchestrated stop/start is the most common way this happens, because nobody chose to reboot — pause or exclude that automation before you attach the disk.
Recommendations
💡 Tip: The simplest prevention is to create the rescue VM from a different image than the broken VM — for example, plain
ubuntu24.04when recovering anubuntu22.04-based disk. Different images do not share filesystem identities, which removes the hazard entirely and makes every rule above unnecessary.
If you would rather work from a copy than against the live disk, image the attached disk to a file and detach the original as soon as the copy completes:
sudo dd if=/dev/disk/by-id/virtio-<DISK_SERIAL> \ of=/mnt/backup/rescue-disk.img \ bs=64M conv=noerror,sync status=progress
⚠️ Warning: The destination must be a file path, as above — never another block device. Writing the clone to a second disk (
of=/dev/vdc) reproduces the duplicate filesystem identity on a third disk and recreates exactly the hazard this article is about.conv=noerror,synclets the copy continue past unreadable sectors on a damaged source disk instead of aborting.
Once a full copy exists, the original can be detached and the risk window is closed.
Recovery If It Has Already Happened
Symptoms: after a restart with the same-image disk attached, the rescue VM shows Running, SSH connections hang rather than being refused, and the serial console shows no output.
To recover, stop the rescue VM, detach the broken VM's boot disk, and start the rescue VM again. With only its own boot disk present, it will boot correctly.
ℹ️ Note: If the disk was attached by Crusoe Support as part of the rescue procedure, contact Support to perform the detach — some operations are restricted while a rescue attachment is active.
Example
A team standardizes every VM on ubuntu22.04. One VM stops responding, so they follow the rescue guide: spin up a second VM from the same ubuntu22.04 image, attach the broken VM's boot disk, and mount it read-only to pull logs.
Partway through, a nightly cost-saving job stops and starts idle VMs — including the rescue VM. It comes back up showing Running, but SSH hangs and the serial console is silent. The rescue VM booted from the broken disk, because both disks answer to cloudimg-rootfs.
They stop the rescue VM, detach the broken disk, and start it again — it boots normally. On the second attempt they build the rescue VM from ubuntu24.04 instead and exclude it from the nightly job, and the recovery completes without incident.
FAQs
Does attaching the disk to a running VM risk my data?
No. Attaching and mounting read-only does not modify the disk. The hazard affects which disk the VM boots from, not the contents of either disk.
Does this apply to data disks?
No. Ordinary data disks — created blank rather than from an image — have no boot entries and no duplicated identities. This hazard is specific to boot and OS disks created from images.