Mário Nogueira

System Recovery Procedure

(LLM used to help organize and translate this post)

Why

I had backups, but I had never proven that I could get from a dead disk back to a working laptop. Every night restic sends my data and configuration to Backblaze B2, and a healthcheck tells me when something is stale. A backup you have never restored is an assumption.

And the data is only part of the problem. After a failure I would have to remember the disk layout, the encryption setup, the package list, which files in /etc matter, and the order of it all, probably on a bad day and without my own notes, which would be on the dead machine.

So the goal was concrete: a procedure I could follow with nothing but a USB stick, a paper card and a network connection, and which I had actually run.

The constraints are the same as for the rest of my setup: security, privacy, minimalism, sovereignty and longevity. That meant POSIX shell, busybox tools, the stock Alpine installer, and no extra framework.

How

I split recovery into three layers, each simple on its own.

The bundle. A daily script on the laptop collects everything needed to rebuild it except the data: facts about the machine (disk layout, LVM, LUKS headers, the full package list), curated non-secret configuration, and the secrets (ssh host keys, WireGuard, restic credentials). It publishes atomically and verifies itself with checksums.

Building the stick

The stick is built by one script, make-stick.sh -i ISO TARGET. The target is either a whole-disk device for a real stick or a plain file path for a QEMU test image. Both go through the same code, so the test exercises what the real stick does.

The layout is deliberately boring. The stock Alpine extended ISO is written unmodified with dd; it is a hybrid ISO, so it boots on UEFI and BIOS. One extra partition is appended after the ISO's own partitions with sfdisk --append. That partition is a raw LUKS2 container holding a btrfs filesystem, with recovery/ (the scripts, the runbook, the paper card), bundles/<host>/ and a checksum file. The installer part holds no secrets, and nothing in the installer knows the payload exists.

The order of operations is what makes it safe:

  1. Check the tools, the ISO (the ISO 9660 signature and its published .sha256) and every bundle (against its own MANIFEST.sha256).
  2. Check the target: a whole disk, not a partition, nothing mounted, no holders, no larger than 256 GiB (a guard against pointing it at an internal disk), a warning if the device is not removable, and a typed confirmation of its path.
  3. Build the payload first, as a file: cryptsetup luksFormat (LUKS2), mkfs.btrfs, copy the files in, close it. Nothing has touched the target yet, so a failure here costs nothing.
  4. Write the ISO to the target and read it back to verify.
  5. Append the third partition and write the payload into it, again with a read-back check.

One quirk found in testing shapes the unlock step. The live installer sometimes mounts the boot medium through the whole disk (/dev/sda on /media/sda), and the kernel then refuses to open any partition of that disk ("in use"). So the card says to copy the 1.5 GiB payload partition into RAM with dd and open the copy with cryptsetup. It works in both cases, and nothing is ever written to the stick.

The recovery procedure

Recovery runs in three stages, each a short POSIX shell script on the stick. The stages are separated by reboots, so each one runs in the environment it needs: the installer for the first, the installed system for the other two.

Stage 1: base install (installer environment). 10_base_install.sh DISK refuses to run unless it is root, the target is a block device, the machine booted in UEFI mode and nothing on the disk is mounted. It shows fdisk -l for the target and makes me type the disk path. Then it runs the installer's own tool: ROOTFS=btrfs setup-disk -m sys -L -e -s 4096 DISK, which means LVM (-L), encryption (-e) and a 4 GiB swap volume. The result matches the old laptop: GPT, a 300 MB vfat /boot, LUKS, LVM group vg0 with lv_swap and lv_root, btrfs, GRUB for EFI. I choose a new passphrase. The installer closes everything at the end, so the script reopens the volumes and leaves the new system mounted on /mnt. In the QEMU run the kernel command line and fstab matched the real laptop's. The script stores no disk size: lv_root takes whatever is left after /boot and swap, so a different disk works as long as it is big enough for the data.

Stage 2: system (installed system, network up). After the reboot I unlock the stick again and run 20_restore_system.sh, first with -n for a dry run that only prints the package plan. For real it does this, in order:

Stage 3: data (restic). 30_restic.sh check proves the credentials and shows the latest snapshot per tag. restore [-n] [-t DIR] [-i PATH]... TAG wraps restic restore latest --tag TAG --host HOST, with a dry run and a confirmation showing the free space. The data snapshot (936 GiB) restores to its original path with numeric owners. The small config snapshot restores only paths under /home and /root onto /; its /etc goes to a side directory for comparison, never over the live system. A complete data restore writes a marker file. Finally activate puts the deferred cron and periodic jobs back and restarts crond, and it refuses to run without that marker. Activating earlier would store a snapshot of a half-empty machine in the real repository. An external-drive rsync is the planned fast path for the data, which is why the marker can also be written by hand.

Testing

I did not trust any of this until I had run it. I built a QEMU test bench with UEFI, a virtual NVMe disk and the stick as a USB drive, and followed the runbook step by step, writing down every deviation. That found real problems: files whose permissions had been stripped, a keymap missing from the bundle, a DHCP client that overwrote my DNS setup, and a console that sat silent until I pressed Enter.

Completeness check

The most useful fix was this one. The bundle now asks the package manager which files under /etc differ from their packages, and warns about any that are not in the bundle. It found five more files, including the PAM file for my screen locker, which would have locked me out.

What

The result is one folder, 02_Recovery, with about a dozen small files, and a procedure I have tested end to end in a virtual machine. On the real laptop I have booted the stick, unlocked it and confirmed that Wi-Fi, the NVMe disk and the network work.

Still open:

The main lesson is that the value was in the testing, not in the scripts. Every gap I found would have appeared on the worst possible day.