From 0ea90200b4776095b83e0747e7f86da21d3531ca Mon Sep 17 00:00:00 2001 From: Erik Simon Date: Fri, 24 Jul 2026 01:53:45 +0200 Subject: [PATCH] deploy: automate a full local reinstall, self-elevating and interactive-safe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ./scripts/deploy install localhost now branches on is_live_installer() (checks uname -n): outside a live installer it builds installer-iso, stages its kernel/initrd on the ESP and the iso file on a disk the caller picks (never auto-picked — the wrong disk here is destroyed mid-install), writes a systemd-boot one-shot findiso= entry with homelab.install= on the kernel cmdline, and does a real systemctl reboot (not kexec — terra's kexec-local hang is specifically in kexec's device-shutdown pass, a real ACPI reboot never runs that code at all). installer-iso gains homelab-auto-install.service: once homelab-checkout.service clones the repo, it reads homelab.install= back off /proc/cmdline and re-runs the identical deploy command itself, now genuinely inside the installer, so it takes the disko+nixos-install branch instead of preparing again. The whole reinstall is one command and unattended after the first reboot. Also: every root-requiring path (kexec-local, the new prepare-and-reboot branch, the disko+nixos-install branch) self-elevates via a require_root() helper that re-execs the original invocation under sudo -E, instead of dying and asking the caller to prefix sudo themselves. Uses an absolute script path captured before the script's own cd, so the re-exec is correct regardless of how it was invoked. Co-Authored-By: Claude Sonnet 5 --- CLAUDE.md | 17 +++++- README.md | 77 +++++++++++++------------- flake.nix | 27 ++++++++++ scripts/deploy | 143 +++++++++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 222 insertions(+), 42 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 5478638..47c0636 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -114,7 +114,22 @@ kept its ssh host key. Run it after ANY change to the kexec paths. cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT): discrete AMD GPUs are known to hang during kexec's device-shutdown pass with no clean handoff before the jump — same class of issue as jupiter's `reboot=pci` workaround, just fatal here instead of - slow. Use the USB installer path instead (README's "First install on terra" section). + slow. Use `./scripts/deploy install terra localhost` instead (README's "First install + on terra" section) — it detects it isn't inside a live installer yet and reboots via + a real `systemctl reboot` + systemd-boot one-shot `findiso=` entry, not kexec. +- **`./scripts/deploy install localhost`'s behavior depends on `uname -n`** + (`is_live_installer()`): on a real running OS it builds `installer-iso`, stages it + locally, and reboots into it (`local_install_prepare_and_reboot()`); only inside + `nixos-installer` (kexec) or `homelab-installer` (installer-iso) does it actually run + disko + `nixos-install`. `installer-iso`'s `homelab-auto-install.service` closes the + loop: it reads `homelab.install=` back off `/proc/cmdline` (set by the prepare + step) and re-runs the identical command itself once `homelab-checkout.service` has + cloned the repo — the whole reinstall is one command and unattended after the first + reboot. It always ASKS where to stage the iso file (never auto-picks — the wrong disk + here is destroyed mid-install) and refuses if that turns out to be the disk + `disk-config.nix` is about to wipe; `HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for + scripted use. Both this and `kexec-local` self-elevate via `sudo` (`require_root()`) + rather than requiring you to prefix the command yourself. - **disko wipes only the OS disk** named in `hosts//disk-config.nix`; data disks are plain `fileSystems` in `configuration.nix`. - `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the diff --git a/README.md b/README.md index c1cbf93..f8efd27 100644 --- a/README.md +++ b/README.md @@ -84,14 +84,15 @@ an installer, partitions via disko, installs. Manual alternative (USB ISO): boot installer, `disko` the disk, then `nixos-install --flake .#jupiter`. -## First install on terra — USB installer (replacing CachyOS) +## First install on terra — no-USB findiso reinstall (replacing CachyOS) terra is a Ryzen 9 5900X / Radeon RX 6800 XT desktop, currently running -CachyOS. Everything is already prepped in this repo: real OS-disk id in -`disk-config.nix`, real login pubkey in `common.nix`, terra's age recipient in -`.sops.yaml`, its host key pre-generated at `~/.config/homelab/terra/`, and -`secrets/terra.yaml` already holds real `darman_password` / `tailscale_authkey` -values. Nothing to fill in — just run it. +CachyOS with a writable root and systemd-boot. Everything is already prepped +in this repo: real OS-disk id in `disk-config.nix`, real login pubkey in +`common.nix`, terra's age recipient in `.sops.yaml`, its host key +pre-generated at `~/.config/homelab/terra/`, and `secrets/terra.yaml` already +holds real `darman_password` / `tailscale_authkey` values. Nothing to fill +in — just run it. > ⚠️ **`./scripts/deploy kexec-local` does NOT work on terra — do not use it.** > Confirmed on real hardware: the jump hangs completely (kexec's own @@ -101,10 +102,9 @@ values. Nothing to fill in — just run it. > a slow jump). Near-certainly amdgpu: discrete AMD GPUs are known to hang > during kexec's device-shutdown pass with no clean way to hand control back > before the jump — same class of issue as jupiter's `reboot=pci` warm-reboot -> workaround, just fatal here instead of merely slow. Untried mitigations -> (unbind/`rmmod amdgpu` from a bare VT first, kernel params) might work but -> aren't worth the risk of repeating a hard hang on a daily-driver box for a -> one-time install — use the USB path below instead. +> workaround, just fatal here instead of merely slow. The path below instead +> triggers a real ACPI reboot through firmware POST — a materially different +> code path that never runs kexec's device-shutdown pass at all. > ⚠️ The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The > dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs @@ -112,32 +112,37 @@ values. Nothing to fill in — just run it. > `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have > changed since `disk-config.nix` was written. -1. Build the installer ISO and write it to a USB stick (from your laptop — - this only needs to be done once; the ISO clones current `master` fresh on - every boot, so the same stick stays useful indefinitely): - ``` - nix build .#nixosConfigurations.installer-iso.config.system.build.isoImage - sudo dd if=result/iso/*.iso of=/dev/sdX bs=4M status=progress conv=fsync - ``` -2. Boot terra from the USB stick. It comes up with sshd + DHCP + your laptop - pubkey authorized for root, hostname `homelab-installer`, and a - `homelab-checkout.service` that clones the (public) repo to - `/root/homelab` once network is up. If you ssh in before that finishes, - `systemctl status homelab-checkout` tells you; retry with - `systemctl restart homelab-checkout` if DHCP was still coming up at boot. -3. SSH in (or use the physical console) and run the install directly: - ``` - cd /root/homelab - ./scripts/deploy install terra localhost - ``` - `localhost`/`127.0.0.1` skips nixos-anywhere/ssh and runs disko + - `nixos-install` directly against `/mnt`. Ships terra's pre-generated host - key so `/run/secrets/*` decrypts on boot #1. -4. Reboot into NixOS (remove the USB stick first). Then, same as any other - host: - ``` - ssh darman@terra sudo -v # DO NOT SKIP — see below - ``` +One command does the whole thing — no need to `sudo` it yourself, it +self-elevates: +``` +./scripts/deploy install terra localhost +``` +`scripts/deploy` detects it isn't already inside a live installer (checks +`uname -n`) and instead: builds `installer-iso`'s kernel + initrd + iso image, +asks where to stage the iso file (never auto-picks — the wrong disk here is +destroyed mid-install; `HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for +scripted use), refuses if that resolves to the same disk `disk-config.nix` is +about to wipe, copies the kernel/initrd to the ESP, writes a systemd-boot +**one-shot** entry (`bootctl set-oneshot` — reverts to the normal default +after this one boot, even if it fails) with `findiso=` + `homelab.install=terra` +on its kernel cmdline, and reboots — a real `systemctl reboot`, not kexec. + +The booted installer clones the repo (`homelab-checkout.service`, needs +network — it's public now, no credentials involved) and then +`homelab-auto-install.service` reads `homelab.install=terra` back off +`/proc/cmdline` and re-runs the exact same `./scripts/deploy install terra +localhost` itself — now genuinely inside the installer, so it takes the +disko + `nixos-install` branch instead of preparing again. Ships terra's +pre-generated host key first so `/run/secrets/*` decrypts on boot #1. The +whole thing is unattended after the initial reboot; ssh into +`homelab-installer` (same pubkey as the ISO everywhere else) to watch +progress — `journalctl -u homelab-checkout -u homelab-auto-install -f`. + +When it's done, reboot again into the freshly installed NixOS. Then, same as +any other host: +``` +ssh darman@terra sudo -v # DO NOT SKIP — see below +``` `darman` is created with `mutableUsers = true`, so `/etc/shadow` is written **once**. If the sops secret wasn't readable at that moment the account gets diff --git a/flake.nix b/flake.nix index c36e4d4..53a1392 100644 --- a/flake.nix +++ b/flake.nix @@ -205,6 +205,33 @@ git clone --depth 1 https://git.mgaction.town/darman/homelab.git /root/homelab ''; }; + + # Finishes a local_install_prepare_and_reboot() run (scripts/deploy) + # unattended: that function stages this ISO, points a systemd-boot + # one-shot entry at it with `homelab.install=` on the kernel + # cmdline, and reboots. Once booted here, this re-runs the exact same + # `./scripts/deploy install localhost` command — now genuinely + # inside the installer (hostname homelab-installer), so is_live_installer + # takes the disko+nixos-install branch instead of preparing again. + # A manual boot of this ISO with no such cmdline param is a no-op. + systemd.services.homelab-auto-install = { + description = "Auto-run the homelab install if homelab.install= was passed on the kernel cmdline"; + after = [ "homelab-checkout.service" ]; + requires = [ "homelab-checkout.service" ]; + wantedBy = [ "multi-user.target" ]; + path = [ pkgs.gnugrep ]; + serviceConfig.Type = "oneshot"; + script = '' + cfg=$(grep -o 'homelab\.install=[^ ]*' /proc/cmdline | cut -d= -f2 || true) + if [ -z "$cfg" ]; then + echo "no homelab.install= on the kernel cmdline — nothing to auto-install" + exit 0 + fi + echo "auto-installing $cfg (homelab.install= on the kernel cmdline)" + cd /root/homelab + exec ./scripts/deploy install "$cfg" localhost + ''; + }; }) ]; }; diff --git a/scripts/deploy b/scripts/deploy index c8851b3..f4959ff 100755 --- a/scripts/deploy +++ b/scripts/deploy @@ -20,8 +20,23 @@ # ./deploy install first install. Wipes the OS disk. Ships the # host's sops key. =localhost/127.0.0.1 # skips nixos-anywhere/ssh and runs disko + -# nixos-install directly against /mnt (use -# after `kexec-local`, or on a live ISO). +# nixos-install directly against /mnt — but +# ONLY once actually inside a live installer +# (hostname nixos-installer, from kexec, or +# homelab-installer, from installer-iso). +# Run from the REAL running OS instead (e.g. +# a box where kexec-local doesn't work), +# it builds installer-iso, stages its +# kernel/initrd on the ESP + the iso file on +# a non-OS-disk partition, sets a systemd-boot +# one-shot entry with homelab.install= +# on its kernel cmdline, and reboots — a real +# ACPI reboot, not a kexec jump. The booted +# installer's homelab-auto-install.service +# reads that cmdline param and re-runs this +# exact command itself once its repo checkout +# (homelab-checkout.service) succeeds, finishing +# the install unattended. See CLAUDE.md. # ./deploy switch rebuild + activate on a running host. # ./deploy boot stage for next boot, don't activate now. # ./deploy test activate without adding a boot entry. @@ -46,8 +61,16 @@ set -euo pipefail shopt -s nullglob +# Captured before anything shifts/parses $@, so require_root() below can +# re-exec the ORIGINAL invocation under sudo — inside a function, "$@"/"$1" +# refer to the function's own args (empty here), not the script's, so this +# has to be a global array instead of relying on positional-parameter scoping. +SCRIPT_ARGS=("$@") + # Locate the repo root (flake dir) regardless of where this script lives on disk. -SCRIPT_DIR="$(cd "$(dirname "$(realpath "$0")")" && pwd)" +SCRIPT_PATH="$(realpath "$0")" # absolute — "$0" itself may be relative, + # and require_root() re-execs after cd "$REPO" +SCRIPT_DIR="$(dirname "$SCRIPT_PATH")" REPO="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || dirname "$SCRIPT_DIR")" cd "$REPO" export PATH="/nix/var/nix/profiles/default/bin:$PATH" @@ -56,6 +79,15 @@ die() { echo "error: $*" >&2; exit 1; } need() { command -v "$1" >/dev/null 2>&1 || die "missing required tool: $1"; } +# Self-elevate instead of dying: re-exec this exact invocation under sudo. +# -E preserves the environment (HOMELAB_* overrides, Proton Pass vault vars) +# across the re-exec. A no-op once already root. +require_root() { + [ "$(id -u)" = 0 ] && return 0 + echo ">> $1 needs root — re-executing under sudo" >&2 + exec sudo -E -- "$SCRIPT_PATH" "${SCRIPT_ARGS[@]}" +} + # Exactly one path matching a glob, or die. `ls glob | head -1` silently yields # an empty string when nothing matches (head exits 0, so set -e never fires) and # the failure only surfaces later as a confusing tar/dd error. @@ -92,6 +124,99 @@ kexec_artifacts() { fi } +# True inside one of the throwaway live-installer environments this repo +# produces (kexec's nixos-installer, or installer-iso's homelab-installer) — +# i.e. `install localhost` should wipe/install right here. False on +# any real running OS, where the same command instead means "prepare and +# reboot into an installer for THIS box" (see local_install_prepare_and_reboot). +is_live_installer() { + case "$(uname -n)" in + nixos-installer | homelab-installer) return 0 ;; + *) return 1 ;; + esac +} + +# `install localhost` run on a REAL running OS (not already inside a +# live installer): builds installer-iso, stages its kernel/initrd + iso file +# locally, points a systemd-boot one-shot entry at them with +# homelab.install= on the kernel cmdline, and reboots — a real ACPI +# reboot through firmware POST, deliberately NOT a kexec jump (see terra's +# kexec-local gotcha in CLAUDE.md). The booted installer's +# homelab-auto-install.service reads that cmdline param and re-runs this exact +# `install localhost` command itself (now genuinely inside the +# installer) once homelab-checkout.service has fetched the repo, finishing the +# job unattended. +local_install_prepare_and_reboot() { + local config="$1" + require_root "preparing a local reinstall" + [ -d /sys/firmware/efi ] || die "not booted UEFI — the one-shot boot entry needs systemd-boot" + need bootctl + need nix + need lsblk + need findmnt + + # No default/auto-picked location — the wrong disk here is destroyed + # mid-install (see the OS-disk check below), so this always asks rather + # than guessing. HOMELAB_INSTALLER_STAGE_DIR skips the prompt for scripted + # use, but is otherwise just as explicit a choice as typing it in. + local stagedir="${HOMELAB_INSTALLER_STAGE_DIR:-}" + if [ -z "$stagedir" ]; then + echo ">> currently mounted filesystems:" + lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT + read -rp ">> path to stage the installer iso on (must NOT be on the OS disk being wiped): " stagedir + fi + [ -n "$stagedir" ] || die "no staging path given" + [ -d "$stagedir" ] \ + || die "staging dir $stagedir doesn't exist — needs to be an existing partition that is NOT the OS disk being wiped" + + # Refuse if the staging partition turns out to live on the same disk + # disko is about to wipe — the iso file (and the running installer + # loopback-mounted from it) would be destroyed mid-install. + local osdisk osdisk_real stage_src stage_pkname stage_disk_real + osdisk="$(nix eval --raw ".#nixosConfigurations.$config.config.disko.devices.disk" \ + --apply 'd: (builtins.head (builtins.attrValues d)).device' 2>/dev/null)" \ + || die "couldn't read the OS disk device from hosts/$config/disk-config.nix" + osdisk_real="$(readlink -f "$osdisk")" + stage_src="$(findmnt -no SOURCE --target "$stagedir")" \ + || die "$stagedir doesn't resolve to a mounted filesystem" + stage_pkname="$(lsblk -no PKNAME "$stage_src" 2>/dev/null || true)" + if [ -n "$stage_pkname" ]; then + stage_disk_real="$(readlink -f "/dev/$stage_pkname")" + [ "$stage_disk_real" = "$osdisk_real" ] \ + && die "$stagedir is on the OS disk ($osdisk) that install would wipe — re-run and pick a different disk" + fi + + echo ">> building installer-iso (kernel + initrd + iso image)" + local kernel initrd isodir iso mnt_point iso_relpath + kernel="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.kernel)/bzImage" + initrd="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.initialRamdisk)/initrd" + isodir="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.isoImage)" + iso="$(one_match 'installer iso' "$isodir"/iso/*.iso)" + + echo ">> staging kernel/initrd on the ESP, iso image on $stagedir" + install -Dm644 "$kernel" /boot/homelab-installer/bzImage + install -Dm644 "$initrd" /boot/homelab-installer/initrd + install -Dm644 "$iso" "$stagedir/homelab-installer.iso" + + # findiso= is a path relative to whatever partition the initrd finds it on + # (it mounts every blkid-visible partition looking for it) — not necessarily + # relative to `/`, if $stagedir is a subdirectory of a bigger filesystem + # rather than a mountpoint itself. + mnt_point="$(findmnt -no TARGET --target "$stagedir")" + iso_relpath="${stagedir#"$mnt_point"}/homelab-installer.iso" + + cat >/boot/loader/entries/homelab-installer.conf <> one-shot boot into the installer, then rebooting — it will finish this install itself" + bootctl set-oneshot homelab-installer.conf + systemctl reboot +} + # Flakes only see git-tracked files: an untracked hosts// is silently # invisible to `nix build`/`nixos-install`, which then fails obscurely or builds # a stale config. Check before doing anything destructive. @@ -240,7 +365,7 @@ case "$cmd" in # This is a one-way trip on the machine you are typing at, so every check # that can fail is done BEFORE the point of no return, and nothing that the # jump depends on is cleaned up behind it (see the trap discussion below). - [ "$(id -u)" = 0 ] || die "kexec-local must run as root (sudo ./deploy kexec-local)" + require_root "kexec-local" assume_yes="" [ "${2:-}" = "--yes" ] && assume_yes=1 @@ -352,10 +477,18 @@ case "$cmd" in require_tracked "$config" if [ "$host" = "localhost" ] || [ "$host" = "127.0.0.1" ]; then + if ! is_live_installer; then + # Not already inside a live installer: build one, stage it, one-shot + # boot into it, and let it finish this exact command itself. See + # local_install_prepare_and_reboot above and CLAUDE.md. + local_install_prepare_and_reboot "$config" + exit 0 + fi + # Local install: no ssh, no nixos-anywhere. Run after `kexec-local` (or # from a live ISO) so /mnt is free to wipe — this IS the box, no second # machine in the loop, so skip straight to disko + nixos-install. - [ "$(id -u)" = 0 ] || die "local install must run as root" + require_root "local install" [ -f "./hosts/$config/disk-config.nix" ] || die "no ./hosts/$config/disk-config.nix" echo ">> disko .#$config onto this box's OS disk (WILL be wiped)"