deploy: automate a full local reinstall, self-elevating and interactive-safe

./scripts/deploy install <config> localhost now branches on is_live_installer() (checks uname -n): outside a live installer it builds installer-iso, stages its kernel/initrd on the ESP and the iso file on a disk the caller picks (never auto-picked — the wrong disk here is destroyed mid-install), writes a systemd-boot one-shot findiso= entry with homelab.install=<config> on the kernel cmdline, and does a real systemctl reboot (not kexec — terra's kexec-local hang is specifically in kexec's device-shutdown pass, a real ACPI reboot never runs that code at all). installer-iso gains homelab-auto-install.service: once homelab-checkout.service clones the repo, it reads homelab.install= back off /proc/cmdline and re-runs the identical deploy command itself, now genuinely inside the installer, so it takes the disko+nixos-install branch instead of preparing again. The whole reinstall is one command and unattended after the first reboot. Also: every root-requiring path (kexec-local, the new prepare-and-reboot branch, the disko+nixos-install branch) self-elevates via a require_root() helper that re-execs the original invocation under sudo -E, instead of dying and asking the caller to prefix sudo themselves. Uses an absolute script path captured before the script's own cd, so the re-exec is correct regardless of how it was invoked. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-24 01:53:45 +02:00
parent fd8328d7b3
commit 0ea90200b4
4 changed files with 222 additions and 42 deletions
@@ -114,7 +114,22 @@ kept its ssh host key. Run it after ANY change to the kexec paths.
  cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT): discrete AMD GPUs are known
  to hang during kexec's device-shutdown pass with no clean handoff before the jump —
  same class of issue as jupiter's `reboot=pci` workaround, just fatal here instead of
-  slow. Use the USB installer path instead (README's "First install on terra" section).
+  slow. Use `./scripts/deploy install terra localhost` instead (README's "First install
+  on terra" section) — it detects it isn't inside a live installer yet and reboots via
+  a real `systemctl reboot` + systemd-boot one-shot `findiso=` entry, not kexec.
+- **`./scripts/deploy install <config> localhost`'s behavior depends on `uname -n`**
+  (`is_live_installer()`): on a real running OS it builds `installer-iso`, stages it
+  locally, and reboots into it (`local_install_prepare_and_reboot()`); only inside
+  `nixos-installer` (kexec) or `homelab-installer` (installer-iso) does it actually run
+  disko + `nixos-install`. `installer-iso`'s `homelab-auto-install.service` closes the
+  loop: it reads `homelab.install=<config>` back off `/proc/cmdline` (set by the prepare
+  step) and re-runs the identical command itself once `homelab-checkout.service` has
+  cloned the repo — the whole reinstall is one command and unattended after the first
+  reboot. It always ASKS where to stage the iso file (never auto-picks — the wrong disk
+  here is destroyed mid-install) and refuses if that turns out to be the disk
+  `disk-config.nix` is about to wipe; `HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for
+  scripted use. Both this and `kexec-local` self-elevate via `sudo` (`require_root()`)
+  rather than requiring you to prefix the command yourself.
 - **disko wipes only the OS disk** named in `hosts/<h>/disk-config.nix`; data disks are
  plain `fileSystems` in `configuration.nix`.
 - `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the
@@ -84,14 +84,15 @@ an installer, partitions via disko, installs.
 Manual alternative (USB ISO): boot installer, `disko` the disk, then
 `nixos-install --flake .#jupiter`.

-## First install on terra — USB installer (replacing CachyOS)
+## First install on terra — no-USB findiso reinstall (replacing CachyOS)

 terra is a Ryzen 9 5900X / Radeon RX 6800 XT desktop, currently running
-CachyOS. Everything is already prepped in this repo: real OS-disk id in
-`disk-config.nix`, real login pubkey in `common.nix`, terra's age recipient in
-`.sops.yaml`, its host key pre-generated at `~/.config/homelab/terra/`, and
-`secrets/terra.yaml` already holds real `darman_password` / `tailscale_authkey`
-values. Nothing to fill in — just run it.
+CachyOS with a writable root and systemd-boot. Everything is already prepped
+in this repo: real OS-disk id in `disk-config.nix`, real login pubkey in
+`common.nix`, terra's age recipient in `.sops.yaml`, its host key
+pre-generated at `~/.config/homelab/terra/`, and `secrets/terra.yaml` already
+holds real `darman_password` / `tailscale_authkey` values. Nothing to fill
+in — just run it.

 > ⚠️  **`./scripts/deploy kexec-local` does NOT work on terra — do not use it.**
 > Confirmed on real hardware: the jump hangs completely (kexec's own
@@ -101,10 +102,9 @@ values. Nothing to fill in — just run it.
 > a slow jump). Near-certainly amdgpu: discrete AMD GPUs are known to hang
 > during kexec's device-shutdown pass with no clean way to hand control back
 > before the jump — same class of issue as jupiter's `reboot=pci` warm-reboot
-> workaround, just fatal here instead of merely slow. Untried mitigations
-> (unbind/`rmmod amdgpu` from a bare VT first, kernel params) might work but
-> aren't worth the risk of repeating a hard hang on a daily-driver box for a
-> one-time install — use the USB path below instead.
+> workaround, just fatal here instead of merely slow. The path below instead
+> triggers a real ACPI reboot through firmware POST — a materially different
+> code path that never runs kexec's device-shutdown pass at all.

 > ⚠️  The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The
 > dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs
@@ -112,32 +112,37 @@ values. Nothing to fill in — just run it.
 > `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have
 > changed since `disk-config.nix` was written.

-1. Build the installer ISO and write it to a USB stick (from your laptop —
-   this only needs to be done once; the ISO clones current `master` fresh on
-   every boot, so the same stick stays useful indefinitely):
-   ```
-   nix build .#nixosConfigurations.installer-iso.config.system.build.isoImage
-   sudo dd if=result/iso/*.iso of=/dev/sdX bs=4M status=progress conv=fsync
-   ```
-2. Boot terra from the USB stick. It comes up with sshd + DHCP + your laptop
-   pubkey authorized for root, hostname `homelab-installer`, and a
-   `homelab-checkout.service` that clones the (public) repo to
-   `/root/homelab` once network is up. If you ssh in before that finishes,
-   `systemctl status homelab-checkout` tells you; retry with
-   `systemctl restart homelab-checkout` if DHCP was still coming up at boot.
-3. SSH in (or use the physical console) and run the install directly:
-   ```
-   cd /root/homelab
-   ./scripts/deploy install terra localhost
-   ```
-   `localhost`/`127.0.0.1` skips nixos-anywhere/ssh and runs disko +
-   `nixos-install` directly against `/mnt`. Ships terra's pre-generated host
-   key so `/run/secrets/*` decrypts on boot #1.
-4. Reboot into NixOS (remove the USB stick first). Then, same as any other
-   host:
-   ```
-   ssh darman@terra sudo -v          # DO NOT SKIP — see below
-   ```
+One command does the whole thing — no need to `sudo` it yourself, it
+self-elevates:
+```
+./scripts/deploy install terra localhost
+```
+`scripts/deploy` detects it isn't already inside a live installer (checks
+`uname -n`) and instead: builds `installer-iso`'s kernel + initrd + iso image,
+asks where to stage the iso file (never auto-picks — the wrong disk here is
+destroyed mid-install; `HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for
+scripted use), refuses if that resolves to the same disk `disk-config.nix` is
+about to wipe, copies the kernel/initrd to the ESP, writes a systemd-boot
+**one-shot** entry (`bootctl set-oneshot` — reverts to the normal default
+after this one boot, even if it fails) with `findiso=` + `homelab.install=terra`
+on its kernel cmdline, and reboots — a real `systemctl reboot`, not kexec.
+
+The booted installer clones the repo (`homelab-checkout.service`, needs
+network — it's public now, no credentials involved) and then
+`homelab-auto-install.service` reads `homelab.install=terra` back off
+`/proc/cmdline` and re-runs the exact same `./scripts/deploy install terra
+localhost` itself — now genuinely inside the installer, so it takes the
+disko + `nixos-install` branch instead of preparing again. Ships terra's
+pre-generated host key first so `/run/secrets/*` decrypts on boot #1. The
+whole thing is unattended after the initial reboot; ssh into
+`homelab-installer` (same pubkey as the ISO everywhere else) to watch
+progress — `journalctl -u homelab-checkout -u homelab-auto-install -f`.
+
+When it's done, reboot again into the freshly installed NixOS. Then, same as
+any other host:
+```
+ssh darman@terra sudo -v          # DO NOT SKIP — see below
+```

 `darman` is created with `mutableUsers = true`, so `/etc/shadow` is written
 **once**. If the sops secret wasn't readable at that moment the account gets
@@ -205,6 +205,33 @@
                  git clone --depth 1 https://git.mgaction.town/darman/homelab.git /root/homelab
                '';
              };
+
+              # Finishes a local_install_prepare_and_reboot() run (scripts/deploy)
+              # unattended: that function stages this ISO, points a systemd-boot
+              # one-shot entry at it with `homelab.install=<config>` on the kernel
+              # cmdline, and reboots. Once booted here, this re-runs the exact same
+              # `./scripts/deploy install <config> localhost` command — now genuinely
+              # inside the installer (hostname homelab-installer), so is_live_installer
+              # takes the disko+nixos-install branch instead of preparing again.
+              # A manual boot of this ISO with no such cmdline param is a no-op.
+              systemd.services.homelab-auto-install = {
+                description = "Auto-run the homelab install if homelab.install= was passed on the kernel cmdline";
+                after = [ "homelab-checkout.service" ];
+                requires = [ "homelab-checkout.service" ];
+                wantedBy = [ "multi-user.target" ];
+                path = [ pkgs.gnugrep ];
+                serviceConfig.Type = "oneshot";
+                script = ''
+                  cfg=$(grep -o 'homelab\.install=[^ ]*' /proc/cmdline | cut -d= -f2 || true)
+                  if [ -z "$cfg" ]; then
+                    echo "no homelab.install= on the kernel cmdline — nothing to auto-install"
+                    exit 0
+                  fi
+                  echo "auto-installing $cfg (homelab.install= on the kernel cmdline)"
+                  cd /root/homelab
+                  exec ./scripts/deploy install "$cfg" localhost
+                '';
+              };
            })
          ];
        };
@@ -20,8 +20,23 @@
 #   ./deploy install <config> <host>  first install. Wipes the OS disk. Ships the
 #                                     host's sops key. <host>=localhost/127.0.0.1
 #                                     skips nixos-anywhere/ssh and runs disko +
-#                                     nixos-install directly against /mnt (use
-#                                     after `kexec-local`, or on a live ISO).
+#                                     nixos-install directly against /mnt — but
+#                                     ONLY once actually inside a live installer
+#                                     (hostname nixos-installer, from kexec, or
+#                                     homelab-installer, from installer-iso).
+#                                     Run from the REAL running OS instead (e.g.
+#                                     a box where kexec-local doesn't work),
+#                                     it builds installer-iso, stages its
+#                                     kernel/initrd on the ESP + the iso file on
+#                                     a non-OS-disk partition, sets a systemd-boot
+#                                     one-shot entry with homelab.install=<config>
+#                                     on its kernel cmdline, and reboots — a real
+#                                     ACPI reboot, not a kexec jump. The booted
+#                                     installer's homelab-auto-install.service
+#                                     reads that cmdline param and re-runs this
+#                                     exact command itself once its repo checkout
+#                                     (homelab-checkout.service) succeeds, finishing
+#                                     the install unattended. See CLAUDE.md.
 #   ./deploy switch  <config> <host>  rebuild + activate on a running host.
 #   ./deploy boot    <config> <host>  stage for next boot, don't activate now.
 #   ./deploy test    <config> <host>  activate without adding a boot entry.
@@ -46,8 +61,16 @@
 set -euo pipefail
 shopt -s nullglob

+# Captured before anything shifts/parses $@, so require_root() below can
+# re-exec the ORIGINAL invocation under sudo — inside a function, "$@"/"$1"
+# refer to the function's own args (empty here), not the script's, so this
+# has to be a global array instead of relying on positional-parameter scoping.
+SCRIPT_ARGS=("$@")
+
 # Locate the repo root (flake dir) regardless of where this script lives on disk.
-SCRIPT_DIR="$(cd "$(dirname "$(realpath "$0")")" && pwd)"
+SCRIPT_PATH="$(realpath "$0")"           # absolute — "$0" itself may be relative,
+                                          # and require_root() re-execs after cd "$REPO"
+SCRIPT_DIR="$(dirname "$SCRIPT_PATH")"
 REPO="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || dirname "$SCRIPT_DIR")"
 cd "$REPO"
 export PATH="/nix/var/nix/profiles/default/bin:$PATH"
@@ -56,6 +79,15 @@ die() { echo "error: $*" >&2; exit 1; }

 need() { command -v "$1" >/dev/null 2>&1 || die "missing required tool: $1"; }

+# Self-elevate instead of dying: re-exec this exact invocation under sudo.
+# -E preserves the environment (HOMELAB_* overrides, Proton Pass vault vars)
+# across the re-exec. A no-op once already root.
+require_root() {
+  [ "$(id -u)" = 0 ] && return 0
+  echo ">> $1 needs root — re-executing under sudo" >&2
+  exec sudo -E -- "$SCRIPT_PATH" "${SCRIPT_ARGS[@]}"
+}
+
 # Exactly one path matching a glob, or die. `ls glob | head -1` silently yields
 # an empty string when nothing matches (head exits 0, so set -e never fires) and
 # the failure only surfaces later as a confusing tar/dd error.
@@ -92,6 +124,99 @@ kexec_artifacts() {
  fi
 }

+# True inside one of the throwaway live-installer environments this repo
+# produces (kexec's nixos-installer, or installer-iso's homelab-installer) —
+# i.e. `install <config> localhost` should wipe/install right here. False on
+# any real running OS, where the same command instead means "prepare and
+# reboot into an installer for THIS box" (see local_install_prepare_and_reboot).
+is_live_installer() {
+  case "$(uname -n)" in
+    nixos-installer | homelab-installer) return 0 ;;
+    *) return 1 ;;
+  esac
+}
+
+# `install <config> localhost` run on a REAL running OS (not already inside a
+# live installer): builds installer-iso, stages its kernel/initrd + iso file
+# locally, points a systemd-boot one-shot entry at them with
+# homelab.install=<config> on the kernel cmdline, and reboots — a real ACPI
+# reboot through firmware POST, deliberately NOT a kexec jump (see terra's
+# kexec-local gotcha in CLAUDE.md). The booted installer's
+# homelab-auto-install.service reads that cmdline param and re-runs this exact
+# `install <config> localhost` command itself (now genuinely inside the
+# installer) once homelab-checkout.service has fetched the repo, finishing the
+# job unattended.
+local_install_prepare_and_reboot() {
+  local config="$1"
+  require_root "preparing a local reinstall"
+  [ -d /sys/firmware/efi ] || die "not booted UEFI — the one-shot boot entry needs systemd-boot"
+  need bootctl
+  need nix
+  need lsblk
+  need findmnt
+
+  # No default/auto-picked location — the wrong disk here is destroyed
+  # mid-install (see the OS-disk check below), so this always asks rather
+  # than guessing. HOMELAB_INSTALLER_STAGE_DIR skips the prompt for scripted
+  # use, but is otherwise just as explicit a choice as typing it in.
+  local stagedir="${HOMELAB_INSTALLER_STAGE_DIR:-}"
+  if [ -z "$stagedir" ]; then
+    echo ">> currently mounted filesystems:"
+    lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT
+    read -rp ">> path to stage the installer iso on (must NOT be on the OS disk being wiped): " stagedir
+  fi
+  [ -n "$stagedir" ] || die "no staging path given"
+  [ -d "$stagedir" ] \
+    || die "staging dir $stagedir doesn't exist — needs to be an existing partition that is NOT the OS disk being wiped"
+
+  # Refuse if the staging partition turns out to live on the same disk
+  # disko is about to wipe — the iso file (and the running installer
+  # loopback-mounted from it) would be destroyed mid-install.
+  local osdisk osdisk_real stage_src stage_pkname stage_disk_real
+  osdisk="$(nix eval --raw ".#nixosConfigurations.$config.config.disko.devices.disk" \
+    --apply 'd: (builtins.head (builtins.attrValues d)).device' 2>/dev/null)" \
+    || die "couldn't read the OS disk device from hosts/$config/disk-config.nix"
+  osdisk_real="$(readlink -f "$osdisk")"
+  stage_src="$(findmnt -no SOURCE --target "$stagedir")" \
+    || die "$stagedir doesn't resolve to a mounted filesystem"
+  stage_pkname="$(lsblk -no PKNAME "$stage_src" 2>/dev/null || true)"
+  if [ -n "$stage_pkname" ]; then
+    stage_disk_real="$(readlink -f "/dev/$stage_pkname")"
+    [ "$stage_disk_real" = "$osdisk_real" ] \
+      && die "$stagedir is on the OS disk ($osdisk) that install would wipe — re-run and pick a different disk"
+  fi
+
+  echo ">> building installer-iso (kernel + initrd + iso image)"
+  local kernel initrd isodir iso mnt_point iso_relpath
+  kernel="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.kernel)/bzImage"
+  initrd="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.initialRamdisk)/initrd"
+  isodir="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.isoImage)"
+  iso="$(one_match 'installer iso' "$isodir"/iso/*.iso)"
+
+  echo ">> staging kernel/initrd on the ESP, iso image on $stagedir"
+  install -Dm644 "$kernel" /boot/homelab-installer/bzImage
+  install -Dm644 "$initrd" /boot/homelab-installer/initrd
+  install -Dm644 "$iso"    "$stagedir/homelab-installer.iso"
+
+  # findiso= is a path relative to whatever partition the initrd finds it on
+  # (it mounts every blkid-visible partition looking for it) — not necessarily
+  # relative to `/`, if $stagedir is a subdirectory of a bigger filesystem
+  # rather than a mountpoint itself.
+  mnt_point="$(findmnt -no TARGET --target "$stagedir")"
+  iso_relpath="${stagedir#"$mnt_point"}/homelab-installer.iso"
+
+  cat >/boot/loader/entries/homelab-installer.conf <<EOF
+title Homelab Installer ($config, findiso)
+linux /homelab-installer/bzImage
+initrd /homelab-installer/initrd
+options nohibernate root=fstab loglevel=4 lsm=landlock,yama,bpf findiso=$iso_relpath homelab.install=$config
+EOF
+
+  echo ">> one-shot boot into the installer, then rebooting — it will finish this install itself"
+  bootctl set-oneshot homelab-installer.conf
+  systemctl reboot
+}
+
 # Flakes only see git-tracked files: an untracked hosts/<config>/ is silently
 # invisible to `nix build`/`nixos-install`, which then fails obscurely or builds
 # a stale config. Check before doing anything destructive.
@@ -240,7 +365,7 @@ case "$cmd" in
    # This is a one-way trip on the machine you are typing at, so every check
    # that can fail is done BEFORE the point of no return, and nothing that the
    # jump depends on is cleaned up behind it (see the trap discussion below).
-    [ "$(id -u)" = 0 ] || die "kexec-local must run as root (sudo ./deploy kexec-local)"
+    require_root "kexec-local"

    assume_yes=""
    [ "${2:-}" = "--yes" ] && assume_yes=1
@@ -352,10 +477,18 @@ case "$cmd" in
    require_tracked "$config"

    if [ "$host" = "localhost" ] || [ "$host" = "127.0.0.1" ]; then
+      if ! is_live_installer; then
+        # Not already inside a live installer: build one, stage it, one-shot
+        # boot into it, and let it finish this exact command itself. See
+        # local_install_prepare_and_reboot above and CLAUDE.md.
+        local_install_prepare_and_reboot "$config"
+        exit 0
+      fi
+
      # Local install: no ssh, no nixos-anywhere. Run after `kexec-local` (or
      # from a live ISO) so /mnt is free to wipe — this IS the box, no second
      # machine in the loop, so skip straight to disko + nixos-install.
-      [ "$(id -u)" = 0 ] || die "local install must run as root"
+      require_root "local install"
      [ -f "./hosts/$config/disk-config.nix" ] || die "no ./hosts/$config/disk-config.nix"

      echo ">> disko .#$config onto this box's OS disk (WILL be wiped)"