deploy: harden kexec-local, key vault items by config, add VM test

kexec-local could never actually jump. nixos-images' kexec-run.sh ends with
`nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &` and returns immediately, so
the EXIT trap's `rm -rf "$stage"` deleted the kexec binary out from under the
sleeping shell. The box stayed on the old kernel and it looked like a slow boot.
Clear the trap before jumping, verify /sys/kernel/kexec_loaded, then sleep past
the timer.

Preflight everything before the point of no return, since this jumps the machine
you are typing at: CONFIG_KEXEC, kernel lockdown, exec-capable staging dir, free
space, RAM vs image size, and that the tarball holds all five expected files.
Stage on /var/tmp rather than /tmp because kexec-run.sh appends to initrd in
place and execs from that directory. sync before jumping (kexec -e skips
unmount). Confirmation prompt naming the host, since run in the wrong terminal
this kexecs the laptop; --yes skips it.

Drop the ssh-keygen -R added to the remote kexec path: kexec-run.sh copies
/etc/ssh/ssh_host_* into the appended initrd and restore-remote-access.nix
installs them back, so the host key survives the jump.

Proton Pass items are now keyed by <config> instead of <host>, since the address
is incidental and the config name is stable. kexec therefore takes <config>
<host>. Resolve titles among --filter-state active items first: a trashed item
with the same title shadowed the active one and returned an empty password,
which is indistinguishable from "no entry" and silently fell back to prompting
(hit on darman@neptun).

Other fixes: replace `ls glob | head -1` (returns empty with exit 0 on no match)
with a helper that dies; guard against untracked hosts/<config> since flakes
ignore untracked files; feed the sudo password more than once under setsid;
handle empty arrays under set -u; tolerate empty FSTYPE in the SD-card root
partition lookup; preflight zstdcat/dd/lsblk before the destructive dd; list
image and flash in the usage strings.

Add checks.x86_64-linux.kexec-local, a VM test driving the real script. It is
the only way to exercise kexec-local, which cannot be rehearsed on hardware. It
asserts the box left the old kernel, returned as nixos-installer, lost its old
/run, and kept its ssh host key. HOMELAB_KEXEC_TARBALL lets it reuse a prebuilt
installer instead of building ~500MB inside the guest.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CLAUDE.md

@@ -38,11 +38,22 @@ Deploy (from a non-NixOS laptop too — runs nixos-rebuild/nixos-anywhere via `n
 ```
 ./scripts/deploy switch  <config> <host>     # daily rebuild + activate
 ./scripts/deploy install <config> <host>     # first install (nixos-anywhere, wipes OS disk)
-./scripts/deploy kexec   <host>              # RO-root box (ZimaOS): kexec into a RAM installer first
+./scripts/deploy kexec   <config> <host>     # RO-root box (ZimaOS): kexec into a RAM installer first
+sudo ./scripts/deploy kexec-local [--yes]    # kexec THIS box (no ssh); confirm prompt unless --yes
 ./scripts/deploy image   mercury             # build the aarch64 SD image
 ./scripts/deploy flash   mercury /dev/sdX    # build + write SD + drop the sops age key
 ```
 
+Both password prompts are auto-filled from the "HomeLab" Proton Pass vault, keyed by
+**`<config>`, not `<host>`** — items `darman@<config>` (sudo) and `root@<config>` (ssh).
+A *trashed* Proton Pass item with the same title shadows the active one and yields an
+empty password, so the script resolves the title among `--filter-state active` items
+first; a plain `pass-cli item view --item-title` silently returns the trashed copy and
+you get an interactive prompt with no explanation.
+
+`HOMELAB_KEXEC_TARBALL` (+ `_CPIO` / `_GZIP`) makes `kexec`/`kexec-local` reuse a
+prebuilt installer instead of rebuilding ~500MB. The VM test below uses this.
+
 Secrets (needs the admin age key at `~/.config/sops/age/keys.txt`):
 ```
 ./scripts/edit_secrets secrets/<host>.yaml
@@ -55,8 +66,16 @@ nix build .#nixosConfigurations.mercury-vm.config.system.build.vm -o result
 ./result/bin/run-mercury-vm-vm            # ssh -p 2223 darman@localhost (pw: test)
 # jupiter services as a VirtualBox OVA
 nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
+# end-to-end VM test of `deploy kexec-local` (~45s once the tarball is built)
+nix build .#checks.x86_64-linux.kexec-local -L
 ```
 
+`checks.kexec-local` is the only way to exercise `kexec-local` at all: it jumps the
+machine you are typing at, so it cannot be rehearsed on real hardware and a failure
+looks exactly like a slow boot. It asserts the box actually left the old kernel
+(SSH drops then returns), came back as `nixos-installer`, lost its old `/run`, and
+kept its ssh host key. Run it after ANY change to the kexec paths.
+
 ## Secrets (sops-nix)
 
 - Each `secrets/<host>.yaml` is encrypted to the **admin** key (edit) + that **host's**
@@ -93,3 +112,15 @@ nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
 - `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the
   `./scripts/deploy kexec` step that streams a RAM installer (with static cpio/gzip since
   ZimaOS lacks them).
+- **`kexec/run` jumps ~6s AFTER it returns**: nixos-images' `kexec-run.sh` ends with
+  `nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &`. So the staging dir must OUTLIVE the
+  script — an `rm -rf` in an EXIT trap deletes the binary that performs the jump and the
+  box silently stays on the old kernel. `kexec-local` clears its trap before jumping and
+  then sleeps 60s on purpose. Covered by `checks.kexec-local`.
+- **The kexec installer KEEPS the box's ssh host key**: `kexec-run.sh` copies
+  `/etc/ssh/ssh_host_*` into the appended initrd and `restore-remote-access.nix` installs
+  them back. So do NOT `ssh-keygen -R` after a kexec — the key does not change, and
+  clearing it just throws away the known_hosts record.
+- **`kexec-local` stages on `/var/tmp`, not `/tmp`**: `kexec-run.sh` appends a fresh cpio
+  to `kexec/initrd` in place and execs binaries from that dir, so a size-capped or
+  `noexec` tmpfs gives a half-written initrd or a bare "Permission denied".

README.md

@@ -89,7 +89,7 @@ Manual alternative (USB ISO): boot installer, `disko` the disk, then
 All arguments mandatory — no default host, no default config.
 
 ```
-./deploy kexec   <host>            # headless kexec into a RAM installer (RO-root box)
+./deploy kexec   <config> <host>   # headless kexec into a RAM installer (RO-root box)
 ./deploy install <config> <host>   # first install; wipes OS disk, ships host key
 ./deploy switch  <config> <host>   # rebuild + activate on a running host
 ./deploy boot|test <config> <host> # stage for next boot / activate without boot entry

flake.nix

@@ -20,6 +20,10 @@
       url = "github:nix-community/nixos-images";
       inputs.nixpkgs.follows = "nixpkgs";
     };
+    home-manager = {
+      url = "github:nix-community/home-manager/release-26.05";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
     mediamanager-nix = {
       url = "github:strangeglyph/mediamanager-nix";
       inputs.nixpkgs.follows = "nixpkgs";
@@ -30,7 +34,7 @@
     authentik-nix.url = "github:nix-community/authentik-nix";
   };
 
-  outputs = { self, nixpkgs, disko, sops-nix, nixos-images, mediamanager-nix, authentik-nix, ... }@inputs:
+  outputs = { self, nixpkgs, disko, sops-nix, nixos-images, home-manager, mediamanager-nix, authentik-nix, ... }@inputs:
     let
       system = "x86_64-linux";
     in
@@ -59,6 +63,19 @@
           ];
         };
 
+        # terra — Ryzen 9 5900X desktop (MSI MS-7A32). Replaces CachyOS on the
+        # OS SSD; Hyprland desktop + tailnet node. See hosts/terra/*.
+        terra = nixpkgs.lib.nixosSystem {
+          inherit system;
+          specialArgs = { inherit inputs; };
+          modules = [
+            disko.nixosModules.disko
+            sops-nix.nixosModules.sops
+            home-manager.nixosModules.home-manager
+            ./hosts/terra/configuration.nix
+          ];
+        };
+
         # mercury — Raspberry Pi 3B+ (aarch64), DNS/DHCP. Boots from an SD image:
         #   nix build .#nixosConfigurations.mercury.config.system.build.sdImage
         # (aarch64 build — needs binfmt/qemu on this x86 host, or a remote/aarch64
@@ -146,5 +163,151 @@
           ];
         };
       };
+
+      # VM test for `./scripts/deploy kexec-local`. Run:
+      #   nix build .#checks.x86_64-linux.kexec-local -L
+      #
+      # Worth having because kexec-local is the one command that cannot be
+      # rehearsed on real hardware: it jumps the machine you are typing at, and
+      # a failure looks exactly like a slow boot. It regression-tests the
+      # subtle one — kexec-run.sh backgrounds `sleep 6 && kexec -e` and returns,
+      # so anything that cleans up the staging dir on exit deletes the binary
+      # that performs the jump and the box silently never leaves the old kernel.
+      #
+      # After the jump the test driver's backdoor is gone with the old kernel,
+      # so the installer is driven over a forwarded ssh port instead (the same
+      # approach nixos-images uses in its own kexec test).
+      checks.${system}.kexec-local =
+        let
+          pkgs = nixpkgs.legacyPackages.${system};
+          tarball = self.nixosConfigurations.kexec.config.system.build.kexecInstallerTarball;
+          sshKey = nixos-images + "/nix/kexec-installer/ssh-keys/id_ed25519";
+        in
+        pkgs.testers.runNixOSTest {
+          name = "deploy-kexec-local";
+
+          nodes.machine = { modulesPath, ... }: {
+            imports = [ (modulesPath + "/profiles/minimal.nix") ];
+            virtualisation.vlans = [ ];
+            # kexec-local refuses to run if RAM < 3x the installer image, and
+            # the staging dir needs ~3x the tarball on /var/tmp.
+            virtualisation.memorySize = 4 * 1024;
+            virtualisation.diskSize = 12 * 1024;
+            virtualisation.forwardPorts = [{ host.port = 2222; guest.port = 22; }];
+
+            services.openssh.enable = true;
+            users.users.root.openssh.authorizedKeys.keyFiles = [ "${sshKey}.pub" ];
+
+            # Everything the script shells out to, minus nix — the test uses the
+            # HOMELAB_KEXEC_* hook so no build happens inside the VM.
+            environment.systemPackages = with pkgs; [
+              bash gnutar coreutils findutils util-linux cpio gzip
+            ];
+            system.extraDependencies = [ tarball pkgs.cpio pkgs.gzip ];
+
+            environment.etc."deploy".source = ./scripts/deploy;
+          };
+
+          testScript = /* python */ ''
+            import os, shutil, subprocess, tempfile, time
+
+            start_all()
+            machine.wait_for_unit("sshd.service")
+
+            # ssh refuses a private key that is group/world readable, and nix
+            # store paths are 0444 — copy it out and tighten the mode.
+            keydir = tempfile.mkdtemp()
+            key = os.path.join(keydir, "id_ed25519")
+            shutil.copyfile("${sshKey}", key)
+            os.chmod(key, 0o600)
+
+            def ssh(cmd, check=True, stdout=None):
+                return subprocess.run(
+                    [ "${pkgs.openssh}/bin/ssh",
+                      "-o", "StrictHostKeyChecking=no",
+                      "-o", "UserKnownHostsFile=/dev/null",
+                      "-o", "ConnectTimeout=1",
+                      "-i", key,
+                      "-p", "2222", "root@127.0.0.1", "--" ] + cmd,
+                    text=True, check=check, stdout=stdout)
+
+            machine.succeed("install -Dm755 /etc/deploy /root/deploy")
+
+            # systemd-run starts units with a bare PATH that lacks
+            # /run/current-system/sw/bin, so `#!/usr/bin/env bash` cannot even
+            # resolve bash, let alone tar/findmnt/nohup. Set it explicitly.
+            env = (
+                " --setenv=PATH=/run/wrappers/bin:/run/current-system/sw/bin"
+                " --setenv=HOMELAB_KEXEC_TARBALL=${tarball}/nixos-kexec-installer-${system}.tar.gz"
+                " --setenv=HOMELAB_KEXEC_CPIO=${pkgs.cpio}/bin/cpio"
+                " --setenv=HOMELAB_KEXEC_GZIP=${pkgs.gzip}/bin/gzip"
+            )
+            # Same values for the foreground (non-systemd-run) invocation below.
+            envsh = (
+                "HOMELAB_KEXEC_TARBALL=${tarball}/nixos-kexec-installer-${system}.tar.gz"
+                " HOMELAB_KEXEC_CPIO=${pkgs.cpio}/bin/cpio"
+                " HOMELAB_KEXEC_GZIP=${pkgs.gzip}/bin/gzip"
+            )
+
+            # Marker on a tmpfs: it must NOT survive the jump, proving we really
+            # booted a new kernel rather than just restarting a service.
+            machine.succeed("touch /run/pre-kexec-marker")
+            host_key_before = machine.succeed("cat /etc/ssh/ssh_host_ed25519_key.pub").strip()
+
+            while ssh(["true"], check=False).returncode != 0:
+                time.sleep(1)
+
+            # Refuses without --yes when stdin is not a tty (read gets EOF).
+            # Must reach the confirmation prompt, so it needs the same env —
+            # otherwise it just dies early on the nix build and proves nothing.
+            out = machine.fail(f"{envsh} /root/deploy kexec-local </dev/null 2>&1")
+            assert "using prebuilt kexec installer" in out, \
+                f"never reached the prompt, so the refusal proves nothing:\n{out}"
+
+            # systemd-run so the call returns immediately: the script stays
+            # alive ~60s on purpose, outliving kexec-run.sh's `sleep 6`.
+            machine.succeed(f"systemd-run --collect --unit=kexec-local{env} /root/deploy kexec-local --yes")
+
+            print("waiting for the jump...")
+            deadline = time.time() + 300
+            while ssh(["true"], check=False).returncode == 0:
+                # Surface a dead unit immediately instead of stalling until the
+                # deadline and blaming "never left the old kernel".
+                st = ssh(["systemctl", "is-active", "kexec-local"],
+                         check=False, stdout=subprocess.PIPE).stdout or ""
+                if st.strip() in ("failed", "inactive"):
+                    # NB: not `log` — the driver already binds that name to its
+                    # AbstractLogger and the type check rejects the shadowing.
+                    unit_log = ssh(["journalctl", "-u", "kexec-local", "--no-pager"],
+                                   check=False, stdout=subprocess.PIPE).stdout or ""
+                    raise AssertionError(
+                        f"kexec-local.service ended ({st.strip()}) without jumping:\n{unit_log}")
+                assert time.time() < deadline, "machine never left the old kernel"
+                time.sleep(1)
+
+            print("waiting for the installer...")
+            deadline = time.time() + 300
+            while ssh(["true"], check=False).returncode != 0:
+                assert time.time() < deadline, "installer never came up"
+                time.sleep(1)
+
+            # It really is the RAM installer, not the old system.
+            host = ssh(["hostname"], stdout=subprocess.PIPE).stdout.strip()
+            assert host == "nixos-installer", f"hostname is {host}, not nixos-installer"
+
+            assert ssh(["ls", "/run/pre-kexec-marker"], check=False).returncode != 0, \
+                "old /run survived — this was not a fresh kernel"
+
+            # The host key is carried across (kexec-run.sh copies /etc/ssh into
+            # the appended initrd), which is why `kexec` does no ssh-keygen -R.
+            host_key_after = ssh(
+                ["cat", "/etc/ssh/ssh_host_ed25519_key.pub"], stdout=subprocess.PIPE
+            ).stdout.strip()
+            assert host_key_before == host_key_after, \
+                f"host key changed: {host_key_before} != {host_key_after}"
+
+            machine.crash()
+          '';
+        };
     };
 }

scripts/deploy

@@ -1,12 +1,27 @@
 #!/usr/bin/env bash
 # Deploy a NixOS host from this flake. ALL arguments are mandatory (no defaults).
 #
-#   ./deploy kexec   <host>            headless kexec into a RAM installer, for a
-#                                      read-only-root box (ZimaOS) where
-#                                      nixos-anywhere can't ssh-copy-id. Ships our
-#                                      SSH login key. Then run `install`.
-#   ./deploy install <config> <host>  first install (nixos-anywhere). Wipes the
-#                                      OS disk. Ships the host's sops key.
+#   ./deploy kexec   <config> <host>  headless kexec into a RAM installer, for a
+#                                     read-only-root box (ZimaOS) where
+#                                     nixos-anywhere can't ssh-copy-id. Ships our
+#                                     SSH login key. Then run `install`. <config>
+#                                     is only used to look up the vault item.
+#   ./deploy kexec-local [--yes]      kexec THIS machine into the RAM installer,
+#                                     no ssh/second machine involved. Run as root,
+#                                     locally, on the box you're installing onto.
+#                                     Disks are untouched; console drops for
+#                                     ~1-2 min then comes back as the installer.
+#                                     Prompts for confirmation (--yes skips it),
+#                                     because run on the wrong terminal this
+#                                     kexecs your laptop. TMPDIR (default
+#                                     /var/tmp) must be exec-capable and hold
+#                                     ~3x the tarball.
+#                                     Then run `install <config> localhost`.
+#   ./deploy install <config> <host>  first install. Wipes the OS disk. Ships the
+#                                     host's sops key. <host>=localhost/127.0.0.1
+#                                     skips nixos-anywhere/ssh and runs disko +
+#                                     nixos-install directly against /mnt (use
+#                                     after `kexec-local`, or on a live ISO).
 #   ./deploy switch  <config> <host>  rebuild + activate on a running host.
 #   ./deploy boot    <config> <host>  stage for next boot, don't activate now.
 #   ./deploy test    <config> <host>  activate without adding a boot entry.
@@ -22,11 +37,14 @@
 #
 # Password prompts are auto-filled from the "HomeLab" Proton Pass vault when
 # `pass-cli` is installed and logged in; otherwise every command prompts exactly
-# as before. Items:
+# as before. Both items are keyed by <config>, never by <host>: the address is
+# incidental (DHCP, a new box, localhost) while the config name is the stable
+# identity of the machine being built.
 #   darman@<config>   darman's sudo password  (switch/boot/test)
-#   root@<host>       root's ssh password     (kexec/install)
+#   root@<config>     root's ssh password     (kexec/install)
 # Override with HOMELAB_PASS_ITEM / HOMELAB_PASS_ROOT_ITEM / HOMELAB_PASS_VAULT.
 set -euo pipefail
+shopt -s nullglob
 
 # Locate the repo root (flake dir) regardless of where this script lives on disk.
 SCRIPT_DIR="$(cd "$(dirname "$(realpath "$0")")" && pwd)"
@@ -36,16 +54,93 @@ export PATH="/nix/var/nix/profiles/default/bin:$PATH"
 
 die() { echo "error: $*" >&2; exit 1; }
 
+need() { command -v "$1" >/dev/null 2>&1 || die "missing required tool: $1"; }
+
+# Exactly one path matching a glob, or die. `ls glob | head -1` silently yields
+# an empty string when nothing matches (head exits 0, so set -e never fires) and
+# the failure only surfaces later as a confusing tar/dd error.
+one_match() {
+  local what="$1"; shift
+  local f=("$@")                       # caller expands the glob (nullglob is on)
+  [ "${#f[@]}" -gt 0 ] || die "no $what found — did the build actually produce one?"
+  printf '%s\n' "${f[0]}"
+}
+
+# Sets tb / cpio / bbox — the kexec tarball plus the static cpio+gzip that
+# kexec-run.sh needs on PATH to rebuild its initrd.
+#
+# HOMELAB_KEXEC_TARBALL (with _CPIO / _GZIP) short-circuits the build and uses a
+# prebuilt installer instead. That lets the VM test in flake.nix drive this
+# script offline, and lets you re-kexec a box without rebuilding ~500MB.
+kexec_artifacts() {
+  if [ -n "${HOMELAB_KEXEC_TARBALL:-}" ]; then
+    tb="$HOMELAB_KEXEC_TARBALL"
+    [ -f "$tb" ] || die "HOMELAB_KEXEC_TARBALL=$tb is not a file"
+    cpio="${HOMELAB_KEXEC_CPIO:-$(command -v cpio || true)}"
+    bbox="${HOMELAB_KEXEC_GZIP:-$(command -v gzip || true)}"
+    { [ -n "$cpio" ] && [ -n "$bbox" ]; } \
+      || die "set HOMELAB_KEXEC_CPIO / HOMELAB_KEXEC_GZIP, or put cpio+gzip on PATH"
+    echo ">> using prebuilt kexec installer: $tb"
+  else
+    need nix
+    echo ">> building kexec installer + static tools"
+    nix build .#nixosConfigurations.kexec.config.system.build.kexecInstallerTarball \
+      -o result-kexec
+    tb="$(one_match 'kexec tarball' result-kexec/*.tar.gz)"
+    cpio="$(nix build --no-link --print-out-paths nixpkgs#pkgsStatic.cpio)/bin/cpio"
+    bbox="$(nix build --no-link --print-out-paths nixpkgs#pkgsStatic.busybox)/bin/busybox"
+  fi
+}
+
+# Flakes only see git-tracked files: an untracked hosts/<config>/ is silently
+# invisible to `nix build`/`nixos-install`, which then fails obscurely or builds
+# a stale config. Check before doing anything destructive.
+require_tracked() {
+  local config="$1" cfgfile="hosts/$1/configuration.nix"
+  [ -e "$cfgfile" ] || die "no $cfgfile in the repo"
+  git -C "$REPO" ls-files --error-unmatch "$cfgfile" >/dev/null 2>&1 \
+    || die "$cfgfile is untracked — 'git add hosts/$config' first (flakes ignore untracked files)"
+}
+
 # The password field of a Proton Pass item ("--field password" prints the bare
 # value, one line), or empty if pass-cli is missing / logged out / has no such
 # item — every caller then falls back to the normal interactive prompt.
+#
+# Resolve the title to an item id among ACTIVE items first, because `item view
+# --item-title` has no state filter: Proton Pass keeps deleted items in the
+# trash, and if a trashed item shares the title, view can match THAT one and
+# return an empty password with exit 0. Empty is indistinguishable from "no such
+# item", so the only symptom is a silent fall back to the interactive prompt
+# even though the vault clearly holds the entry. (Hit for real on darman@neptun,
+# which had an Active and a Trashed copy.)
 proton_pass_password() {
-  local title="$1"
+  local title="$1" vault="${HOMELAB_PASS_VAULT:-HomeLab}" id pw
   command -v pass-cli >/dev/null 2>&1 || return 0
-  pass-cli item view \
-    --vault-name "${HOMELAB_PASS_VAULT:-HomeLab}" \
-    --item-title "$title" \
-    --field password --output human 2>/dev/null | head -1
+
+  # Lines look like:  - [ITEM_ID]: the title (state=Active)
+  id="$(pass-cli item list --vault-name "$vault" --filter-state active \
+          --output human 2>/dev/null \
+        | awk -v t="$title" '
+            { i = index($0, "]: "); if (i == 0) next
+              id   = substr($0, 4, i - 4)
+              rest = substr($0, i + 3)
+              sub(/ \(state=[^)]*\)$/, "", rest)
+              if (rest == t) { print id; exit } }' || true)"
+
+  if [ -n "$id" ]; then
+    pw="$(pass-cli item view --vault-name "$vault" --item-id "$id" \
+            --field password --output human 2>/dev/null | head -1 || true)"
+    # Resolved an active item but its password field is blank: worth saying so,
+    # otherwise this looks identical to having no vault entry at all.
+    [ -n "$pw" ] || echo ">> vault item '$title' resolved but its password field is empty" >&2
+  else
+    # No active match — fall back to the title lookup so an older pass-cli
+    # without --filter-state still works exactly as it used to.
+    pw="$(pass-cli item view --vault-name "$vault" --item-title "$title" \
+            --field password --output human 2>/dev/null | head -1 || true)"
+  fi
+
+  printf '%s' "$pw"
 }
 
 # Path to an sshpass binary (system one, else built from nixpkgs). Empty if
@@ -56,20 +151,18 @@ sshpass_bin() {
     | sed 's|$|/bin/sshpass|'
 }
 
-cmd="${1:-}"; [ -n "$cmd" ] || die "usage: ./deploy <kexec|install|switch|boot|test> ..."
+cmd="${1:-}"; [ -n "$cmd" ] || die "usage: ./deploy <kexec|kexec-local|install|switch|boot|test|image|flash> ..."
 
 case "$cmd" in
   kexec)
-    host="${2:-}"; [ -n "$host" ] || die "usage: ./deploy kexec <host>"
+    config="${2:-}"; host="${3:-}"
+    { [ -n "$config" ] && [ -n "$host" ]; } || die "usage: ./deploy kexec <config> <host>"
+
+    need ssh; need scp
 
-    echo ">> building kexec installer + static tools"
-    nix build .#nixosConfigurations.kexec.config.system.build.kexecInstallerTarball \
-      -o result-kexec
-    tb="$(ls result-kexec/*.tar.gz | head -1)"
     # kexec/run rebuilds an initrd with `cpio` + `gzip` from PATH — ZimaOS lacks
     # both. Ship static ones: GNU cpio (reliable -o -H newc), busybox as gzip.
-    cpio="$(nix build --no-link --print-out-paths nixpkgs#pkgsStatic.cpio)/bin/cpio"
-    bbox="$(nix build --no-link --print-out-paths nixpkgs#pkgsStatic.busybox)/bin/busybox"
+    kexec_artifacts
 
     # One password prompt: multiplex scp + ssh over a shared control connection.
     cm="/tmp/homelab-cm-%r@%h:%p"
@@ -78,13 +171,21 @@ case "$cmd" in
 
     # Root's password from Proton Pass, fed to ssh/scp via sshpass -e. Only the
     # first (master) connection authenticates; the rest ride the control socket.
-    # SSHPASS is exported into the sshpass child only, never onto a command line.
+    #
+    # SSHPASS is exported here rather than passed as `env SSHPASS=... sshpass`.
+    # Both end up equally safe at rest: `env` execs its target immediately, so
+    # the assignment is only in argv for the sub-millisecond before exec, after
+    # which /proc/PID/cmdline reads plain `sshpass -e`. Exporting just closes
+    # that race window and drops a process. Either way the secret lives in the
+    # child's environ, which is readable by the owner and root only.
     sp=()
-    root_pw="$(proton_pass_password "${HOMELAB_PASS_ROOT_ITEM:-root@$host}" || true)"
+    root_item="${HOMELAB_PASS_ROOT_ITEM:-root@$config}"
+    root_pw="$(proton_pass_password "$root_item" || true)"
     if [ -n "$root_pw" ]; then
       sshpass="$(sshpass_bin || true)"
       if [ -n "$sshpass" ]; then
-        sp=(env "SSHPASS=$root_pw" "$sshpass" -e)
+        export SSHPASS="$root_pw"
+        sp=("$sshpass" -e)
         # sshpass drives the password prompt; don't let a key/agent short-circuit
         # into an interactive one for a host that only accepts passwords.
         o+=(-o PreferredAuthentications=password -o PubkeyAuthentication=no)
@@ -92,26 +193,150 @@ case "$cmd" in
         echo ">> sshpass unavailable — falling back to the interactive prompt" >&2
       fi
     fi
+    unset root_pw
 
-    if [ ${#sp[@]} -gt 0 ]; then
-      echo ">> connecting to root@$host  (password from Proton Pass)"
+    if [ "${#sp[@]}" -gt 0 ]; then
+      echo ">> connecting to root@$host  (password from Proton Pass: $root_item)"
     else
       echo ">> connecting to root@$host  (enter the root password once)"
     fi
-    "${sp[@]}" ssh "${o[@]}" "root@$host" 'mkdir -p /tmp/bin'
+    "${sp[@]+"${sp[@]}"}" ssh "${o[@]}" "root@$host" 'mkdir -p /tmp/bin'
     scp "${o[@]}" "$cpio" "root@$host:/tmp/bin/cpio"
     scp "${o[@]}" "$bbox" "root@$host:/tmp/bin/gzip"   # busybox as gzip (argv0)
-    unset root_pw
 
     echo ">> streaming installer + kexec-ing. SSH drops as the box jumps into the"
     echo "   RAM installer. Disks are untouched."
+    rc=0
     ssh "${o[@]}" "root@$host" \
       'chmod +x /tmp/bin/*; mkdir -p /tmp/k && tar -C /tmp/k -xzf - && PATH=/tmp/bin:$PATH /tmp/k/kexec/run' \
-      < "$tb" || true
+      < "$tb" || rc=$?
+    # A dropped connection IS the success case here, so this can't be fatal —
+    # but a full /tmp, a tar error or a missing static tool looks identical from
+    # the outside, so at least surface the code instead of swallowing it.
+    [ "$rc" -eq 0 ] || echo ">> ssh exited $rc — expected if the box jumped; suspect this if install can't connect" >&2
 
     ssh "${o[@]}" -O exit "root@$host" 2>/dev/null || true  # close control socket
+    unset SSHPASS
+
+    # NB: no ssh-keygen -R here on purpose. kexec-run.sh copies /etc/ssh/ssh_host_*
+    # into the appended initrd and restore-remote-access.nix installs them back
+    # into the installer's /etc/ssh, so the host key SURVIVES the jump. Clearing
+    # known_hosts would just throw away the TOFU record for no reason.
+
     echo ">> box is kexec-ing. Wait ~1-2 min for the installer + network, then:"
-    echo "   ./deploy install <config> $host"
+    echo "   ./deploy install $config $host"
+    ;;
+
+  kexec-local)
+    # No ssh, no second machine: build the same RAM installer as `kexec`, but
+    # run it directly on this box (you're sitting at it). The current shell
+    # drops when the kernel switches, same as any reboot — that's expected,
+    # not a failure. Disks are untouched; only the running kernel changes.
+    #
+    # This is a one-way trip on the machine you are typing at, so every check
+    # that can fail is done BEFORE the point of no return, and nothing that the
+    # jump depends on is cleaned up behind it (see the trap discussion below).
+    [ "$(id -u)" = 0 ] || die "kexec-local must run as root (sudo ./deploy kexec-local)"
+
+    assume_yes=""
+    [ "${2:-}" = "--yes" ] && assume_yes=1
+
+    need tar; need install; need mktemp; need find; need sync; need nohup
+
+    # --- preflight: reasons the jump would fail, checked while it's still safe --
+    # CONFIG_KEXEC. Without it kexec --load fails and you've built ~1GB for
+    # nothing; with lockdown it fails at load time too (Secure Boot blocks the
+    # kexec_load syscall unless the image is signed).
+    [ -e /sys/kernel/kexec_loaded ] \
+      || die "this kernel has no kexec support (CONFIG_KEXEC) — boot install media instead"
+    if [ -r /sys/kernel/security/lockdown ] \
+       && ! grep -q '\[none\]' /sys/kernel/security/lockdown 2>/dev/null; then
+      die "kernel lockdown is active — kexec_load is blocked. Disable Secure Boot, or boot install media."
+    fi
+
+    kexec_artifacts
+
+    # Staging lives on /var/tmp, not /tmp: kexec-run.sh APPENDS a fresh cpio to
+    # kexec/initrd in place and runs binaries out of this directory, so it needs
+    # real space and exec permission. A tmpfs /tmp is both size-capped (ENOSPC
+    # mid-append = a half-written initrd) and frequently noexec.
+    stage="$(TMPDIR="${TMPDIR:-/var/tmp}" mktemp -d)"
+    trap 'rm -rf "$stage"' EXIT
+
+    # noexec would only surface as a cryptic "Permission denied" on kexec/run.
+    printf '#!/bin/sh\nexit 0\n' > "$stage/.execmove"
+    chmod +x "$stage/.execmove"
+    "$stage/.execmove" 2>/dev/null \
+      || die "$stage is mounted noexec — set TMPDIR to an exec-capable filesystem"
+    rm -f "$stage/.execmove"
+
+    # Need room for the tarball, its extracted contents, and the appended cpio.
+    tb_sz="$(stat -Lc %s "$tb")"
+    avail="$(df -B1 --output=avail "$stage" | tail -1 | tr -d ' ')"
+    [ "$avail" -ge $(( tb_sz * 3 )) ] \
+      || die "only $(( avail / 1024 / 1024 ))MB free on $stage, need ~$(( tb_sz * 3 / 1024 / 1024 ))MB — set TMPDIR elsewhere"
+
+    install -Dm755 "$cpio" "$stage/bin/cpio"
+    install -Dm755 "$bbox" "$stage/bin/gzip"   # busybox as gzip (argv0)
+    mkdir -p "$stage/k" && tar -C "$stage/k" -xzf "$tb"
+
+    # Fail loudly here rather than with a bare "not found" from kexec/run.
+    for f in run kexec bzImage initrd ip; do
+      [ -f "$stage/k/kexec/$f" ] || die "kexec tarball is missing kexec/$f — bad build?"
+    done
+
+    # The installer root is a RAM disk: the initrd has to fit in memory with
+    # room to unpack. Refuse rather than OOM halfway into the new kernel, where
+    # there is no way back except the power button.
+    img_sz="$(( $(stat -Lc %s "$stage/k/kexec/initrd") + $(stat -Lc %s "$stage/k/kexec/bzImage") ))"
+    mem_kb="$(awk '/^MemTotal:/{print $2}' /proc/meminfo)"
+    [ "$(( mem_kb * 1024 ))" -ge "$(( img_sz * 3 ))" ] \
+      || die "only $(( mem_kb / 1024 ))MB RAM for a $(( img_sz / 1024 / 1024 ))MB RAM-disk installer — too tight, boot install media instead"
+
+    # --- confirmation: this script normally runs on the LAPTOP ------------------
+    # A stray `kexec-local` here would jump the machine you develop on, not the
+    # target. Name it explicitly so a wrong-terminal mistake is visible.
+    echo ">> about to kexec THIS machine:"
+    echo "   hostname: $(uname -n)"
+    echo "   kernel:   $(uname -r)"
+    echo "   root:     $(findmnt -no SOURCE / 2>/dev/null || echo '?')"
+    echo "   Disks are untouched; only the running kernel changes. Console drops"
+    echo "   for ~1-2 min, then comes back as the installer."
+    if [ -z "$assume_yes" ]; then
+      read -rp ">> type 'yes' to kexec $(uname -n) into the RAM installer: " ok
+      [ "$ok" = yes ] || die "aborted"
+    fi
+
+    # kexec -e jumps straight to the new kernel: no unmount, no journal flush,
+    # no systemd shutdown. Anything still in page cache is lost and the OS disk
+    # is left dirty. Cheap insurance, since the box may yet be rebooted back.
+    sync
+
+    echo ">> loading the new kernel"
+    if ! PATH="$stage/bin:$PATH" "$stage/k/kexec/run"; then
+      rm -rf "$stage"
+      die "kexec/run failed — machine untouched, still on the old kernel"
+    fi
+
+    # kexec --load succeeded iff the kernel now reports a loaded image. If this
+    # is 0 the background `kexec -e` below will do nothing and we'd hang forever
+    # waiting for a jump that cannot happen.
+    [ "$(cat /sys/kernel/kexec_loaded 2>/dev/null || echo 0)" = 1 ] \
+      || { rm -rf "$stage"; die "kexec reported success but no image is loaded — aborting"; }
+
+    # THE trap MUST GO NOW. kexec-run.sh backgrounds `nohup sh -c "sleep 6 &&
+    # $SCRIPT_DIR/kexec -e"` and returns immediately, so the binary that
+    # performs the jump still has to exist ~6s after this script would normally
+    # exit. Letting the EXIT trap rm -rf "$stage" deletes it out from under that
+    # sleeping shell and the machine silently never jumps.
+    trap - EXIT
+
+    sync
+    echo ">> kernel loaded; jumping in ~6s. (staging left at $stage on purpose —"
+    echo "   the backgrounded kexec still needs it; it's gone after the jump.)"
+    # Outlive the sleep 6 so the jump happens while this script is still alive.
+    sleep 60
+    die "still here after 60s — the jump did not happen. Check dmesg; the old system is intact."
     ;;
 
   install)
@@ -120,35 +345,61 @@ case "$cmd" in
     hostkey="$HOME/.config/homelab/$config/ssh_host_ed25519_key"
     [ -f "$hostkey" ] || die "missing host key: $hostkey"
     [ -d "./hosts/$config" ] || die "no ./hosts/$config directory in the repo"
+    require_tracked "$config"
 
-    # Stage the pre-generated SSH host key so sops can decrypt on boot #1.
-    stage="$(mktemp -d)"
-    trap 'rm -rf "$stage"' EXIT
-    install -Dm600 "$hostkey"      "$stage/etc/ssh/ssh_host_ed25519_key"
-    install -Dm644 "$hostkey.pub"  "$stage/etc/ssh/ssh_host_ed25519_key.pub"
+    if [ "$host" = "localhost" ] || [ "$host" = "127.0.0.1" ]; then
+      # Local install: no ssh, no nixos-anywhere. Run after `kexec-local` (or
+      # from a live ISO) so /mnt is free to wipe — this IS the box, no second
+      # machine in the loop, so skip straight to disko + nixos-install.
+      [ "$(id -u)" = 0 ] || die "local install must run as root"
+      [ -f "./hosts/$config/disk-config.nix" ] || die "no ./hosts/$config/disk-config.nix"
 
-    echo ">> nixos-anywhere .#$config onto root@$host (OS disk WILL be wiped)"
-    anywhere=(--flake ".#$config"
-              --extra-files "$stage"
-              --generate-hardware-config nixos-generate-config "./hosts/$config/hardware-configuration.nix"
-              --target-host "root@$host")
+      echo ">> disko .#$config onto this box's OS disk (WILL be wiped)"
+      nix run github:nix-community/disko -- \
+        --mode disko "./hosts/$config/disk-config.nix"
 
-    # nixos-anywhere's --env-password reads root's ssh password from $SSHPASS
-    # (it ships its own sshpass), so a vault hit skips the ssh-copy-id prompt.
-    root_pw="$(proton_pass_password "${HOMELAB_PASS_ROOT_ITEM:-root@$host}" || true)"
-    if [ -n "$root_pw" ]; then
-      echo ">> root ssh password from Proton Pass"
-      env "SSHPASS=$root_pw" nix run github:nix-community/nixos-anywhere -- \
-        --env-password "${anywhere[@]}"
+      echo ">> installing sops host key so it can decrypt on boot #1"
+      install -Dm600 "$hostkey"     /mnt/etc/ssh/ssh_host_ed25519_key
+      install -Dm644 "$hostkey.pub" /mnt/etc/ssh/ssh_host_ed25519_key.pub
+
+      echo ">> nixos-install .#$config into /mnt"
+      nixos-install --root /mnt --flake ".#$config"
     else
-      nix run github:nix-community/nixos-anywhere -- "${anywhere[@]}"
+      # Stage the pre-generated SSH host key so sops can decrypt on boot #1.
+      stage="$(mktemp -d)"
+      trap 'rm -rf "$stage"' EXIT
+      install -Dm600 "$hostkey"      "$stage/etc/ssh/ssh_host_ed25519_key"
+      install -Dm644 "$hostkey.pub"  "$stage/etc/ssh/ssh_host_ed25519_key.pub"
+
+      echo ">> nixos-anywhere .#$config onto root@$host (OS disk WILL be wiped)"
+      anywhere=(--flake ".#$config"
+                --extra-files "$stage"
+                --generate-hardware-config nixos-generate-config "./hosts/$config/hardware-configuration.nix"
+                --target-host "root@$host")
+
+      # nixos-anywhere's --env-password reads root's ssh password from $SSHPASS
+      # (it ships its own sshpass), so a vault hit skips the ssh-copy-id prompt.
+      # Exported rather than `env SSHPASS=...` for consistency with `kexec`;
+      # see the note there — it's a marginal win, not a leak fix.
+      root_item="${HOMELAB_PASS_ROOT_ITEM:-root@$config}"
+      root_pw="$(proton_pass_password "$root_item" || true)"
+      if [ -n "$root_pw" ]; then
+        echo ">> root ssh password from Proton Pass ($root_item)"
+        export SSHPASS="$root_pw"
+        unset root_pw
+        nix run github:nix-community/nixos-anywhere -- \
+          --env-password "${anywhere[@]}"
+        unset SSHPASS
+      else
+        nix run github:nix-community/nixos-anywhere -- "${anywhere[@]}"
+      fi
     fi
-    unset root_pw
     ;;
 
   switch|boot|test)
     config="${2:-}"; host="${3:-}"
     { [ -n "$config" ] && [ -n "$host" ]; } || die "usage: ./deploy $cmd <config> <host>"
+    require_tracked "$config"
 
     echo ">> nixos-rebuild $cmd .#$config on darman@$host"
     # --ask-sudo-password, not the deprecated --use-remote-sudo: common.nix sets
@@ -167,8 +418,14 @@ case "$cmd" in
       # piped stdin. setsid drops the controlling terminal, so getpass falls back
       # to stdin and takes the vault password (it warns about echo — harmless,
       # nothing is echoed since the password never reaches the terminal).
+      #
+      # Caveat of dropping the tty: EVERY prompt in the subtree now reads this
+      # stdin, not just the sudo one. Feed the line a few times so a retry or a
+      # second sudo ask doesn't hit EOF and hang. Anything else that prompts
+      # (an ssh key passphrase, a host-key confirmation) will still fail — fix
+      # those out of band rather than by feeding more lines here.
       echo ">> sudo password from Proton Pass ($item)"
-      printf '%s\n' "$pw" | setsid -w "${rebuild[@]}"
+      printf '%s\n%s\n%s\n' "$pw" "$pw" "$pw" | setsid -w "${rebuild[@]}"
     else
       "${rebuild[@]}"
     fi
@@ -176,16 +433,22 @@ case "$cmd" in
     ;;
 
   image|flash)
-    config="${2:-}"; [ -n "$config" ] || die "usage: ./deploy $cmd <config> [<dev>]"
-    # Validate the device BEFORE building, so a bad `flash` fails fast.
     if [ "$cmd" = flash ]; then
-      dev="${3:-}"; [ -n "$dev" ] || die "usage: ./deploy flash <config> <dev>  (e.g. /dev/sdX)"
+      config="${2:-}"; dev="${3:-}"
+      { [ -n "$config" ] && [ -n "$dev" ]; } \
+        || die "usage: ./deploy flash <config> <dev>  (e.g. /dev/sdX)"
+      # Validate the device and the tools BEFORE building, so a bad `flash`
+      # fails fast instead of dying halfway through writing the card.
       [ -b "$dev" ] || die "$dev is not a block device"
+      need zstdcat; need dd; need lsblk
+    else
+      config="${2:-}"; [ -n "$config" ] || die "usage: ./deploy image <config>"
     fi
+    require_tracked "$config"
 
     echo ">> building SD image for .#$config (aarch64 needs qemu binfmt)"
     nix build ".#nixosConfigurations.$config.config.system.build.sdImage" -o result-sd
-    img="$(ls result-sd/sd-image/*.img.zst | head -1)"
+    img="$(one_match 'SD image' result-sd/sd-image/*.img.zst)"
     echo ">> image: $img"
     [ "$cmd" = image ] && exit 0
 
@@ -205,8 +468,12 @@ case "$cmd" in
       echo ">> installing sops age key onto the root partition"
       sudo partprobe "$dev" 2>/dev/null || sudo blockdev --rereadpt "$dev" 2>/dev/null || true
       sudo udevadm settle 2>/dev/null || true
-      # largest ext4 partition = the NixOS root.
-      rootpart="$(lsblk -blno PATH,FSTYPE,SIZE "$dev" | awk '$2=="ext4"{print $3, $1}' | sort -rn | head -1 | awk '{print $2}')"
+      # Largest ext4 partition = the NixOS root. -P (KEY="value" pairs) instead
+      # of a columnar listing: an empty FSTYPE collapses under whitespace
+      # splitting and shifts every later field.
+      rootpart="$(lsblk -bPo PATH,FSTYPE,SIZE "$dev" \
+        | sed -n 's/^PATH="\([^"]*\)" FSTYPE="ext4" SIZE="\([0-9]*\)"$/\2 \1/p' \
+        | sort -rn | head -1 | cut -d' ' -f2)"
       [ -n "$rootpart" ] || die "no ext4 root partition found on $dev — place $keyfile at /var/lib/sops-nix/age.txt manually"
       mnt="$(mktemp -d)"
       sudo mount "$rootpart" "$mnt"
@@ -219,6 +486,6 @@ case "$cmd" in
     ;;
 
   *)
-    die "unknown command '$cmd' (kexec|install|switch|boot|test)"
+    die "unknown command '$cmd' (kexec|kexec-local|install|switch|boot|test|image|flash)"
     ;;
 esac