deploy: harden kexec-local, key vault items by config, add VM test

kexec-local could never actually jump. nixos-images' kexec-run.sh ends with
`nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &` and returns immediately, so
the EXIT trap's `rm -rf "$stage"` deleted the kexec binary out from under the
sleeping shell. The box stayed on the old kernel and it looked like a slow boot.
Clear the trap before jumping, verify /sys/kernel/kexec_loaded, then sleep past
the timer.

Preflight everything before the point of no return, since this jumps the machine
you are typing at: CONFIG_KEXEC, kernel lockdown, exec-capable staging dir, free
space, RAM vs image size, and that the tarball holds all five expected files.
Stage on /var/tmp rather than /tmp because kexec-run.sh appends to initrd in
place and execs from that directory. sync before jumping (kexec -e skips
unmount). Confirmation prompt naming the host, since run in the wrong terminal
this kexecs the laptop; --yes skips it.

Drop the ssh-keygen -R added to the remote kexec path: kexec-run.sh copies
/etc/ssh/ssh_host_* into the appended initrd and restore-remote-access.nix
installs them back, so the host key survives the jump.

Proton Pass items are now keyed by <config> instead of <host>, since the address
is incidental and the config name is stable. kexec therefore takes <config>
<host>. Resolve titles among --filter-state active items first: a trashed item
with the same title shadowed the active one and returned an empty password,
which is indistinguishable from "no entry" and silently fell back to prompting
(hit on darman@neptun).

Other fixes: replace `ls glob | head -1` (returns empty with exit 0 on no match)
with a helper that dies; guard against untracked hosts/<config> since flakes
ignore untracked files; feed the sudo password more than once under setsid;
handle empty arrays under set -u; tolerate empty FSTYPE in the SD-card root
partition lookup; preflight zstdcat/dd/lsblk before the destructive dd; list
image and flash in the usage strings.

Add checks.x86_64-linux.kexec-local, a VM test driving the real script. It is
the only way to exercise kexec-local, which cannot be rehearsed on hardware. It
asserts the box left the old kernel, returned as nixos-installer, lost its old
/run, and kept its ssh host key. HOMELAB_KEXEC_TARBALL lets it reuse a prebuilt
installer instead of building ~500MB inside the guest.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CLAUDE.md

@@ -38,11 +38,22 @@ Deploy (from a non-NixOS laptop too — runs nixos-rebuild/nixos-anywhere via `n
 ```
 ./scripts/deploy switch  <config> <host>     # daily rebuild + activate
 ./scripts/deploy install <config> <host>     # first install (nixos-anywhere, wipes OS disk)
-./scripts/deploy kexec   <host>              # RO-root box (ZimaOS): kexec into a RAM installer first
+./scripts/deploy kexec   <config> <host>     # RO-root box (ZimaOS): kexec into a RAM installer first
+sudo ./scripts/deploy kexec-local [--yes]    # kexec THIS box (no ssh); confirm prompt unless --yes
 ./scripts/deploy image   mercury             # build the aarch64 SD image
 ./scripts/deploy flash   mercury /dev/sdX    # build + write SD + drop the sops age key
 ```
 
+Both password prompts are auto-filled from the "HomeLab" Proton Pass vault, keyed by
+**`<config>`, not `<host>`** — items `darman@<config>` (sudo) and `root@<config>` (ssh).
+A *trashed* Proton Pass item with the same title shadows the active one and yields an
+empty password, so the script resolves the title among `--filter-state active` items
+first; a plain `pass-cli item view --item-title` silently returns the trashed copy and
+you get an interactive prompt with no explanation.
+
+`HOMELAB_KEXEC_TARBALL` (+ `_CPIO` / `_GZIP`) makes `kexec`/`kexec-local` reuse a
+prebuilt installer instead of rebuilding ~500MB. The VM test below uses this.
+
 Secrets (needs the admin age key at `~/.config/sops/age/keys.txt`):
 ```
 ./scripts/edit_secrets secrets/<host>.yaml
@@ -55,8 +66,16 @@ nix build .#nixosConfigurations.mercury-vm.config.system.build.vm -o result
 ./result/bin/run-mercury-vm-vm            # ssh -p 2223 darman@localhost (pw: test)
 # jupiter services as a VirtualBox OVA
 nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
+# end-to-end VM test of `deploy kexec-local` (~45s once the tarball is built)
+nix build .#checks.x86_64-linux.kexec-local -L
 ```
 
+`checks.kexec-local` is the only way to exercise `kexec-local` at all: it jumps the
+machine you are typing at, so it cannot be rehearsed on real hardware and a failure
+looks exactly like a slow boot. It asserts the box actually left the old kernel
+(SSH drops then returns), came back as `nixos-installer`, lost its old `/run`, and
+kept its ssh host key. Run it after ANY change to the kexec paths.
+
 ## Secrets (sops-nix)
 
 - Each `secrets/<host>.yaml` is encrypted to the **admin** key (edit) + that **host's**
@@ -93,3 +112,15 @@ nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
 - `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the
   `./scripts/deploy kexec` step that streams a RAM installer (with static cpio/gzip since
   ZimaOS lacks them).
+- **`kexec/run` jumps ~6s AFTER it returns**: nixos-images' `kexec-run.sh` ends with
+  `nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &`. So the staging dir must OUTLIVE the
+  script — an `rm -rf` in an EXIT trap deletes the binary that performs the jump and the
+  box silently stays on the old kernel. `kexec-local` clears its trap before jumping and
+  then sleeps 60s on purpose. Covered by `checks.kexec-local`.
+- **The kexec installer KEEPS the box's ssh host key**: `kexec-run.sh` copies
+  `/etc/ssh/ssh_host_*` into the appended initrd and `restore-remote-access.nix` installs
+  them back. So do NOT `ssh-keygen -R` after a kexec — the key does not change, and
+  clearing it just throws away the known_hosts record.
+- **`kexec-local` stages on `/var/tmp`, not `/tmp`**: `kexec-run.sh` appends a fresh cpio
+  to `kexec/initrd` in place and execs binaries from that dir, so a size-capped or
+  `noexec` tmpfs gives a half-written initrd or a bare "Permission denied".