From 80c2b4fc7b384d45f794c8fed0a107a8550147c2 Mon Sep 17 00:00:00 2001 From: Erik Simon Date: Wed, 22 Jul 2026 23:47:20 +0200 Subject: [PATCH] deploy: harden kexec-local, key vault items by config, add VM test kexec-local could never actually jump. nixos-images' kexec-run.sh ends with `nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &` and returns immediately, so the EXIT trap's `rm -rf "$stage"` deleted the kexec binary out from under the sleeping shell. The box stayed on the old kernel and it looked like a slow boot. Clear the trap before jumping, verify /sys/kernel/kexec_loaded, then sleep past the timer. Preflight everything before the point of no return, since this jumps the machine you are typing at: CONFIG_KEXEC, kernel lockdown, exec-capable staging dir, free space, RAM vs image size, and that the tarball holds all five expected files. Stage on /var/tmp rather than /tmp because kexec-run.sh appends to initrd in place and execs from that directory. sync before jumping (kexec -e skips unmount). Confirmation prompt naming the host, since run in the wrong terminal this kexecs the laptop; --yes skips it. Drop the ssh-keygen -R added to the remote kexec path: kexec-run.sh copies /etc/ssh/ssh_host_* into the appended initrd and restore-remote-access.nix installs them back, so the host key survives the jump. Proton Pass items are now keyed by instead of , since the address is incidental and the config name is stable. kexec therefore takes . Resolve titles among --filter-state active items first: a trashed item with the same title shadowed the active one and returned an empty password, which is indistinguishable from "no entry" and silently fell back to prompting (hit on darman@neptun). Other fixes: replace `ls glob | head -1` (returns empty with exit 0 on no match) with a helper that dies; guard against untracked hosts/ since flakes ignore untracked files; feed the sudo password more than once under setsid; handle empty arrays under set -u; tolerate empty FSTYPE in the SD-card root partition lookup; preflight zstdcat/dd/lsblk before the destructive dd; list image and flash in the usage strings. Add checks.x86_64-linux.kexec-local, a VM test driving the real script. It is the only way to exercise kexec-local, which cannot be rehearsed on hardware. It asserts the box left the old kernel, returned as nixos-installer, lost its old /run, and kept its ssh host key. HOMELAB_KEXEC_TARBALL lets it reuse a prebuilt installer instead of building ~500MB inside the guest. Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 33 ++++- README.md | 2 +- flake.nix | 165 ++++++++++++++++++++- scripts/deploy | 381 +++++++++++++++++++++++++++++++++++++++++-------- 4 files changed, 521 insertions(+), 60 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index a7d3e0e..171a8a3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,11 +38,22 @@ Deploy (from a non-NixOS laptop too — runs nixos-rebuild/nixos-anywhere via `n ``` ./scripts/deploy switch # daily rebuild + activate ./scripts/deploy install # first install (nixos-anywhere, wipes OS disk) -./scripts/deploy kexec # RO-root box (ZimaOS): kexec into a RAM installer first +./scripts/deploy kexec # RO-root box (ZimaOS): kexec into a RAM installer first +sudo ./scripts/deploy kexec-local [--yes] # kexec THIS box (no ssh); confirm prompt unless --yes ./scripts/deploy image mercury # build the aarch64 SD image ./scripts/deploy flash mercury /dev/sdX # build + write SD + drop the sops age key ``` +Both password prompts are auto-filled from the "HomeLab" Proton Pass vault, keyed by +**``, not ``** — items `darman@` (sudo) and `root@` (ssh). +A *trashed* Proton Pass item with the same title shadows the active one and yields an +empty password, so the script resolves the title among `--filter-state active` items +first; a plain `pass-cli item view --item-title` silently returns the trashed copy and +you get an interactive prompt with no explanation. + +`HOMELAB_KEXEC_TARBALL` (+ `_CPIO` / `_GZIP`) makes `kexec`/`kexec-local` reuse a +prebuilt installer instead of rebuilding ~500MB. The VM test below uses this. + Secrets (needs the admin age key at `~/.config/sops/age/keys.txt`): ``` ./scripts/edit_secrets secrets/.yaml @@ -55,8 +66,16 @@ nix build .#nixosConfigurations.mercury-vm.config.system.build.vm -o result ./result/bin/run-mercury-vm-vm # ssh -p 2223 darman@localhost (pw: test) # jupiter services as a VirtualBox OVA nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA +# end-to-end VM test of `deploy kexec-local` (~45s once the tarball is built) +nix build .#checks.x86_64-linux.kexec-local -L ``` +`checks.kexec-local` is the only way to exercise `kexec-local` at all: it jumps the +machine you are typing at, so it cannot be rehearsed on real hardware and a failure +looks exactly like a slow boot. It asserts the box actually left the old kernel +(SSH drops then returns), came back as `nixos-installer`, lost its old `/run`, and +kept its ssh host key. Run it after ANY change to the kexec paths. + ## Secrets (sops-nix) - Each `secrets/.yaml` is encrypted to the **admin** key (edit) + that **host's** @@ -93,3 +112,15 @@ nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA - `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the `./scripts/deploy kexec` step that streams a RAM installer (with static cpio/gzip since ZimaOS lacks them). +- **`kexec/run` jumps ~6s AFTER it returns**: nixos-images' `kexec-run.sh` ends with + `nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &`. So the staging dir must OUTLIVE the + script — an `rm -rf` in an EXIT trap deletes the binary that performs the jump and the + box silently stays on the old kernel. `kexec-local` clears its trap before jumping and + then sleeps 60s on purpose. Covered by `checks.kexec-local`. +- **The kexec installer KEEPS the box's ssh host key**: `kexec-run.sh` copies + `/etc/ssh/ssh_host_*` into the appended initrd and `restore-remote-access.nix` installs + them back. So do NOT `ssh-keygen -R` after a kexec — the key does not change, and + clearing it just throws away the known_hosts record. +- **`kexec-local` stages on `/var/tmp`, not `/tmp`**: `kexec-run.sh` appends a fresh cpio + to `kexec/initrd` in place and execs binaries from that dir, so a size-capped or + `noexec` tmpfs gives a half-written initrd or a bare "Permission denied". diff --git a/README.md b/README.md index f33b769..7c277b6 100644 --- a/README.md +++ b/README.md @@ -89,7 +89,7 @@ Manual alternative (USB ISO): boot installer, `disko` the disk, then All arguments mandatory — no default host, no default config. ``` -./deploy kexec # headless kexec into a RAM installer (RO-root box) +./deploy kexec # headless kexec into a RAM installer (RO-root box) ./deploy install # first install; wipes OS disk, ships host key ./deploy switch # rebuild + activate on a running host ./deploy boot|test # stage for next boot / activate without boot entry diff --git a/flake.nix b/flake.nix index d178ff7..99730ad 100644 --- a/flake.nix +++ b/flake.nix @@ -20,6 +20,10 @@ url = "github:nix-community/nixos-images"; inputs.nixpkgs.follows = "nixpkgs"; }; + home-manager = { + url = "github:nix-community/home-manager/release-26.05"; + inputs.nixpkgs.follows = "nixpkgs"; + }; mediamanager-nix = { url = "github:strangeglyph/mediamanager-nix"; inputs.nixpkgs.follows = "nixpkgs"; @@ -30,7 +34,7 @@ authentik-nix.url = "github:nix-community/authentik-nix"; }; - outputs = { self, nixpkgs, disko, sops-nix, nixos-images, mediamanager-nix, authentik-nix, ... }@inputs: + outputs = { self, nixpkgs, disko, sops-nix, nixos-images, home-manager, mediamanager-nix, authentik-nix, ... }@inputs: let system = "x86_64-linux"; in @@ -59,6 +63,19 @@ ]; }; + # terra — Ryzen 9 5900X desktop (MSI MS-7A32). Replaces CachyOS on the + # OS SSD; Hyprland desktop + tailnet node. See hosts/terra/*. + terra = nixpkgs.lib.nixosSystem { + inherit system; + specialArgs = { inherit inputs; }; + modules = [ + disko.nixosModules.disko + sops-nix.nixosModules.sops + home-manager.nixosModules.home-manager + ./hosts/terra/configuration.nix + ]; + }; + # mercury — Raspberry Pi 3B+ (aarch64), DNS/DHCP. Boots from an SD image: # nix build .#nixosConfigurations.mercury.config.system.build.sdImage # (aarch64 build — needs binfmt/qemu on this x86 host, or a remote/aarch64 @@ -146,5 +163,151 @@ ]; }; }; + + # VM test for `./scripts/deploy kexec-local`. Run: + # nix build .#checks.x86_64-linux.kexec-local -L + # + # Worth having because kexec-local is the one command that cannot be + # rehearsed on real hardware: it jumps the machine you are typing at, and + # a failure looks exactly like a slow boot. It regression-tests the + # subtle one — kexec-run.sh backgrounds `sleep 6 && kexec -e` and returns, + # so anything that cleans up the staging dir on exit deletes the binary + # that performs the jump and the box silently never leaves the old kernel. + # + # After the jump the test driver's backdoor is gone with the old kernel, + # so the installer is driven over a forwarded ssh port instead (the same + # approach nixos-images uses in its own kexec test). + checks.${system}.kexec-local = + let + pkgs = nixpkgs.legacyPackages.${system}; + tarball = self.nixosConfigurations.kexec.config.system.build.kexecInstallerTarball; + sshKey = nixos-images + "/nix/kexec-installer/ssh-keys/id_ed25519"; + in + pkgs.testers.runNixOSTest { + name = "deploy-kexec-local"; + + nodes.machine = { modulesPath, ... }: { + imports = [ (modulesPath + "/profiles/minimal.nix") ]; + virtualisation.vlans = [ ]; + # kexec-local refuses to run if RAM < 3x the installer image, and + # the staging dir needs ~3x the tarball on /var/tmp. + virtualisation.memorySize = 4 * 1024; + virtualisation.diskSize = 12 * 1024; + virtualisation.forwardPorts = [{ host.port = 2222; guest.port = 22; }]; + + services.openssh.enable = true; + users.users.root.openssh.authorizedKeys.keyFiles = [ "${sshKey}.pub" ]; + + # Everything the script shells out to, minus nix — the test uses the + # HOMELAB_KEXEC_* hook so no build happens inside the VM. + environment.systemPackages = with pkgs; [ + bash gnutar coreutils findutils util-linux cpio gzip + ]; + system.extraDependencies = [ tarball pkgs.cpio pkgs.gzip ]; + + environment.etc."deploy".source = ./scripts/deploy; + }; + + testScript = /* python */ '' + import os, shutil, subprocess, tempfile, time + + start_all() + machine.wait_for_unit("sshd.service") + + # ssh refuses a private key that is group/world readable, and nix + # store paths are 0444 — copy it out and tighten the mode. + keydir = tempfile.mkdtemp() + key = os.path.join(keydir, "id_ed25519") + shutil.copyfile("${sshKey}", key) + os.chmod(key, 0o600) + + def ssh(cmd, check=True, stdout=None): + return subprocess.run( + [ "${pkgs.openssh}/bin/ssh", + "-o", "StrictHostKeyChecking=no", + "-o", "UserKnownHostsFile=/dev/null", + "-o", "ConnectTimeout=1", + "-i", key, + "-p", "2222", "root@127.0.0.1", "--" ] + cmd, + text=True, check=check, stdout=stdout) + + machine.succeed("install -Dm755 /etc/deploy /root/deploy") + + # systemd-run starts units with a bare PATH that lacks + # /run/current-system/sw/bin, so `#!/usr/bin/env bash` cannot even + # resolve bash, let alone tar/findmnt/nohup. Set it explicitly. + env = ( + " --setenv=PATH=/run/wrappers/bin:/run/current-system/sw/bin" + " --setenv=HOMELAB_KEXEC_TARBALL=${tarball}/nixos-kexec-installer-${system}.tar.gz" + " --setenv=HOMELAB_KEXEC_CPIO=${pkgs.cpio}/bin/cpio" + " --setenv=HOMELAB_KEXEC_GZIP=${pkgs.gzip}/bin/gzip" + ) + # Same values for the foreground (non-systemd-run) invocation below. + envsh = ( + "HOMELAB_KEXEC_TARBALL=${tarball}/nixos-kexec-installer-${system}.tar.gz" + " HOMELAB_KEXEC_CPIO=${pkgs.cpio}/bin/cpio" + " HOMELAB_KEXEC_GZIP=${pkgs.gzip}/bin/gzip" + ) + + # Marker on a tmpfs: it must NOT survive the jump, proving we really + # booted a new kernel rather than just restarting a service. + machine.succeed("touch /run/pre-kexec-marker") + host_key_before = machine.succeed("cat /etc/ssh/ssh_host_ed25519_key.pub").strip() + + while ssh(["true"], check=False).returncode != 0: + time.sleep(1) + + # Refuses without --yes when stdin is not a tty (read gets EOF). + # Must reach the confirmation prompt, so it needs the same env — + # otherwise it just dies early on the nix build and proves nothing. + out = machine.fail(f"{envsh} /root/deploy kexec-local &1") + assert "using prebuilt kexec installer" in out, \ + f"never reached the prompt, so the refusal proves nothing:\n{out}" + + # systemd-run so the call returns immediately: the script stays + # alive ~60s on purpose, outliving kexec-run.sh's `sleep 6`. + machine.succeed(f"systemd-run --collect --unit=kexec-local{env} /root/deploy kexec-local --yes") + + print("waiting for the jump...") + deadline = time.time() + 300 + while ssh(["true"], check=False).returncode == 0: + # Surface a dead unit immediately instead of stalling until the + # deadline and blaming "never left the old kernel". + st = ssh(["systemctl", "is-active", "kexec-local"], + check=False, stdout=subprocess.PIPE).stdout or "" + if st.strip() in ("failed", "inactive"): + # NB: not `log` — the driver already binds that name to its + # AbstractLogger and the type check rejects the shadowing. + unit_log = ssh(["journalctl", "-u", "kexec-local", "--no-pager"], + check=False, stdout=subprocess.PIPE).stdout or "" + raise AssertionError( + f"kexec-local.service ended ({st.strip()}) without jumping:\n{unit_log}") + assert time.time() < deadline, "machine never left the old kernel" + time.sleep(1) + + print("waiting for the installer...") + deadline = time.time() + 300 + while ssh(["true"], check=False).returncode != 0: + assert time.time() < deadline, "installer never came up" + time.sleep(1) + + # It really is the RAM installer, not the old system. + host = ssh(["hostname"], stdout=subprocess.PIPE).stdout.strip() + assert host == "nixos-installer", f"hostname is {host}, not nixos-installer" + + assert ssh(["ls", "/run/pre-kexec-marker"], check=False).returncode != 0, \ + "old /run survived — this was not a fresh kernel" + + # The host key is carried across (kexec-run.sh copies /etc/ssh into + # the appended initrd), which is why `kexec` does no ssh-keygen -R. + host_key_after = ssh( + ["cat", "/etc/ssh/ssh_host_ed25519_key.pub"], stdout=subprocess.PIPE + ).stdout.strip() + assert host_key_before == host_key_after, \ + f"host key changed: {host_key_before} != {host_key_after}" + + machine.crash() + ''; + }; }; } diff --git a/scripts/deploy b/scripts/deploy index 98aad7b..4998d24 100755 --- a/scripts/deploy +++ b/scripts/deploy @@ -1,12 +1,27 @@ #!/usr/bin/env bash # Deploy a NixOS host from this flake. ALL arguments are mandatory (no defaults). # -# ./deploy kexec headless kexec into a RAM installer, for a -# read-only-root box (ZimaOS) where -# nixos-anywhere can't ssh-copy-id. Ships our -# SSH login key. Then run `install`. -# ./deploy install first install (nixos-anywhere). Wipes the -# OS disk. Ships the host's sops key. +# ./deploy kexec headless kexec into a RAM installer, for a +# read-only-root box (ZimaOS) where +# nixos-anywhere can't ssh-copy-id. Ships our +# SSH login key. Then run `install`. +# is only used to look up the vault item. +# ./deploy kexec-local [--yes] kexec THIS machine into the RAM installer, +# no ssh/second machine involved. Run as root, +# locally, on the box you're installing onto. +# Disks are untouched; console drops for +# ~1-2 min then comes back as the installer. +# Prompts for confirmation (--yes skips it), +# because run on the wrong terminal this +# kexecs your laptop. TMPDIR (default +# /var/tmp) must be exec-capable and hold +# ~3x the tarball. +# Then run `install localhost`. +# ./deploy install first install. Wipes the OS disk. Ships the +# host's sops key. =localhost/127.0.0.1 +# skips nixos-anywhere/ssh and runs disko + +# nixos-install directly against /mnt (use +# after `kexec-local`, or on a live ISO). # ./deploy switch rebuild + activate on a running host. # ./deploy boot stage for next boot, don't activate now. # ./deploy test activate without adding a boot entry. @@ -22,11 +37,14 @@ # # Password prompts are auto-filled from the "HomeLab" Proton Pass vault when # `pass-cli` is installed and logged in; otherwise every command prompts exactly -# as before. Items: +# as before. Both items are keyed by , never by : the address is +# incidental (DHCP, a new box, localhost) while the config name is the stable +# identity of the machine being built. # darman@ darman's sudo password (switch/boot/test) -# root@ root's ssh password (kexec/install) +# root@ root's ssh password (kexec/install) # Override with HOMELAB_PASS_ITEM / HOMELAB_PASS_ROOT_ITEM / HOMELAB_PASS_VAULT. set -euo pipefail +shopt -s nullglob # Locate the repo root (flake dir) regardless of where this script lives on disk. SCRIPT_DIR="$(cd "$(dirname "$(realpath "$0")")" && pwd)" @@ -36,16 +54,93 @@ export PATH="/nix/var/nix/profiles/default/bin:$PATH" die() { echo "error: $*" >&2; exit 1; } +need() { command -v "$1" >/dev/null 2>&1 || die "missing required tool: $1"; } + +# Exactly one path matching a glob, or die. `ls glob | head -1` silently yields +# an empty string when nothing matches (head exits 0, so set -e never fires) and +# the failure only surfaces later as a confusing tar/dd error. +one_match() { + local what="$1"; shift + local f=("$@") # caller expands the glob (nullglob is on) + [ "${#f[@]}" -gt 0 ] || die "no $what found — did the build actually produce one?" + printf '%s\n' "${f[0]}" +} + +# Sets tb / cpio / bbox — the kexec tarball plus the static cpio+gzip that +# kexec-run.sh needs on PATH to rebuild its initrd. +# +# HOMELAB_KEXEC_TARBALL (with _CPIO / _GZIP) short-circuits the build and uses a +# prebuilt installer instead. That lets the VM test in flake.nix drive this +# script offline, and lets you re-kexec a box without rebuilding ~500MB. +kexec_artifacts() { + if [ -n "${HOMELAB_KEXEC_TARBALL:-}" ]; then + tb="$HOMELAB_KEXEC_TARBALL" + [ -f "$tb" ] || die "HOMELAB_KEXEC_TARBALL=$tb is not a file" + cpio="${HOMELAB_KEXEC_CPIO:-$(command -v cpio || true)}" + bbox="${HOMELAB_KEXEC_GZIP:-$(command -v gzip || true)}" + { [ -n "$cpio" ] && [ -n "$bbox" ]; } \ + || die "set HOMELAB_KEXEC_CPIO / HOMELAB_KEXEC_GZIP, or put cpio+gzip on PATH" + echo ">> using prebuilt kexec installer: $tb" + else + need nix + echo ">> building kexec installer + static tools" + nix build .#nixosConfigurations.kexec.config.system.build.kexecInstallerTarball \ + -o result-kexec + tb="$(one_match 'kexec tarball' result-kexec/*.tar.gz)" + cpio="$(nix build --no-link --print-out-paths nixpkgs#pkgsStatic.cpio)/bin/cpio" + bbox="$(nix build --no-link --print-out-paths nixpkgs#pkgsStatic.busybox)/bin/busybox" + fi +} + +# Flakes only see git-tracked files: an untracked hosts// is silently +# invisible to `nix build`/`nixos-install`, which then fails obscurely or builds +# a stale config. Check before doing anything destructive. +require_tracked() { + local config="$1" cfgfile="hosts/$1/configuration.nix" + [ -e "$cfgfile" ] || die "no $cfgfile in the repo" + git -C "$REPO" ls-files --error-unmatch "$cfgfile" >/dev/null 2>&1 \ + || die "$cfgfile is untracked — 'git add hosts/$config' first (flakes ignore untracked files)" +} + # The password field of a Proton Pass item ("--field password" prints the bare # value, one line), or empty if pass-cli is missing / logged out / has no such # item — every caller then falls back to the normal interactive prompt. +# +# Resolve the title to an item id among ACTIVE items first, because `item view +# --item-title` has no state filter: Proton Pass keeps deleted items in the +# trash, and if a trashed item shares the title, view can match THAT one and +# return an empty password with exit 0. Empty is indistinguishable from "no such +# item", so the only symptom is a silent fall back to the interactive prompt +# even though the vault clearly holds the entry. (Hit for real on darman@neptun, +# which had an Active and a Trashed copy.) proton_pass_password() { - local title="$1" + local title="$1" vault="${HOMELAB_PASS_VAULT:-HomeLab}" id pw command -v pass-cli >/dev/null 2>&1 || return 0 - pass-cli item view \ - --vault-name "${HOMELAB_PASS_VAULT:-HomeLab}" \ - --item-title "$title" \ - --field password --output human 2>/dev/null | head -1 + + # Lines look like: - [ITEM_ID]: the title (state=Active) + id="$(pass-cli item list --vault-name "$vault" --filter-state active \ + --output human 2>/dev/null \ + | awk -v t="$title" ' + { i = index($0, "]: "); if (i == 0) next + id = substr($0, 4, i - 4) + rest = substr($0, i + 3) + sub(/ \(state=[^)]*\)$/, "", rest) + if (rest == t) { print id; exit } }' || true)" + + if [ -n "$id" ]; then + pw="$(pass-cli item view --vault-name "$vault" --item-id "$id" \ + --field password --output human 2>/dev/null | head -1 || true)" + # Resolved an active item but its password field is blank: worth saying so, + # otherwise this looks identical to having no vault entry at all. + [ -n "$pw" ] || echo ">> vault item '$title' resolved but its password field is empty" >&2 + else + # No active match — fall back to the title lookup so an older pass-cli + # without --filter-state still works exactly as it used to. + pw="$(pass-cli item view --vault-name "$vault" --item-title "$title" \ + --field password --output human 2>/dev/null | head -1 || true)" + fi + + printf '%s' "$pw" } # Path to an sshpass binary (system one, else built from nixpkgs). Empty if @@ -56,20 +151,18 @@ sshpass_bin() { | sed 's|$|/bin/sshpass|' } -cmd="${1:-}"; [ -n "$cmd" ] || die "usage: ./deploy ..." +cmd="${1:-}"; [ -n "$cmd" ] || die "usage: ./deploy ..." case "$cmd" in kexec) - host="${2:-}"; [ -n "$host" ] || die "usage: ./deploy kexec " + config="${2:-}"; host="${3:-}" + { [ -n "$config" ] && [ -n "$host" ]; } || die "usage: ./deploy kexec " + + need ssh; need scp - echo ">> building kexec installer + static tools" - nix build .#nixosConfigurations.kexec.config.system.build.kexecInstallerTarball \ - -o result-kexec - tb="$(ls result-kexec/*.tar.gz | head -1)" # kexec/run rebuilds an initrd with `cpio` + `gzip` from PATH — ZimaOS lacks # both. Ship static ones: GNU cpio (reliable -o -H newc), busybox as gzip. - cpio="$(nix build --no-link --print-out-paths nixpkgs#pkgsStatic.cpio)/bin/cpio" - bbox="$(nix build --no-link --print-out-paths nixpkgs#pkgsStatic.busybox)/bin/busybox" + kexec_artifacts # One password prompt: multiplex scp + ssh over a shared control connection. cm="/tmp/homelab-cm-%r@%h:%p" @@ -78,13 +171,21 @@ case "$cmd" in # Root's password from Proton Pass, fed to ssh/scp via sshpass -e. Only the # first (master) connection authenticates; the rest ride the control socket. - # SSHPASS is exported into the sshpass child only, never onto a command line. + # + # SSHPASS is exported here rather than passed as `env SSHPASS=... sshpass`. + # Both end up equally safe at rest: `env` execs its target immediately, so + # the assignment is only in argv for the sub-millisecond before exec, after + # which /proc/PID/cmdline reads plain `sshpass -e`. Exporting just closes + # that race window and drops a process. Either way the secret lives in the + # child's environ, which is readable by the owner and root only. sp=() - root_pw="$(proton_pass_password "${HOMELAB_PASS_ROOT_ITEM:-root@$host}" || true)" + root_item="${HOMELAB_PASS_ROOT_ITEM:-root@$config}" + root_pw="$(proton_pass_password "$root_item" || true)" if [ -n "$root_pw" ]; then sshpass="$(sshpass_bin || true)" if [ -n "$sshpass" ]; then - sp=(env "SSHPASS=$root_pw" "$sshpass" -e) + export SSHPASS="$root_pw" + sp=("$sshpass" -e) # sshpass drives the password prompt; don't let a key/agent short-circuit # into an interactive one for a host that only accepts passwords. o+=(-o PreferredAuthentications=password -o PubkeyAuthentication=no) @@ -92,26 +193,150 @@ case "$cmd" in echo ">> sshpass unavailable — falling back to the interactive prompt" >&2 fi fi + unset root_pw - if [ ${#sp[@]} -gt 0 ]; then - echo ">> connecting to root@$host (password from Proton Pass)" + if [ "${#sp[@]}" -gt 0 ]; then + echo ">> connecting to root@$host (password from Proton Pass: $root_item)" else echo ">> connecting to root@$host (enter the root password once)" fi - "${sp[@]}" ssh "${o[@]}" "root@$host" 'mkdir -p /tmp/bin' + "${sp[@]+"${sp[@]}"}" ssh "${o[@]}" "root@$host" 'mkdir -p /tmp/bin' scp "${o[@]}" "$cpio" "root@$host:/tmp/bin/cpio" scp "${o[@]}" "$bbox" "root@$host:/tmp/bin/gzip" # busybox as gzip (argv0) - unset root_pw echo ">> streaming installer + kexec-ing. SSH drops as the box jumps into the" echo " RAM installer. Disks are untouched." + rc=0 ssh "${o[@]}" "root@$host" \ 'chmod +x /tmp/bin/*; mkdir -p /tmp/k && tar -C /tmp/k -xzf - && PATH=/tmp/bin:$PATH /tmp/k/kexec/run' \ - < "$tb" || true + < "$tb" || rc=$? + # A dropped connection IS the success case here, so this can't be fatal — + # but a full /tmp, a tar error or a missing static tool looks identical from + # the outside, so at least surface the code instead of swallowing it. + [ "$rc" -eq 0 ] || echo ">> ssh exited $rc — expected if the box jumped; suspect this if install can't connect" >&2 ssh "${o[@]}" -O exit "root@$host" 2>/dev/null || true # close control socket + unset SSHPASS + + # NB: no ssh-keygen -R here on purpose. kexec-run.sh copies /etc/ssh/ssh_host_* + # into the appended initrd and restore-remote-access.nix installs them back + # into the installer's /etc/ssh, so the host key SURVIVES the jump. Clearing + # known_hosts would just throw away the TOFU record for no reason. + echo ">> box is kexec-ing. Wait ~1-2 min for the installer + network, then:" - echo " ./deploy install $host" + echo " ./deploy install $config $host" + ;; + + kexec-local) + # No ssh, no second machine: build the same RAM installer as `kexec`, but + # run it directly on this box (you're sitting at it). The current shell + # drops when the kernel switches, same as any reboot — that's expected, + # not a failure. Disks are untouched; only the running kernel changes. + # + # This is a one-way trip on the machine you are typing at, so every check + # that can fail is done BEFORE the point of no return, and nothing that the + # jump depends on is cleaned up behind it (see the trap discussion below). + [ "$(id -u)" = 0 ] || die "kexec-local must run as root (sudo ./deploy kexec-local)" + + assume_yes="" + [ "${2:-}" = "--yes" ] && assume_yes=1 + + need tar; need install; need mktemp; need find; need sync; need nohup + + # --- preflight: reasons the jump would fail, checked while it's still safe -- + # CONFIG_KEXEC. Without it kexec --load fails and you've built ~1GB for + # nothing; with lockdown it fails at load time too (Secure Boot blocks the + # kexec_load syscall unless the image is signed). + [ -e /sys/kernel/kexec_loaded ] \ + || die "this kernel has no kexec support (CONFIG_KEXEC) — boot install media instead" + if [ -r /sys/kernel/security/lockdown ] \ + && ! grep -q '\[none\]' /sys/kernel/security/lockdown 2>/dev/null; then + die "kernel lockdown is active — kexec_load is blocked. Disable Secure Boot, or boot install media." + fi + + kexec_artifacts + + # Staging lives on /var/tmp, not /tmp: kexec-run.sh APPENDS a fresh cpio to + # kexec/initrd in place and runs binaries out of this directory, so it needs + # real space and exec permission. A tmpfs /tmp is both size-capped (ENOSPC + # mid-append = a half-written initrd) and frequently noexec. + stage="$(TMPDIR="${TMPDIR:-/var/tmp}" mktemp -d)" + trap 'rm -rf "$stage"' EXIT + + # noexec would only surface as a cryptic "Permission denied" on kexec/run. + printf '#!/bin/sh\nexit 0\n' > "$stage/.execmove" + chmod +x "$stage/.execmove" + "$stage/.execmove" 2>/dev/null \ + || die "$stage is mounted noexec — set TMPDIR to an exec-capable filesystem" + rm -f "$stage/.execmove" + + # Need room for the tarball, its extracted contents, and the appended cpio. + tb_sz="$(stat -Lc %s "$tb")" + avail="$(df -B1 --output=avail "$stage" | tail -1 | tr -d ' ')" + [ "$avail" -ge $(( tb_sz * 3 )) ] \ + || die "only $(( avail / 1024 / 1024 ))MB free on $stage, need ~$(( tb_sz * 3 / 1024 / 1024 ))MB — set TMPDIR elsewhere" + + install -Dm755 "$cpio" "$stage/bin/cpio" + install -Dm755 "$bbox" "$stage/bin/gzip" # busybox as gzip (argv0) + mkdir -p "$stage/k" && tar -C "$stage/k" -xzf "$tb" + + # Fail loudly here rather than with a bare "not found" from kexec/run. + for f in run kexec bzImage initrd ip; do + [ -f "$stage/k/kexec/$f" ] || die "kexec tarball is missing kexec/$f — bad build?" + done + + # The installer root is a RAM disk: the initrd has to fit in memory with + # room to unpack. Refuse rather than OOM halfway into the new kernel, where + # there is no way back except the power button. + img_sz="$(( $(stat -Lc %s "$stage/k/kexec/initrd") + $(stat -Lc %s "$stage/k/kexec/bzImage") ))" + mem_kb="$(awk '/^MemTotal:/{print $2}' /proc/meminfo)" + [ "$(( mem_kb * 1024 ))" -ge "$(( img_sz * 3 ))" ] \ + || die "only $(( mem_kb / 1024 ))MB RAM for a $(( img_sz / 1024 / 1024 ))MB RAM-disk installer — too tight, boot install media instead" + + # --- confirmation: this script normally runs on the LAPTOP ------------------ + # A stray `kexec-local` here would jump the machine you develop on, not the + # target. Name it explicitly so a wrong-terminal mistake is visible. + echo ">> about to kexec THIS machine:" + echo " hostname: $(uname -n)" + echo " kernel: $(uname -r)" + echo " root: $(findmnt -no SOURCE / 2>/dev/null || echo '?')" + echo " Disks are untouched; only the running kernel changes. Console drops" + echo " for ~1-2 min, then comes back as the installer." + if [ -z "$assume_yes" ]; then + read -rp ">> type 'yes' to kexec $(uname -n) into the RAM installer: " ok + [ "$ok" = yes ] || die "aborted" + fi + + # kexec -e jumps straight to the new kernel: no unmount, no journal flush, + # no systemd shutdown. Anything still in page cache is lost and the OS disk + # is left dirty. Cheap insurance, since the box may yet be rebooted back. + sync + + echo ">> loading the new kernel" + if ! PATH="$stage/bin:$PATH" "$stage/k/kexec/run"; then + rm -rf "$stage" + die "kexec/run failed — machine untouched, still on the old kernel" + fi + + # kexec --load succeeded iff the kernel now reports a loaded image. If this + # is 0 the background `kexec -e` below will do nothing and we'd hang forever + # waiting for a jump that cannot happen. + [ "$(cat /sys/kernel/kexec_loaded 2>/dev/null || echo 0)" = 1 ] \ + || { rm -rf "$stage"; die "kexec reported success but no image is loaded — aborting"; } + + # THE trap MUST GO NOW. kexec-run.sh backgrounds `nohup sh -c "sleep 6 && + # $SCRIPT_DIR/kexec -e"` and returns immediately, so the binary that + # performs the jump still has to exist ~6s after this script would normally + # exit. Letting the EXIT trap rm -rf "$stage" deletes it out from under that + # sleeping shell and the machine silently never jumps. + trap - EXIT + + sync + echo ">> kernel loaded; jumping in ~6s. (staging left at $stage on purpose —" + echo " the backgrounded kexec still needs it; it's gone after the jump.)" + # Outlive the sleep 6 so the jump happens while this script is still alive. + sleep 60 + die "still here after 60s — the jump did not happen. Check dmesg; the old system is intact." ;; install) @@ -120,35 +345,61 @@ case "$cmd" in hostkey="$HOME/.config/homelab/$config/ssh_host_ed25519_key" [ -f "$hostkey" ] || die "missing host key: $hostkey" [ -d "./hosts/$config" ] || die "no ./hosts/$config directory in the repo" + require_tracked "$config" - # Stage the pre-generated SSH host key so sops can decrypt on boot #1. - stage="$(mktemp -d)" - trap 'rm -rf "$stage"' EXIT - install -Dm600 "$hostkey" "$stage/etc/ssh/ssh_host_ed25519_key" - install -Dm644 "$hostkey.pub" "$stage/etc/ssh/ssh_host_ed25519_key.pub" + if [ "$host" = "localhost" ] || [ "$host" = "127.0.0.1" ]; then + # Local install: no ssh, no nixos-anywhere. Run after `kexec-local` (or + # from a live ISO) so /mnt is free to wipe — this IS the box, no second + # machine in the loop, so skip straight to disko + nixos-install. + [ "$(id -u)" = 0 ] || die "local install must run as root" + [ -f "./hosts/$config/disk-config.nix" ] || die "no ./hosts/$config/disk-config.nix" - echo ">> nixos-anywhere .#$config onto root@$host (OS disk WILL be wiped)" - anywhere=(--flake ".#$config" - --extra-files "$stage" - --generate-hardware-config nixos-generate-config "./hosts/$config/hardware-configuration.nix" - --target-host "root@$host") + echo ">> disko .#$config onto this box's OS disk (WILL be wiped)" + nix run github:nix-community/disko -- \ + --mode disko "./hosts/$config/disk-config.nix" - # nixos-anywhere's --env-password reads root's ssh password from $SSHPASS - # (it ships its own sshpass), so a vault hit skips the ssh-copy-id prompt. - root_pw="$(proton_pass_password "${HOMELAB_PASS_ROOT_ITEM:-root@$host}" || true)" - if [ -n "$root_pw" ]; then - echo ">> root ssh password from Proton Pass" - env "SSHPASS=$root_pw" nix run github:nix-community/nixos-anywhere -- \ - --env-password "${anywhere[@]}" + echo ">> installing sops host key so it can decrypt on boot #1" + install -Dm600 "$hostkey" /mnt/etc/ssh/ssh_host_ed25519_key + install -Dm644 "$hostkey.pub" /mnt/etc/ssh/ssh_host_ed25519_key.pub + + echo ">> nixos-install .#$config into /mnt" + nixos-install --root /mnt --flake ".#$config" else - nix run github:nix-community/nixos-anywhere -- "${anywhere[@]}" + # Stage the pre-generated SSH host key so sops can decrypt on boot #1. + stage="$(mktemp -d)" + trap 'rm -rf "$stage"' EXIT + install -Dm600 "$hostkey" "$stage/etc/ssh/ssh_host_ed25519_key" + install -Dm644 "$hostkey.pub" "$stage/etc/ssh/ssh_host_ed25519_key.pub" + + echo ">> nixos-anywhere .#$config onto root@$host (OS disk WILL be wiped)" + anywhere=(--flake ".#$config" + --extra-files "$stage" + --generate-hardware-config nixos-generate-config "./hosts/$config/hardware-configuration.nix" + --target-host "root@$host") + + # nixos-anywhere's --env-password reads root's ssh password from $SSHPASS + # (it ships its own sshpass), so a vault hit skips the ssh-copy-id prompt. + # Exported rather than `env SSHPASS=...` for consistency with `kexec`; + # see the note there — it's a marginal win, not a leak fix. + root_item="${HOMELAB_PASS_ROOT_ITEM:-root@$config}" + root_pw="$(proton_pass_password "$root_item" || true)" + if [ -n "$root_pw" ]; then + echo ">> root ssh password from Proton Pass ($root_item)" + export SSHPASS="$root_pw" + unset root_pw + nix run github:nix-community/nixos-anywhere -- \ + --env-password "${anywhere[@]}" + unset SSHPASS + else + nix run github:nix-community/nixos-anywhere -- "${anywhere[@]}" + fi fi - unset root_pw ;; switch|boot|test) config="${2:-}"; host="${3:-}" { [ -n "$config" ] && [ -n "$host" ]; } || die "usage: ./deploy $cmd " + require_tracked "$config" echo ">> nixos-rebuild $cmd .#$config on darman@$host" # --ask-sudo-password, not the deprecated --use-remote-sudo: common.nix sets @@ -167,8 +418,14 @@ case "$cmd" in # piped stdin. setsid drops the controlling terminal, so getpass falls back # to stdin and takes the vault password (it warns about echo — harmless, # nothing is echoed since the password never reaches the terminal). + # + # Caveat of dropping the tty: EVERY prompt in the subtree now reads this + # stdin, not just the sudo one. Feed the line a few times so a retry or a + # second sudo ask doesn't hit EOF and hang. Anything else that prompts + # (an ssh key passphrase, a host-key confirmation) will still fail — fix + # those out of band rather than by feeding more lines here. echo ">> sudo password from Proton Pass ($item)" - printf '%s\n' "$pw" | setsid -w "${rebuild[@]}" + printf '%s\n%s\n%s\n' "$pw" "$pw" "$pw" | setsid -w "${rebuild[@]}" else "${rebuild[@]}" fi @@ -176,16 +433,22 @@ case "$cmd" in ;; image|flash) - config="${2:-}"; [ -n "$config" ] || die "usage: ./deploy $cmd []" - # Validate the device BEFORE building, so a bad `flash` fails fast. if [ "$cmd" = flash ]; then - dev="${3:-}"; [ -n "$dev" ] || die "usage: ./deploy flash (e.g. /dev/sdX)" + config="${2:-}"; dev="${3:-}" + { [ -n "$config" ] && [ -n "$dev" ]; } \ + || die "usage: ./deploy flash (e.g. /dev/sdX)" + # Validate the device and the tools BEFORE building, so a bad `flash` + # fails fast instead of dying halfway through writing the card. [ -b "$dev" ] || die "$dev is not a block device" + need zstdcat; need dd; need lsblk + else + config="${2:-}"; [ -n "$config" ] || die "usage: ./deploy image " fi + require_tracked "$config" echo ">> building SD image for .#$config (aarch64 needs qemu binfmt)" nix build ".#nixosConfigurations.$config.config.system.build.sdImage" -o result-sd - img="$(ls result-sd/sd-image/*.img.zst | head -1)" + img="$(one_match 'SD image' result-sd/sd-image/*.img.zst)" echo ">> image: $img" [ "$cmd" = image ] && exit 0 @@ -205,8 +468,12 @@ case "$cmd" in echo ">> installing sops age key onto the root partition" sudo partprobe "$dev" 2>/dev/null || sudo blockdev --rereadpt "$dev" 2>/dev/null || true sudo udevadm settle 2>/dev/null || true - # largest ext4 partition = the NixOS root. - rootpart="$(lsblk -blno PATH,FSTYPE,SIZE "$dev" | awk '$2=="ext4"{print $3, $1}' | sort -rn | head -1 | awk '{print $2}')" + # Largest ext4 partition = the NixOS root. -P (KEY="value" pairs) instead + # of a columnar listing: an empty FSTYPE collapses under whitespace + # splitting and shifts every later field. + rootpart="$(lsblk -bPo PATH,FSTYPE,SIZE "$dev" \ + | sed -n 's/^PATH="\([^"]*\)" FSTYPE="ext4" SIZE="\([0-9]*\)"$/\2 \1/p' \ + | sort -rn | head -1 | cut -d' ' -f2)" [ -n "$rootpart" ] || die "no ext4 root partition found on $dev — place $keyfile at /var/lib/sops-nix/age.txt manually" mnt="$(mktemp -d)" sudo mount "$rootpart" "$mnt" @@ -219,6 +486,6 @@ case "$cmd" in ;; *) - die "unknown command '$cmd' (kexec|install|switch|boot|test)" + die "unknown command '$cmd' (kexec|kexec-local|install|switch|boot|test|image|flash)" ;; esac