From e538788907ed8b11134d249246966eeed7966b00 Mon Sep 17 00:00:00 2001 From: Erik Simon Date: Fri, 24 Jul 2026 02:25:14 +0200 Subject: [PATCH] deploy: make the local reinstall path actually work, and fail closed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `install localhost` auto-path added in 0ea9020 could not have completed. Fixed three blockers plus the guard that was silently not guarding. Inside installer-iso the run died before doing anything: - systemd sets no $HOME for a service without User= (SetLoginEnvironment= defaults to false), and this script runs under `set -u`, so it aborted on the bare $HOME with "unbound variable". Added $KEYDIR + Environment=HOME. - the host key it needs to seed /etc/ssh isn't on the ISO at all — that is built from a public repo and carries no credentials on purpose. It now travels on the boot partition, located via homelab.keypart= on the kernel cmdline, and dies with the disko wipe minutes later. Without it sops can't decrypt on boot #1 and mutableUsers locks darman for good. - installation-cd-minimal leaves experimental-features unset, so both `nix run` and `nixos-install --flake` failed. (The nixos-images kexec installer sets them itself, which is why the same branch worked after kexec-local but not from the ISO.) The staging-dir guard passed everything on btrfs: findmnt prints the subvolume as /dev/sdb2[/@], lsblk can't open that, and an empty parent was treated as "different disk" — so it allowed staging the iso on the very disk disko then wiped. terra's current CachyOS root is exactly that layout. Now uses --nofsroot, resolves EVERY whole-disk ancestor (LVM/RAID span several: /mnt/ssd_01 -> sdd+sde), and treats "can't tell" as a hard error. btrfs staging is refused outright — stage-1 mounts a btrfs volume's top level, so an iso inside a subvolume is unreachable. findiso= lost its leading slash whenever the staging mountpoint was /, giving /findisovar/tmp/x.iso and an emergency shell after the reboot. Also: - confirm before rebooting, like flash/kexec-local already do; --yes skips it and is what the ISO passes itself - $BOOT from `bootctl --print-boot-path`, not a hardcoded /boot - free-space checks on both target partitions before the ~1GB copy - `nix run .#disko` / `.#nixos-anywhere` from locked inputs instead of github:... master-of-the-day, resolved while a disk is being wiped - one_match warns instead of silently taking [0]; require_tracked covers every hosts//*.nix; flash traps its mount - drop nixos-images' `inputs.nixpkgs.follows` — that input doesn't exist, it only printed a warning on every nix command Verified: the prepare path exercised under stubs against this box's real disks (btrfs-on-OS-disk, tmpfs, LVM, subdirectory), shellcheck clean, all six configs evaluate, checks.kexec-local still passes. Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 41 +++++++- README.md | 52 +++++++--- flake.lock | 95 +++++++++++++++---- flake.nix | 77 ++++++++++++++- scripts/deploy | 250 ++++++++++++++++++++++++++++++++++++++----------- 5 files changed, 418 insertions(+), 97 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 47c0636..53a0648 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -125,11 +125,42 @@ kept its ssh host key. Run it after ANY change to the kexec paths. loop: it reads `homelab.install=` back off `/proc/cmdline` (set by the prepare step) and re-runs the identical command itself once `homelab-checkout.service` has cloned the repo — the whole reinstall is one command and unattended after the first - reboot. It always ASKS where to stage the iso file (never auto-picks — the wrong disk - here is destroyed mid-install) and refuses if that turns out to be the disk - `disk-config.nix` is about to wipe; `HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for - scripted use. Both this and `kexec-local` self-elevate via `sudo` (`require_root()`) - rather than requiring you to prefix the command yourself. + reboot. It confirms (type `yes`) before rebooting, like `flash`/`kexec-local`; `--yes` + skips that and is what the ISO passes itself. It always ASKS where to stage the iso + file (never auto-picks — the wrong disk here is destroyed mid-install); + `HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for scripted use. Both this and + `kexec-local` self-elevate via `sudo` (`require_root()`) rather than requiring you to + prefix the command yourself. +- **The staging-dir guard must fail CLOSED, and `findmnt` needs `--nofsroot`**: on btrfs + `findmnt -no SOURCE` prints `/dev/sdb2[/@]`, which `lsblk` cannot open, so a naive + parent-device lookup comes back empty. Treating empty as "different disk" silently + allowed staging the iso on the very disk disko then wiped — terra's CachyOS root is + exactly that layout, so it hit the live case. `disks_backing()` (`lsblk -rnso + NAME,TYPE`) returns EVERY whole-disk ancestor because LVM/RAID can span several + (`/mnt/ssd_01` → sdd + sde), and an empty result is a hard error, not a pass. btrfs + staging is refused outright: stage-1 mounts a btrfs volume's TOP level, so an iso + inside a subvolume is unreachable via `findiso=`. +- **`findiso=` must keep its leading slash**: stage-1 tests `[ -e "/findiso$isoPath" ]`, + so stripping the mountpoint prefix off a stagedir whose mountpoint is `/` yields + `var/tmp/x.iso` → `/findisovar/tmp/x.iso` → emergency shell, after you have already + rebooted out of the working OS. +- **The auto-install needs the host key shipped to it, and a `$HOME`**: the ISO is built + from a public repo with no credentials, so `local_install_prepare_and_reboot()` copies + the key onto the boot partition and passes that partition's PARTUUID as + `homelab.keypart=`; the service mounts it and drops the key in + `/root/.config/homelab//` before running the install. Also, systemd does NOT + set `$HOME` for a system service without `User=` (`SetLoginEnvironment=` defaults to + false), and `scripts/deploy` runs under `set -u` — hence `$KEYDIR` instead of a bare + `$HOME`, plus `Environment=HOME=/root` on the unit. +- **`installer-iso` must enable `experimental-features` itself.** `installation-cd-minimal` + leaves them unset, so `nix run` and `nixos-install --flake` both die with "experimental + Nix feature 'nix-command' is disabled". The nixos-images `kexec` installer sets them + itself, which is why the same `install localhost` branch worked after + `kexec-local` but not from the ISO. +- **disko/nixos-anywhere run as `nix run .#disko` / `.#nixos-anywhere`**, from this + flake's locked inputs — not `nix run github:...`. They execute while a disk is being + wiped, so the revision must be the reviewed one in `flake.lock`, and it has to resolve + without network. - **disko wipes only the OS disk** named in `hosts//disk-config.nix`; data disks are plain `fileSystems` in `configuration.nix`. - `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the diff --git a/README.md b/README.md index f8efd27..014f95d 100644 --- a/README.md +++ b/README.md @@ -118,26 +118,50 @@ self-elevates: ./scripts/deploy install terra localhost ``` `scripts/deploy` detects it isn't already inside a live installer (checks -`uname -n`) and instead: builds `installer-iso`'s kernel + initrd + iso image, -asks where to stage the iso file (never auto-picks — the wrong disk here is -destroyed mid-install; `HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for -scripted use), refuses if that resolves to the same disk `disk-config.nix` is -about to wipe, copies the kernel/initrd to the ESP, writes a systemd-boot -**one-shot** entry (`bootctl set-oneshot` — reverts to the normal default -after this one boot, even if it fails) with `findiso=` + `homelab.install=terra` -on its kernel cmdline, and reboots — a real `systemctl reboot`, not kexec. +`uname -n`) and instead: + +1. Asks where to stage the iso file — never auto-picks, because the wrong disk + here is destroyed mid-install (`HOMELAB_INSTALLER_STAGE_DIR` skips the + prompt for scripted use). It **refuses** if that path resolves to a disk + `disk-config.nix` is about to wipe, if it can't work out which physical disk + the path is on at all (fail-closed — LVM and RAID can span several), or if + it's on btrfs (stage-1 mounts a btrfs volume's *top level*, so a path inside + a subvolume never resolves and you boot to an emergency shell). On terra, + `/mnt/hdd_01` is the right answer; the CachyOS root is btrfs on the OS disk + and is rejected on both counts. +2. Prints exactly what it is about to do — OS disk, staging disk, boot entry — + and waits for you to type `yes`. `--yes` skips it; that is what the ISO + passes when it re-runs the command itself. +3. Builds `installer-iso`'s kernel + initrd + iso image, checks both target + partitions have room, then copies the kernel/initrd **and terra's + pre-generated ssh host key** to the boot partition (found via + `bootctl --print-boot-path`, not assumed to be `/boot`) and the iso to the + staging dir. +4. Writes a systemd-boot **one-shot** entry (`bootctl set-oneshot` — reverts to + the normal default after this one boot, even if it fails) with `findiso=` + + `homelab.install=terra` + `homelab.keypart=` on its kernel + cmdline, and reboots — a real `systemctl reboot`, not kexec. The booted installer clones the repo (`homelab-checkout.service`, needs network — it's public now, no credentials involved) and then `homelab-auto-install.service` reads `homelab.install=terra` back off -`/proc/cmdline` and re-runs the exact same `./scripts/deploy install terra -localhost` itself — now genuinely inside the installer, so it takes the -disko + `nixos-install` branch instead of preparing again. Ships terra's -pre-generated host key first so `/run/secrets/*` decrypts on boot #1. The -whole thing is unattended after the initial reboot; ssh into +`/proc/cmdline`, mounts `homelab.keypart=` to pick terra's host key back up +into `/root/.config/homelab/terra/`, and re-runs the exact same +`./scripts/deploy install terra localhost` itself — now genuinely inside the +installer, so it takes the disko + `nixos-install` branch instead of preparing +again. That key is what seeds `/etc/ssh` on the new system, which is what lets +`/run/secrets/*` decrypt on boot #1; it has to travel this way because the ISO +is built from a **public** repo and deliberately carries no credentials. The +copy on the boot partition dies with the disko wipe minutes later. + +The whole thing is unattended after the initial reboot; ssh into `homelab-installer` (same pubkey as the ISO everywhere else) to watch progress — `journalctl -u homelab-checkout -u homelab-auto-install -f`. +> The checkout step resolves `git.mgaction.town`, which goes through mercury's +> pihole on the LAN. If mercury is down, the installer boots fine but never +> gets the repo — fix DNS and `systemctl restart homelab-checkout`. + When it's done, reboot again into the freshly installed NixOS. Then, same as any other host: ``` @@ -155,7 +179,7 @@ All arguments mandatory — no default host, no default config. ``` ./deploy kexec # headless kexec into a RAM installer (RO-root box) -./deploy install # first install; wipes OS disk, ships host key +./deploy install [--yes] # first install; wipes OS disk, ships host key ./deploy switch # rebuild + activate on a running host ./deploy boot|test # stage for next boot / activate without boot entry ./deploy image # build an SD-card image (mercury) diff --git a/flake.lock b/flake.lock index ac5155a..b75b72d 100644 --- a/flake.lock +++ b/flake.lock @@ -179,9 +179,63 @@ "type": "github" } }, + "nix-vm-test": { + "inputs": { + "nixpkgs": [ + "nixos-anywhere", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1783265394, + "narHash": "sha256-cq4YfNFGYzp0NItZP8tEC7xUI8OSgY4fj75AU/NSaPM=", + "owner": "numtide", + "repo": "nix-vm-test", + "rev": "1a587212d2ac8b669c6c32499015f996506b6ba5", + "type": "github" + }, + "original": { + "owner": "numtide", + "repo": "nix-vm-test", + "type": "github" + } + }, + "nixos-anywhere": { + "inputs": { + "disko": [ + "disko" + ], + "nix-vm-test": "nix-vm-test", + "nixos-images": [ + "nixos-images" + ], + "nixos-stable": [ + "nixpkgs" + ], + "nixpkgs": [ + "nixpkgs" + ], + "treefmt-nix": "treefmt-nix" + }, + "locked": { + "lastModified": 1784704708, + "narHash": "sha256-xuzg2Xfalr1X5ah5IPgqB8Vyif+2je1qkCtGxCB+rm0=", + "owner": "nix-community", + "repo": "nixos-anywhere", + "rev": "91fc9b70fc295258c366cce8627efb6f185fd9fb", + "type": "github" + }, + "original": { + "owner": "nix-community", + "repo": "nixos-anywhere", + "type": "github" + } + }, "nixos-images": { "inputs": { - "nixos-stable": "nixos-stable", + "nixos-stable": [ + "nixpkgs" + ], "nixos-unstable": "nixos-unstable" }, "locked": { @@ -198,23 +252,6 @@ "type": "github" } }, - "nixos-stable": { - "locked": { - "lastModified": 1783389287, - "narHash": "sha256-0xIy4dVLqq47rA+mRy0hXDfjhQd4E5PoIns/RmB7nR4=", - "ref": "nixos-26.05", - "rev": "0ad6f47ea4fe188f4bc8f0380f93ae8523337c6c", - "shallow": true, - "type": "git", - "url": "https://github.com/NixOS/nixpkgs" - }, - "original": { - "ref": "nixos-26.05", - "shallow": true, - "type": "git", - "url": "https://github.com/NixOS/nixpkgs" - } - }, "nixos-unstable": { "locked": { "lastModified": 1782175435, @@ -421,6 +458,7 @@ "disko": "disko", "home-manager": "home-manager", "mediamanager-nix": "mediamanager-nix", + "nixos-anywhere": "nixos-anywhere", "nixos-images": "nixos-images", "nixpkgs": "nixpkgs_2", "nixpkgs-unstable": "nixpkgs-unstable", @@ -480,6 +518,27 @@ "url": "ssh://gitea@git.mgaction.town:2222/darman/TOME.git" } }, + "treefmt-nix": { + "inputs": { + "nixpkgs": [ + "nixos-anywhere", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1784369104, + "narHash": "sha256-47cxbcZODibHv3rELFQ9vZly0vUNkND/atn/U7HLeb0=", + "owner": "numtide", + "repo": "treefmt-nix", + "rev": "df3c0640565d04a0261253cdd89fce78ec50168a", + "type": "github" + }, + "original": { + "owner": "numtide", + "repo": "treefmt-nix", + "type": "github" + } + }, "uv2nix": { "inputs": { "nixpkgs": [ diff --git a/flake.nix b/flake.nix index 53a1392..65e7c0d 100644 --- a/flake.nix +++ b/flake.nix @@ -16,9 +16,24 @@ url = "github:Mic92/sops-nix"; inputs.nixpkgs.follows = "nixpkgs"; }; + # NB: no `inputs.nixpkgs.follows` here — nixos-images has no `nixpkgs` + # input (it takes nixos-stable / nixos-unstable), so declaring one only + # printed "override for a non-existent input" on every nix command. nixos-images = { url = "github:nix-community/nixos-images"; + inputs.nixos-stable.follows = "nixpkgs"; + }; + # Pinned as an input rather than `nix run github:...` from scripts/deploy. + # Both it and disko run at the exact moment a disk is being wiped, so the + # revision has to come from flake.lock — reviewed, reproducible, and + # resolvable from the local store — instead of whatever upstream master + # happens to be that day (which also fails outright with no network). + nixos-anywhere = { + url = "github:nix-community/nixos-anywhere"; inputs.nixpkgs.follows = "nixpkgs"; + inputs.nixos-stable.follows = "nixpkgs"; # 26.05 already IS stable + inputs.disko.follows = "disko"; + inputs.nixos-images.follows = "nixos-images"; }; home-manager = { url = "github:nix-community/home-manager/release-26.05"; @@ -48,13 +63,21 @@ }; }; - outputs = { self, nixpkgs, disko, sops-nix, nixos-images, home-manager, mediamanager-nix, authentik-nix, ... }@inputs: + outputs = { self, nixpkgs, disko, nixos-anywhere, sops-nix, nixos-images, home-manager, mediamanager-nix, authentik-nix, ... }@inputs: let system = "x86_64-linux"; in { - packages.${system}.tome = nixpkgs.legacyPackages.${system}.callPackage ./pkgs/tome.nix { - src = inputs.tome; + packages.${system} = { + tome = nixpkgs.legacyPackages.${system}.callPackage ./pkgs/tome.nix { + src = inputs.tome; + }; + + # Re-exported so `./scripts/deploy` can run them as `nix run .#disko` / + # `nix run .#nixos-anywhere`, at the revision flake.lock pins. See the + # nixos-anywhere input above for why that matters. + disko = disko.packages.${system}.disko; + nixos-anywhere = nixos-anywhere.packages.${system}.nixos-anywhere; }; nixosConfigurations = { @@ -185,6 +208,16 @@ networking.hostName = "homelab-installer"; environment.systemPackages = [ pkgs.git ]; + # installation-cd-minimal leaves experimental-features unset, so + # the ISO's nix.conf has no `nix-command`/`flakes` at all (unlike + # the nixos-images kexec installer, which sets + # extra-experimental-features itself — which is why the same + # `install localhost` branch works after kexec-local but + # not here). Without this, both `nix run .#disko` and + # `nixos-install --flake` die with "experimental Nix feature + # 'nix-command' is disabled". + nix.settings.experimental-features = [ "nix-command" "flakes" ]; + # Fresh clone of a PUBLIC repo — no credentials baked into the # ISO. require_tracked() in scripts/deploy still works fine here # (this IS a real git checkout, unlike the old baked-`self` @@ -219,17 +252,51 @@ after = [ "homelab-checkout.service" ]; requires = [ "homelab-checkout.service" ]; wantedBy = [ "multi-user.target" ]; - path = [ pkgs.gnugrep ]; + path = [ pkgs.gnugrep pkgs.util-linux pkgs.coreutils pkgs.git ]; serviceConfig.Type = "oneshot"; + # systemd does NOT set $HOME for a system service without User= + # (systemd.exec(5): SetLoginEnvironment= "defaults to true if + # User=, DynamicUser= or PAMName= are set, false otherwise"), and + # scripts/deploy runs under `set -u`. Without this the whole + # unattended run died on the bare $HOME expansion with an + # "unbound variable" that reads like a bug in the script. + serviceConfig.Environment = "HOME=/root"; script = '' cfg=$(grep -o 'homelab\.install=[^ ]*' /proc/cmdline | cut -d= -f2 || true) if [ -z "$cfg" ]; then echo "no homelab.install= on the kernel cmdline — nothing to auto-install" exit 0 fi + + # The host key scripts/deploy seeds /etc/ssh with (so sops can + # decrypt on boot #1) cannot live in this ISO: it is built from + # a PUBLIC repo and the private keys are deliberately off-repo. + # local_install_prepare_and_reboot() therefore drops it on the + # boot partition and passes that partition's PARTUUID here. + # That copy dies with the disko wipe a few minutes later. + keypart=$(grep -o 'homelab\.keypart=[^ ]*' /proc/cmdline | cut -d= -f2 || true) + if [ -n "$keypart" ]; then + mkdir -p /run/homelab-key + if mount -o ro "/dev/disk/by-partuuid/$keypart" /run/homelab-key; then + src=/run/homelab-key/homelab-installer + if [ -f "$src/ssh_host_ed25519_key" ]; then + echo "picking up $cfg's host key from PARTUUID=$keypart" + install -Dm600 "$src/ssh_host_ed25519_key" \ + "/root/.config/homelab/$cfg/ssh_host_ed25519_key" + install -Dm644 "$src/ssh_host_ed25519_key.pub" \ + "/root/.config/homelab/$cfg/ssh_host_ed25519_key.pub" + else + echo "warning: no host key at $src — the install will refuse" >&2 + fi + umount /run/homelab-key + else + echo "warning: could not mount PARTUUID=$keypart for the host key" >&2 + fi + fi + echo "auto-installing $cfg (homelab.install= on the kernel cmdline)" cd /root/homelab - exec ./scripts/deploy install "$cfg" localhost + exec ./scripts/deploy install "$cfg" localhost --yes ''; }; }) diff --git a/scripts/deploy b/scripts/deploy index f4959ff..45bfb14 100755 --- a/scripts/deploy +++ b/scripts/deploy @@ -17,7 +17,8 @@ # /var/tmp) must be exec-capable and hold # ~3x the tarball. # Then run `install localhost`. -# ./deploy install first install. Wipes the OS disk. Ships the +# ./deploy install [--yes] +# first install. Wipes the OS disk. Ships the # host's sops key. =localhost/127.0.0.1 # skips nixos-anywhere/ssh and runs disko + # nixos-install directly against /mnt — but @@ -27,16 +28,21 @@ # Run from the REAL running OS instead (e.g. # a box where kexec-local doesn't work), # it builds installer-iso, stages its -# kernel/initrd on the ESP + the iso file on -# a non-OS-disk partition, sets a systemd-boot -# one-shot entry with homelab.install= -# on its kernel cmdline, and reboots — a real -# ACPI reboot, not a kexec jump. The booted -# installer's homelab-auto-install.service -# reads that cmdline param and re-runs this -# exact command itself once its repo checkout -# (homelab-checkout.service) succeeds, finishing -# the install unattended. See CLAUDE.md. +# kernel/initrd + the host key on the boot +# partition and the iso file on a non-OS-disk +# partition, sets a systemd-boot one-shot +# entry with homelab.install= + +# homelab.keypart= on its kernel +# cmdline, and reboots — a real ACPI reboot, +# not a kexec jump. The booted installer's +# homelab-auto-install.service reads those +# cmdline params, picks the host key back up +# and re-runs this exact command itself once +# its repo checkout (homelab-checkout.service) +# succeeds, finishing the install unattended. +# It confirms before rebooting; --yes skips +# that (it is what the ISO passes itself). +# See CLAUDE.md. # ./deploy switch rebuild + activate on a running host. # ./deploy boot stage for next boot, don't activate now. # ./deploy test activate without adding a boot entry. @@ -75,6 +81,17 @@ REPO="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || dirname "$ cd "$REPO" export PATH="/nix/var/nix/profiles/default/bin:$PATH" +# Off-repo material keyed by : pre-generated SSH host keys (install) +# and per-config sops age keys (flash). +# +# Resolved defensively rather than as a bare $HOME, because this script also +# runs from installer-iso's homelab-auto-install.service, and systemd does not +# set $HOME for a system service without User= (systemd.exec(5): +# SetLoginEnvironment= "defaults to true if User=, DynamicUser= or PAMName= are +# set, false otherwise"). Under `set -u` that aborted the whole unattended run +# with an "unbound variable" that read like a bug in this script. +KEYDIR="${HOMELAB_KEY_DIR:-${HOME:-/root}/.config/homelab}" + die() { echo "error: $*" >&2; exit 1; } need() { command -v "$1" >/dev/null 2>&1 || die "missing required tool: $1"; } @@ -85,6 +102,11 @@ need() { command -v "$1" >/dev/null 2>&1 || die "missing required tool: $1"; } require_root() { [ "$(id -u)" = 0 ] && return 0 echo ">> $1 needs root — re-executing under sudo" >&2 + # $KEYDIR is derived from $HOME, and whether sudo carries $HOME across + # depends on the local sudoers policy (env_reset/always_set_home). Pin the + # resolved value so the re-exec looks for host keys where the invoking user + # has them, not under /root. + export HOMELAB_KEY_DIR="$KEYDIR" exec sudo -E -- "$SCRIPT_PATH" "${SCRIPT_ARGS[@]}" } @@ -95,9 +117,21 @@ one_match() { local what="$1"; shift local f=("$@") # caller expands the glob (nullglob is on) [ "${#f[@]}" -gt 0 ] || die "no $what found — did the build actually produce one?" + # Say so instead of silently taking [0]: a stale result-sd/ symlink from an + # earlier config is exactly how you flash the wrong image without a word. + [ "${#f[@]}" -eq 1 ] \ + || echo ">> warning: ${#f[@]} candidates for $what, using ${f[0]} (rm the stale ones)" >&2 printf '%s\n' "${f[0]}" } +# Every whole-disk device backing a block device or a mounted path, one per +# line. LVM/RAID/LUKS can sit on several at once (verified on terra: +# /mnt/ssd_01 -> sdd AND sde), so a single lookup is not enough. Empty output +# means "could not determine" — which callers must treat as unsafe, not as OK. +disks_backing() { + lsblk -rnso NAME,TYPE "$1" 2>/dev/null | awk '$2 == "disk" { print "/dev/" $1 }' +} + # Sets tb / cpio / bbox — the kexec tarball plus the static cpio+gzip that # kexec-run.sh needs on PATH to rebuild its initrd. # @@ -137,23 +171,38 @@ is_live_installer() { } # `install localhost` run on a REAL running OS (not already inside a -# live installer): builds installer-iso, stages its kernel/initrd + iso file -# locally, points a systemd-boot one-shot entry at them with -# homelab.install= on the kernel cmdline, and reboots — a real ACPI -# reboot through firmware POST, deliberately NOT a kexec jump (see terra's -# kexec-local gotcha in CLAUDE.md). The booted installer's -# homelab-auto-install.service reads that cmdline param and re-runs this exact -# `install localhost` command itself (now genuinely inside the -# installer) once homelab-checkout.service has fetched the repo, finishing the -# job unattended. +# live installer): builds installer-iso, stages its kernel/initrd + the host's +# pre-generated ssh key on the boot partition and the iso file on a non-OS +# disk, points a systemd-boot one-shot entry at them with +# homelab.install= + homelab.keypart= on the kernel cmdline, +# and reboots — a real ACPI reboot through firmware POST, deliberately NOT a +# kexec jump (see terra's kexec-local gotcha in CLAUDE.md). The booted +# installer's homelab-auto-install.service reads those params, picks the host +# key back up and re-runs this exact `install localhost` command +# itself (now genuinely inside the installer) once homelab-checkout.service has +# fetched the repo, finishing the job unattended. local_install_prepare_and_reboot() { - local config="$1" + local config="$1" hostkey="$2" assume_yes="$3" require_root "preparing a local reinstall" [ -d /sys/firmware/efi ] || die "not booted UEFI — the one-shot boot entry needs systemd-boot" need bootctl need nix need lsblk need findmnt + need awk + need realpath + need stat + need df + + # systemd-boot keeps its entries on $BOOT — the XBOOTLDR partition when there + # is one, the ESP otherwise — which is not always /boot. Hardcoding /boot on + # a box that mounts its ESP elsewhere just creates a directory on the root + # filesystem, and then reboots into an entry the firmware never sees. + local boot + boot="$(bootctl --print-boot-path 2>/dev/null)" \ + || die "bootctl couldn't locate the boot partition — is systemd-boot installed here?" + [ -d "$boot/loader/entries" ] \ + || die "$boot/loader/entries doesn't exist — systemd-boot isn't installed on this box" # No default/auto-picked location — the wrong disk here is destroyed # mid-install (see the OS-disk check below), so this always asks rather @@ -168,48 +217,124 @@ local_install_prepare_and_reboot() { [ -n "$stagedir" ] || die "no staging path given" [ -d "$stagedir" ] \ || die "staging dir $stagedir doesn't exist — needs to be an existing partition that is NOT the OS disk being wiped" + # Absolute + symlink-free: findiso= below is computed by stripping the + # mountpoint prefix off this, and a relative answer at the prompt would + # produce a path the initrd can never resolve. + stagedir="$(realpath "$stagedir")" # Refuse if the staging partition turns out to live on the same disk # disko is about to wipe — the iso file (and the running installer # loopback-mounted from it) would be destroyed mid-install. - local osdisk osdisk_real stage_src stage_pkname stage_disk_real + local osdisk osdisk_real stage_src stage_fstype stage_disks d osdisk="$(nix eval --raw ".#nixosConfigurations.$config.config.disko.devices.disk" \ --apply 'd: (builtins.head (builtins.attrValues d)).device' 2>/dev/null)" \ || die "couldn't read the OS disk device from hosts/$config/disk-config.nix" osdisk_real="$(readlink -f "$osdisk")" - stage_src="$(findmnt -no SOURCE --target "$stagedir")" \ + + # --nofsroot matters: on btrfs, findmnt prints the subvolume as + # `/dev/sdb2[/@]`, which is not a path lsblk can open. Without it the lookup + # came back empty and the guard below was skipped entirely — i.e. it silently + # allowed staging on the very disk about to be wiped. terra's current + # CachyOS root is exactly that layout. + stage_src="$(findmnt -no SOURCE --nofsroot --target "$stagedir")" \ || die "$stagedir doesn't resolve to a mounted filesystem" - stage_pkname="$(lsblk -no PKNAME "$stage_src" 2>/dev/null || true)" - if [ -n "$stage_pkname" ]; then - stage_disk_real="$(readlink -f "/dev/$stage_pkname")" - [ "$stage_disk_real" = "$osdisk_real" ] \ - && die "$stagedir is on the OS disk ($osdisk) that install would wipe — re-run and pick a different disk" + # `|| true` so the explicit check below is what reports the problem: lsblk + # exits nonzero on a device it can't parse, and under `set -e` + pipefail a + # bare assignment from a failing substitution kills the script silently, + # right past the fail-closed message. + stage_disks="$(disks_backing "$stage_src" || true)" + # Fail closed. "Couldn't determine the disk" is not "different disk". + [ -n "$stage_disks" ] \ + || die "couldn't determine which physical disk $stagedir ($stage_src) is on — refusing to guess, since being wrong destroys the install mid-flight" + for d in $stage_disks; do + if [ "$d" = "$osdisk_real" ]; then + die "$stagedir is on the OS disk ($osdisk -> $osdisk_real) that install would wipe — re-run and pick a different disk" + fi + done + + # stage-1 resolves findiso= by mounting each blkid-visible partition and + # testing `-e /findiso$isoPath` (nixos/modules/system/boot/stage-1-init.sh). + # For btrfs it mounts the volume's TOP level, so a path that lives inside a + # subvolume (/@/...) is simply not there and the box boots to an emergency + # shell — after it has already rebooted out of the working OS. + stage_fstype="$(findmnt -no FSTYPE --target "$stagedir")" + [ "$stage_fstype" != btrfs ] \ + || die "$stagedir is btrfs: findiso= mounts the volume's top level, so a path inside a subvolume never resolves. Stage on a non-btrfs partition (ext4/vfat/ntfs)." + + # Last chance to back out. This is the most destructive command in the + # script — it reboots the machine you are typing at and the wipe that + # follows is unattended — so it confirms just like `flash` and `kexec-local` + # do, both of which are less final than this. + if [ "$assume_yes" != "--yes" ]; then + echo ">> about to REINSTALL this machine from scratch:" + echo " hostname: $(uname -n)" + echo " config: $config" + echo " OS disk: $osdisk" + echo " -> $osdisk_real ** WIPED, unattended, after the reboot **" + # Unquoted on purpose: collapses the one-per-line list onto one line. + echo " staging: $stagedir (on $(echo $stage_disks))" + echo " entry: $boot/loader/entries/homelab-installer.conf (one-shot)" + read -rp ">> type 'yes' to build the installer, reboot into it and wipe $osdisk_real: " ok + [ "$ok" = yes ] || die "aborted" fi echo ">> building installer-iso (kernel + initrd + iso image)" - local kernel initrd isodir iso mnt_point iso_relpath + local kernel initrd isodir iso mnt_point iso_relpath boot_src boot_partuuid kernel="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.kernel)/bzImage" initrd="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.initialRamdisk)/initrd" isodir="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.isoImage)" iso="$(one_match 'installer iso' "$isodir"/iso/*.iso)" - echo ">> staging kernel/initrd on the ESP, iso image on $stagedir" - install -Dm644 "$kernel" /boot/homelab-installer/bzImage - install -Dm644 "$initrd" /boot/homelab-installer/initrd + # A short write is not visible until the reboot, when findiso finds a + # truncated iso and drops to an emergency shell. Check first — `install` + # prints no progress and the iso is ~1GB. + local need_stage need_boot avail_stage avail_boot + need_stage="$(stat -Lc %s "$iso")" + need_boot="$(( $(stat -Lc %s "$kernel") + $(stat -Lc %s "$initrd") + $(stat -Lc %s "$hostkey") ))" + avail_stage="$(df -B1 --output=avail "$stagedir" | tail -1 | tr -d ' ')" + avail_boot="$(df -B1 --output=avail "$boot" | tail -1 | tr -d ' ')" + [ "$avail_stage" -ge "$(( need_stage + 64 * 1024 * 1024 ))" ] \ + || die "$stagedir has $(( avail_stage / 1024 / 1024 ))MB free, the iso needs $(( need_stage / 1024 / 1024 ))MB — pick another partition" + [ "$avail_boot" -ge "$(( need_boot + 16 * 1024 * 1024 ))" ] \ + || die "$boot has $(( avail_boot / 1024 / 1024 ))MB free, kernel+initrd need $(( need_boot / 1024 / 1024 ))MB" + + echo ">> staging kernel/initrd/host key on $boot, iso image on $stagedir" + install -Dm644 "$kernel" "$boot/homelab-installer/bzImage" + install -Dm644 "$initrd" "$boot/homelab-installer/initrd" install -Dm644 "$iso" "$stagedir/homelab-installer.iso" - # findiso= is a path relative to whatever partition the initrd finds it on - # (it mounts every blkid-visible partition looking for it) — not necessarily - # relative to `/`, if $stagedir is a subdirectory of a bigger filesystem - # rather than a mountpoint itself. - mnt_point="$(findmnt -no TARGET --target "$stagedir")" - iso_relpath="${stagedir#"$mnt_point"}/homelab-installer.iso" + # The ISO is built from a PUBLIC repo and deliberately carries no + # credentials, so the host key has to travel with the staged installer or + # the auto-install run has nothing to seed /etc/ssh with — and without that, + # sops can't decrypt on boot #1, /etc/shadow gets written once with a locked + # darman, and no later `deploy switch` can fix it (README). + # + # $boot lives on the OS disk, so disko destroys this copy minutes later. The + # mode is advisory on vfat (permissions come from the mount's fmask, 0077 on + # a NixOS/systemd-boot ESP) — it is the wipe, not the mode, doing the work. + install -Dm600 "$hostkey" "$boot/homelab-installer/ssh_host_ed25519_key" + install -Dm644 "$hostkey.pub" "$boot/homelab-installer/ssh_host_ed25519_key.pub" + boot_src="$(findmnt -no SOURCE --nofsroot --target "$boot")" \ + || die "couldn't resolve $boot to a device" + boot_partuuid="$(lsblk -no PARTUUID "$boot_src" 2>/dev/null | head -1 | tr -d ' ' || true)" + [ -n "$boot_partuuid" ] \ + || die "couldn't read a PARTUUID for $boot ($boot_src) — the installer needs it to find the host key" - cat >/boot/loader/entries/homelab-installer.conf <"$boot/loader/entries/homelab-installer.conf" <> one-shot boot into the installer, then rebooting — it will finish this install itself" @@ -221,14 +346,20 @@ EOF # invisible to `nix build`/`nixos-install`, which then fails obscurely or builds # a stale config. Check before doing anything destructive. require_tracked() { - local config="$1" cfgfile="hosts/$1/configuration.nix" + local config="$1" cfgfile="hosts/$1/configuration.nix" f [ -e "$cfgfile" ] || die "no $cfgfile in the repo" - # No .git at all (e.g. a tarball export of the repo, no working tree) means - # there's nothing that CAN be untracked — nothing to check. Only skip on a - # MISSING .git, not on any other git failure. + # No .git at all (e.g. a tarball export of the repo, no working tree), or no + # git binary, means there's nothing that CAN be untracked — nothing to check. + # Only skip on that, not on any other git failure. + command -v git >/dev/null 2>&1 || return 0 git -C "$REPO" rev-parse --is-inside-work-tree >/dev/null 2>&1 || return 0 - git -C "$REPO" ls-files --error-unmatch "$cfgfile" >/dev/null 2>&1 \ - || die "$cfgfile is untracked — 'git add hosts/$config' first (flakes ignore untracked files)" + # Every .nix in hosts//, not just configuration.nix: an untracked + # disk-config.nix is exactly as invisible to the flake, and it is the file + # that decides which disk gets wiped. + for f in "hosts/$config"/*.nix; do + git -C "$REPO" ls-files --error-unmatch "$f" >/dev/null 2>&1 \ + || die "$f is untracked — 'git add hosts/$config' first (flakes ignore untracked files)" + done } # The password field of a Proton Pass item ("--field password" prints the bare @@ -469,9 +600,12 @@ case "$cmd" in ;; install) - config="${2:-}"; host="${3:-}" - { [ -n "$config" ] && [ -n "$host" ]; } || die "usage: ./deploy install " - hostkey="$HOME/.config/homelab/$config/ssh_host_ed25519_key" + config="${2:-}"; host="${3:-}"; assume_yes="${4:-}" + { [ -n "$config" ] && [ -n "$host" ]; } || die "usage: ./deploy install [--yes]" + # $KEYDIR, not a bare $HOME — see its definition. This same check runs + # inside installer-iso, where homelab-auto-install.service has no $HOME and + # has just dropped the key into /root/.config/homelab//. + hostkey="$KEYDIR/$config/ssh_host_ed25519_key" [ -f "$hostkey" ] || die "missing host key: $hostkey" [ -d "./hosts/$config" ] || die "no ./hosts/$config directory in the repo" require_tracked "$config" @@ -481,7 +615,7 @@ case "$cmd" in # Not already inside a live installer: build one, stage it, one-shot # boot into it, and let it finish this exact command itself. See # local_install_prepare_and_reboot above and CLAUDE.md. - local_install_prepare_and_reboot "$config" + local_install_prepare_and_reboot "$config" "$hostkey" "$assume_yes" exit 0 fi @@ -492,7 +626,10 @@ case "$cmd" in [ -f "./hosts/$config/disk-config.nix" ] || die "no ./hosts/$config/disk-config.nix" echo ">> disko .#$config onto this box's OS disk (WILL be wiped)" - nix run github:nix-community/disko -- \ + # `.#disko`, not github:nix-community/disko — the revision comes from this + # repo's flake.lock rather than upstream master-of-the-day, and resolves + # from the local store. See the nixos-anywhere input in flake.nix. + nix run ".#disko" -- \ --mode disko "./hosts/$config/disk-config.nix" echo ">> installing sops host key so it can decrypt on boot #1" @@ -524,11 +661,11 @@ case "$cmd" in echo ">> root ssh password from Proton Pass ($root_item)" export SSHPASS="$root_pw" unset root_pw - nix run github:nix-community/nixos-anywhere -- \ + nix run ".#nixos-anywhere" -- \ --env-password "${anywhere[@]}" unset SSHPASS else - nix run github:nix-community/nixos-anywhere -- "${anywhere[@]}" + nix run ".#nixos-anywhere" -- "${anywhere[@]}" fi fi ;; @@ -600,7 +737,7 @@ case "$cmd" in # partition at /var/lib/sops-nix/age.txt so sops decrypts on first boot. # (The Pi's vfat partition isn't mounted at runtime, so the key can't live # there.) Key stays off-repo, out of the nix store, and out of the image. - keyfile="$HOME/.config/homelab/$config/age.txt" + keyfile="$KEYDIR/$config/age.txt" if [ -f "$keyfile" ]; then echo ">> installing sops age key onto the root partition" sudo partprobe "$dev" 2>/dev/null || sudo blockdev --rereadpt "$dev" 2>/dev/null || true @@ -613,10 +750,13 @@ case "$cmd" in | sort -rn | head -1 | cut -d' ' -f2)" [ -n "$rootpart" ] || die "no ext4 root partition found on $dev — place $keyfile at /var/lib/sops-nix/age.txt manually" mnt="$(mktemp -d)" + # Unmount + remove even if the install fails, so a retry doesn't trip + # over the card still being mounted on a stale temp dir. + trap 'sudo umount "$mnt" 2>/dev/null || true; rmdir "$mnt" 2>/dev/null || true' EXIT sudo mount "$rootpart" "$mnt" sudo install -Dm600 "$keyfile" "$mnt/var/lib/sops-nix/age.txt" sudo sync - sudo umount "$mnt"; rmdir "$mnt" + sudo umount "$mnt"; rmdir "$mnt"; trap - EXIT echo ">> age key installed (/var/lib/sops-nix/age.txt)" fi echo ">> done — insert the card into the Pi and boot."