darman/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: condense comments across the repo

Browse Source 
Comments had drifted into multi-paragraph narrative (git commit
lineage, debugging stories, restated code) in several hot spots
(scripts/deploy, hermes-agent.nix, flake.nix, gitea.nix, headscale.nix).
Trim every comment to its load-bearing "why" — gotchas, safety
warnings, and non-obvious rationale survive verbatim in substance,
just tightened to 1-2 sentences; historical narrative and anything
already covered in CLAUDE.md is cut. No code/logic changed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UJqEmY1y3AYX3JoX4Y6b21
This commit is contained in:
darman
2026-09-18 21:36:30 +02:00
co-authored by Claude Sonnet 5
parent 3899290c5b
commit 6f24ab69ad
47 changed files with 1051 additions and 1965 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/deploy
+126 -281
View File
@@ -1,106 +1,48 @@
#!/usr/bin/env bash
# Deploy a NixOS host from this flake. ALL arguments are mandatory (no defaults).
#
# ./deploy kexec <config> <host> headless kexec into a RAM installer, for a
# read-only-root box (ZimaOS) where
# nixos-anywhere can't ssh-copy-id. Ships our
# SSH login key. Then run `install`. <config>
# is only used to look up the vault item.
# ./deploy kexec-local [--yes] kexec THIS machine into the RAM installer,
# no ssh/second machine involved. Run as root,
# locally, on the box you're installing onto.
# Disks are untouched; console drops for
# ~1-2 min then comes back as the installer.
# Prompts for confirmation (--yes skips it),
# because run on the wrong terminal this
# kexecs your laptop. TMPDIR (default
# /var/tmp) must be exec-capable and hold
# ~3x the tarball.
# Then run `install <config> localhost`.
# ./deploy kexec <config> <host> ZimaOS/RO-root box: kexec into a RAM installer, ships the ssh key, then run `install`.
# ./deploy kexec-local [--yes] kexec THIS machine (no ssh) into the RAM installer; disks untouched. Then `install <config> localhost`.
# ./deploy install <config> <host> [--yes]
# first install. Wipes the OS disk. Ships the
# host's sops key. <host>=localhost/127.0.0.1
# skips nixos-anywhere/ssh and runs disko +
# nixos-install directly against /mnt — but
# ONLY once actually inside a live installer
# (hostname nixos-installer, from kexec, or
# homelab-installer, from installer-iso).
# Run from the REAL running OS instead (e.g.
# a box where kexec-local doesn't work),
# it builds installer-iso, stages its
# kernel/initrd + the host key on the boot
# partition and the iso file on a non-OS-disk
# partition, sets a systemd-boot one-shot
# entry with homelab.install=<config> +
# homelab.keypart=<PARTUUID> on its kernel
# cmdline, and reboots — a real ACPI reboot,
# not a kexec jump. The booted installer's
# homelab-auto-install.service reads those
# cmdline params, picks the host key back up
# and re-runs this exact command itself once
# its repo checkout (homelab-checkout.service)
# succeeds, finishing the install unattended.
# It confirms before rebooting; --yes skips
# that (it is what the ISO passes itself).
# first install (wipes the OS disk, ships the host's sops key). localhost only
# runs disko/nixos-install directly once already inside a live installer;
# from a real running OS it stages installer-iso and reboots into that instead.
# See CLAUDE.md.
# ./deploy switch <config> <host> rebuild + activate on a running host.
# ./deploy boot <config> <host> stage for next boot, don't activate now.
# ./deploy test <config> <host> activate without adding a boot entry.
# ./deploy image <config> build an SD-card image (e.g. rpi mercury).
# ./deploy flash <config> <dev> build SD image, write to <dev>, and (if
# ~/.config/homelab/<config>/age.txt exists)
# drop the sops key on its boot partition.
# ./deploy flash <config> <dev> build SD image, write to <dev>, and drop the sops age key onto it if one exists.
#
# <config> = a nixosConfigurations name (e.g. jupiter, vps). Its pre-generated
# SSH host key must be at ~/.config/homelab/<config>/ssh_host_ed25519_key.
# <config> = a nixosConfigurations name. Its pre-generated SSH host key must be
# at ~/.config/homelab/<config>/ssh_host_ed25519_key. Runs from a non-NixOS host too.
#
# Runs from a non-NixOS host too (nixos-rebuild / nixos-anywhere via `nix run`).
#
# Password prompts are auto-filled from the "HomeLab" Proton Pass vault when
# `pass-cli` is installed and logged in; otherwise every command prompts exactly
# as before. Both items are keyed by <config>, never by <host>: the address is
# incidental (DHCP, a new box, localhost) while the config name is the stable
# identity of the machine being built.
# darman@<config> darman's sudo password (switch/boot/test)
# root@<config> root's ssh password (kexec/install)
# Password prompts auto-fill from the "HomeLab" Proton Pass vault, keyed by
# <config> not <host> (darman@<config> for sudo, root@<config> for ssh).
# Override with HOMELAB_PASS_ITEM / HOMELAB_PASS_ROOT_ITEM / HOMELAB_PASS_VAULT.
set -euo pipefail
shopt -s nullglob
# Captured before anything shifts/parses $@, so require_root() below can
# re-exec the ORIGINAL invocation under sudo — inside a function, "$@"/"$1"
# refer to the function's own args (empty here), not the script's, so this
# has to be a global array instead of relying on positional-parameter scoping.
# Captured before $@ is parsed, so require_root() can re-exec the ORIGINAL
# invocation under sudo (inside a function, "$@" is the function's own args).
SCRIPT_ARGS=("$@")
# Locate the repo root (flake dir) regardless of where this script lives on disk.
SCRIPT_PATH="$(realpath "$0")" # absolute — "$0" itself may be relative,
# and require_root() re-execs after cd "$REPO"
SCRIPT_PATH="$(realpath "$0")"
SCRIPT_DIR="$(dirname "$SCRIPT_PATH")"
REPO="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || dirname "$SCRIPT_DIR")"
cd "$REPO"
export PATH="/nix/var/nix/profiles/default/bin:$PATH"
# Every `nix` call below assumes `nix-command` + `flakes`. Those are ambient on
# a Determinate-Nix laptop, but a STOCK NixOS box leaves both experimental
# features OFF — so bare `nix eval`/`build`/`run` die with "experimental Nix
# feature 'nix-command' is disabled". That box is exactly the prepare host for
# `install <config> localhost` (a fresh NixOS the reinstall runs from), and it
# is why the installer-iso already sets these itself (flake.nix). Enable them
# additively via NIX_CONFIG (extra-, so anything already configured is kept).
# This runs again at the top of the sudo re-exec in require_root(), so root
# gets it too regardless of whether `sudo -E` carries the env across.
# A stock NixOS box (unlike a Determinate-Nix laptop) leaves nix-command/flakes
# disabled, and that's exactly the prepare host for `install <config> localhost`.
# Enable them additively so root gets them too after the require_root() re-exec.
export NIX_CONFIG="$(printf 'extra-experimental-features = nix-command flakes\n%s' "${NIX_CONFIG:-}")"
# Off-repo material keyed by <config>: pre-generated SSH host keys (install)
# and per-config sops age keys (flash).
#
# Resolved defensively rather than as a bare $HOME, because this script also
# runs from installer-iso's homelab-auto-install.service, and systemd does not
# set $HOME for a system service without User= (systemd.exec(5):
# SetLoginEnvironment= "defaults to true if User=, DynamicUser= or PAMName= are
# set, false otherwise"). Under `set -u` that aborted the whole unattended run
# with an "unbound variable" that read like a bug in this script.
# and per-config sops age keys (flash). Resolved defensively, not as a bare
# $HOME, since systemd doesn't set $HOME for a service without User= — this
# also runs unattended from installer-iso's homelab-auto-install.service.
KEYDIR="${HOMELAB_KEY_DIR:-${HOME:-/root}/.config/homelab}"
die() { echo "error: $*" >&2; exit 1; }
@@ -108,48 +50,40 @@ die() { echo "error: $*" >&2; exit 1; }
need() { command -v "$1" >/dev/null 2>&1 || die "missing required tool: $1"; }
# Self-elevate instead of dying: re-exec this exact invocation under sudo.
# -E preserves the environment (HOMELAB_* overrides, Proton Pass vault vars)
# across the re-exec. A no-op once already root.
# -E preserves HOMELAB_*/vault env vars; pin HOMELAB_KEY_DIR too since whether
# sudo carries $HOME across depends on the local sudoers policy. No-op if already root.
require_root() {
[ "$(id -u)" = 0 ] && return 0
echo ">> $1 needs root — re-executing under sudo" >&2
# $KEYDIR is derived from $HOME, and whether sudo carries $HOME across
# depends on the local sudoers policy (env_reset/always_set_home). Pin the
# resolved value so the re-exec looks for host keys where the invoking user
# has them, not under /root.
export HOMELAB_KEY_DIR="$KEYDIR"
exec sudo -E -- "$SCRIPT_PATH" "${SCRIPT_ARGS[@]}"
}
# Exactly one path matching a glob, or die. `ls glob | head -1` silently yields
# an empty string when nothing matches (head exits 0, so set -e never fires) and
# the failure only surfaces later as a confusing tar/dd error.
# an empty string when nothing matches (head exits 0, so set -e never fires).
one_match() {
local what="$1"; shift
local f=("$@") # caller expands the glob (nullglob is on)
[ "${#f[@]}" -gt 0 ] || die "no $what found — did the build actually produce one?"
# Say so instead of silently taking [0]: a stale result-sd/ symlink from an
# earlier config is exactly how you flash the wrong image without a word.
# A stale result-sd/ symlink from an earlier config is how you'd otherwise
# flash the wrong image without a word — warn instead of silently taking [0].
[ "${#f[@]}" -eq 1 ] \
|| echo ">> warning: ${#f[@]} candidates for $what, using ${f[0]} (rm the stale ones)" >&2
printf '%s\n' "${f[0]}"
}
# Every whole-disk device backing a block device or a mounted path, one per
# line. LVM/RAID/LUKS can sit on several at once (verified on terra:
# /mnt/ssd_01 -> sdd AND sde), so a single lookup is not enough. Empty output
# means "could not determine" — which callers must treat as unsafe, not as OK.
# Every whole-disk device backing a block device or mounted path, one per line.
# LVM/RAID/LUKS can span several disks at once (e.g. terra's /mnt/ssd_01),
# so callers must treat empty output as "unknown", not "safe".
disks_backing() {
lsblk -rnso NAME,TYPE "$1" 2>/dev/null | awk '$2 == "disk" { print "/dev/" $1 }'
}
# Label of the temporary UEFI boot entry arm_efi_bootnext() creates. Also the
# key the ISO uses to delete it again once it has booted (see flake.nix).
# Label of the temporary UEFI boot entry arm_efi_bootnext() creates; also what
# the booted ISO matches to delete it again (flake.nix) — must match EXACTLY.
EFI_LABEL="Homelab Installer"
# Boot numbers of every UEFI entry with exactly this label, one per line.
# efibootmgr prints `Boot0002* Limine<TAB>HD(1,GPT,...)/\EFI\...`, so the
# label runs from past the "Boot####* " prefix up to the first TAB.
# (Character classes spelled out rather than {4}: mawk predates ERE intervals.)
efi_entries_named() {
efibootmgr 2>/dev/null | awk -v want="$1" '
@@ -162,17 +96,11 @@ efi_entries_named() {
}'
}
# Arm a genuine one-shot boot of the staged installer WITHOUT any help from the
# bootloader: create a UEFI boot entry that EFI-stub-boots the kernel straight
# off the ESP, and point BootNext at it.
#
# Needed because "boot this once, then go back to normal" is not something
# every bootloader can do. systemd-boot has it; terra's CachyOS runs Limine,
# which reports `One-shot entry control: ✗` and has no equivalent, and whose
# limine.conf is regenerated by pacman hooks anyway. BootNext is a firmware
# feature, so it works underneath all of them — and the firmware clears it
# after that one boot, which is what keeps the "a failed attempt still comes
# back on the normal bootloader" property that makes this safe to try.
# Arm a genuine one-shot boot of the staged installer without bootloader help:
# create a UEFI entry that EFI-stub-boots the kernel off the ESP and point
# BootNext at it. Needed because Limine (terra's CachyOS) has no one-shot
# entry support; BootNext is a firmware feature so it works underneath any
# bootloader, and the firmware clears it after one boot either way.
arm_efi_bootnext() {
local esp="$1" cmdline="$2"
local esp_src esp_disk esp_part num n
@@ -184,20 +112,17 @@ arm_efi_bootnext() {
{ [ -n "$esp_disk" ] && [ -n "$esp_part" ]; } \
|| die "couldn't work out the disk + partition number of the ESP ($esp -> $esp_src)"
# Clear anything left by an earlier attempt first, so repeated runs don't
# slowly fill NVRAM with dead entries pointing at a wiped partition.
# Clear anything left by an earlier attempt so NVRAM doesn't slowly fill
# with dead entries pointing at a wiped partition.
for n in $(efi_entries_named "$EFI_LABEL"); do
echo ">> removing stale UEFI entry Boot$n ($EFI_LABEL)"
efibootmgr -q -B -b "$n"
done
# --create-only, NOT --create: the latter also pushes the entry to the front
# of BootOrder, which would make a wiped installer the permanent default if
# anything went wrong. This way the entry is reachable through BootNext and
# nothing else, i.e. exactly once.
#
# The EFI stub loads `initrd=` off the volume it was itself loaded from, so
# the path is relative to the ESP root and uses backslashes.
# --create-only, NOT --create: --create also pushes the entry to the front of
# BootOrder, which would make a wiped installer the permanent default on any
# failure. This way it's reachable only via BootNext, exactly once. The EFI
# stub loads `initrd=` relative to the ESP root, hence the backslash path.
efibootmgr -q --create-only --disk "$esp_disk" --part "$esp_part" \
--label "$EFI_LABEL" \
--loader '\homelab-installer\bzImage' \
@@ -210,11 +135,8 @@ arm_efi_bootnext() {
}
# Sets tb / cpio / bbox — the kexec tarball plus the static cpio+gzip that
# kexec-run.sh needs on PATH to rebuild its initrd.
#
# HOMELAB_KEXEC_TARBALL (with _CPIO / _GZIP) short-circuits the build and uses a
# prebuilt installer instead. That lets the VM test in flake.nix drive this
# script offline, and lets you re-kexec a box without rebuilding ~500MB.
# kexec-run.sh needs on PATH to rebuild its initrd. HOMELAB_KEXEC_TARBALL (+
# _CPIO/_GZIP) short-circuits the build to reuse a prebuilt installer instead.
kexec_artifacts() {
if [ -n "${HOMELAB_KEXEC_TARBALL:-}" ]; then
tb="$HOMELAB_KEXEC_TARBALL"
@@ -235,11 +157,10 @@ kexec_artifacts() {
fi
}
# True inside one of the throwaway live-installer environments this repo
# produces (kexec's nixos-installer, or installer-iso's homelab-installer) —
# i.e. `install <config> localhost` should wipe/install right here. False on
# any real running OS, where the same command instead means "prepare and
# reboot into an installer for THIS box" (see local_install_prepare_and_reboot).
# True inside one of this repo's throwaway live-installer environments
# (nixos-installer from kexec, or homelab-installer from installer-iso) —
# i.e. `install <config> localhost` should wipe/install right here, not
# prepare-and-reboot (see local_install_prepare_and_reboot).
is_live_installer() {
case "$(uname -n)" in
nixos-installer | homelab-installer) return 0 ;;
@@ -247,17 +168,12 @@ is_live_installer() {
esac
}
# `install <config> localhost` run on a REAL running OS (not already inside a
# live installer): builds installer-iso, stages its kernel/initrd + the host's
# pre-generated ssh key on the boot partition and the iso file on a non-OS
# disk, points a systemd-boot one-shot entry at them with
# homelab.install=<config> + homelab.keypart=<PARTUUID> on the kernel cmdline,
# and reboots — a real ACPI reboot through firmware POST, deliberately NOT a
# kexec jump (see terra's kexec-local gotcha in CLAUDE.md). The booted
# installer's homelab-auto-install.service reads those params, picks the host
# key back up and re-runs this exact `install <config> localhost` command
# itself (now genuinely inside the installer) once homelab-checkout.service has
# fetched the repo, finishing the job unattended.
# `install <config> localhost` on a REAL running OS (not yet inside a live
# installer): stages installer-iso's kernel/initrd + host key on the boot
# partition, arms a one-shot boot with homelab.install=<config> on its
# cmdline, and does a real ACPI reboot — deliberately not a kexec jump, per
# terra's kexec-local gotcha in CLAUDE.md. The booted installer re-runs this
# same command itself once its repo checkout succeeds, finishing unattended.
local_install_prepare_and_reboot() {
local config="$1" hostkey="$2" assume_yes="$3"
require_root "preparing a local reinstall"
@@ -271,17 +187,12 @@ local_install_prepare_and_reboot() {
need stat
need df
# Where to stage the installer, and how to make the box boot it exactly once.
#
# systemd-boot keeps its entries on $BOOT — the XBOOTLDR partition when there
# is one, the ESP otherwise — which is not always /boot. Hardcoding /boot on
# a box that mounts its ESP elsewhere just creates a directory on the root
# filesystem and then reboots into an entry the firmware never sees.
#
# No systemd-boot (terra's CachyOS runs Limine) means no `bootctl set-oneshot`,
# so fall back to the firmware's own BootNext — see arm_efi_bootnext(). That
# path EFI-stub-boots the kernel directly, which requires it to sit on the ESP
# itself rather than on a separate XBOOTLDR.
# Where to stage the installer: use bootctl's reported $BOOT (XBOOTLDR or the
# ESP), not a hardcoded /boot, since that's not always where the ESP mounts.
# No systemd-boot (terra's CachyOS runs Limine) means no `bootctl
# set-oneshot`, so fall back to firmware BootNext (arm_efi_bootnext()) —
# which EFI-stub-boots the kernel directly and needs it on the ESP itself,
# not a separate XBOOTLDR.
local boot boot_mode esp
esp="$(bootctl --print-esp-path 2>/dev/null)" \
|| die "bootctl couldn't locate the ESP — is this box actually UEFI-booted?"
@@ -296,10 +207,9 @@ local_install_prepare_and_reboot() {
echo " own BootNext instead (bootloader in charge here: $(bootctl status 2>/dev/null | awk '/Product:/ {$1=""; print substr($0,2); exit}' || echo unknown))"
fi
# No default/auto-picked location — the wrong disk here is destroyed
# mid-install (see the OS-disk check below), so this always asks rather
# than guessing. HOMELAB_INSTALLER_STAGE_DIR skips the prompt for scripted
# use, but is otherwise just as explicit a choice as typing it in.
# No default/auto-picked location: the wrong disk here is destroyed
# mid-install (see the OS-disk check below), so this always asks unless
# HOMELAB_INSTALLER_STAGE_DIR is set for scripted use.
local stagedir="${HOMELAB_INSTALLER_STAGE_DIR:-}"
if [ -z "$stagedir" ]; then
echo ">> currently mounted filesystems:"
@@ -323,17 +233,13 @@ local_install_prepare_and_reboot() {
|| die "couldn't read the OS disk device from hosts/$config/disk-config.nix"
osdisk_real="$(readlink -f "$osdisk")"
# --nofsroot matters: on btrfs, findmnt prints the subvolume as
# `/dev/sdb2[/@]`, which is not a path lsblk can open. Without it the lookup
# came back empty and the guard below was skipped entirely — i.e. it silently
# allowed staging on the very disk about to be wiped. terra's current
# CachyOS root is exactly that layout.
# --nofsroot matters: on btrfs findmnt prints the subvolume as
# `/dev/sdb2[/@]`, which lsblk can't open, silently skipping the guard
# below and allowing staging on the disk about to be wiped (terra's layout).
stage_src="$(findmnt -no SOURCE --nofsroot --target "$stagedir")" \
|| die "$stagedir doesn't resolve to a mounted filesystem"
# `|| true` so the explicit check below is what reports the problem: lsblk
# exits nonzero on a device it can't parse, and under `set -e` + pipefail a
# bare assignment from a failing substitution kills the script silently,
# right past the fail-closed message.
# `|| true` so the fail-closed check below reports the problem, rather than
# `set -e`/pipefail silently killing the script on lsblk's nonzero exit.
stage_disks="$(disks_backing "$stage_src" || true)"
# Fail closed. "Couldn't determine the disk" is not "different disk".
[ -n "$stage_disks" ] \
@@ -344,30 +250,23 @@ local_install_prepare_and_reboot() {
fi
done
# stage-1 resolves findiso= by mounting each blkid-visible partition and
# testing `-e /findiso$isoPath` (nixos/modules/system/boot/stage-1-init.sh).
# For btrfs it mounts the volume's TOP level, so a path that lives inside a
# subvolume (/@/...) is simply not there and the box boots to an emergency
# shell — after it has already rebooted out of the working OS.
# stage-1 mounts a btrfs volume's TOP level to resolve findiso=, so a path
# inside a subvolume is unreachable and the box boots to an emergency shell
# after it's already left the working OS. Refuse btrfs staging outright.
stage_fstype="$(findmnt -no FSTYPE --target "$stagedir")"
[ "$stage_fstype" != btrfs ] \
|| die "$stagedir is btrfs: findiso= mounts the volume's top level, so a path inside a subvolume never resolves. Stage on a non-btrfs partition (ext4/vfat/ntfs)."
# PARTUUID of the staging partition. Handed to the installer as
# homelab.logpart= so it can mount this partition rw and persist its whole
# run — disko + nixos-install output included — to a file next to the iso.
# This partition is on a DIFFERENT disk from the one disko wipes (guarded
# above), so unlike $boot it SURVIVES the install: a failed attempt otherwise
# leaves nothing to debug, its journal having died on tmpfs at the reboot.
# Best-effort — an LVM/mdraid stage_src has no PARTUUID, in which case logging
# PARTUUID of the staging partition, handed to the installer as
# homelab.logpart= so it can persist the whole install's log there — it's on
# a different disk than the one disko wipes, so it survives a failed
# install. Best-effort: an LVM/mdraid stage_src has no PARTUUID, so logging
# is simply skipped rather than blocking the install.
local stage_partuuid
stage_partuuid="$(lsblk -no PARTUUID "$stage_src" 2>/dev/null | head -1 | tr -d ' ' || true)"
# Last chance to back out. This is the most destructive command in the
# script — it reboots the machine you are typing at and the wipe that
# follows is unattended — so it confirms just like `flash` and `kexec-local`
# do, both of which are less final than this.
# Last chance to back out: this reboots the machine you're typing at into an
# unattended wipe, so it confirms like `flash`/`kexec-local` do.
if [ "$assume_yes" != "--yes" ]; then
echo ">> about to REINSTALL this machine from scratch:"
echo " hostname: $(uname -n)"
@@ -392,16 +291,13 @@ local_install_prepare_and_reboot() {
initrd="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.initialRamdisk)/initrd"
isodir="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.isoImage)"
iso="$(one_match 'installer iso' "$isodir"/iso/*.iso)"
# The live ISO's root is a tmpfs; stage 1 finds the real system's init via
# init=<toplevel>/init, which the grub/isolinux menu supplies on a normal
# boot (iso-image.nix). EFI-stub-booting our own cmdline, we must pass it too
# — omit it and stage 1 loop-mounts the iso fine, then dies on
# The grub/isolinux menu normally supplies init=<toplevel>/init; EFI-stub
# booting our own cmdline means we must pass it too, or stage 1 dies on
# "stage 2 init script (/mnt-root//init) not found".
toplevel="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.toplevel)"
# A short write is not visible until the reboot, when findiso finds a
# truncated iso and drops to an emergency shell. Check first — `install`
# prints no progress and the iso is ~1GB.
# Check space before writing: a short write isn't visible until reboot,
# when findiso finds a truncated ~1GB iso and drops to an emergency shell.
local need_stage need_boot avail_stage avail_boot
need_stage="$(stat -Lc %s "$iso")"
need_boot="$(( $(stat -Lc %s "$kernel") + $(stat -Lc %s "$initrd") + $(stat -Lc %s "$hostkey") ))"
@@ -417,15 +313,9 @@ local_install_prepare_and_reboot() {
install -Dm644 "$initrd" "$boot/homelab-installer/initrd"
install -Dm644 "$iso" "$stagedir/homelab-installer.iso"
# The ISO is built from a PUBLIC repo and deliberately carries no
# credentials, so the host key has to travel with the staged installer or
# the auto-install run has nothing to seed /etc/ssh with — and without that,
# sops can't decrypt on boot #1, /etc/shadow gets written once with a locked
# darman, and no later `deploy switch` can fix it (README).
#
# $boot lives on the OS disk, so disko destroys this copy minutes later. The
# mode is advisory on vfat (permissions come from the mount's fmask, 0077 on
# a NixOS/systemd-boot ESP) — it is the wipe, not the mode, doing the work.
# The ISO is built from a public repo with no credentials, so the host key
# must travel with the staged installer or sops can't decrypt on boot #1
# (README). $boot is on the OS disk, so disko destroys this copy minutes later.
install -Dm600 "$hostkey" "$boot/homelab-installer/ssh_host_ed25519_key"
install -Dm644 "$hostkey.pub" "$boot/homelab-installer/ssh_host_ed25519_key.pub"
boot_src="$(findmnt -no SOURCE --nofsroot --target "$boot")" \
@@ -434,22 +324,15 @@ local_install_prepare_and_reboot() {
[ -n "$boot_partuuid" ] \
|| die "couldn't read a PARTUUID for $boot ($boot_src) — the installer needs it to find the host key"
# findiso= is a path relative to whatever partition the initrd finds it on
# (it mounts every blkid-visible partition looking for it), not to `/`, if
# $stagedir is a subdirectory of a bigger filesystem rather than a mountpoint
# itself. It must KEEP its leading slash: stage-1 tests `-e /findiso$isoPath`,
# so a bare `var/tmp/x.iso` becomes `/findisovar/tmp/x.iso` and never matches.
# Prefixing then squeezing handles both ends: stagedir == the mountpoint
# (strip leaves "") and mnt_point == "/" (strip leaves a relative path).
# findiso= is relative to whichever partition the initrd finds it on, and
# must KEEP its leading slash: stage-1 tests `-e /findiso$isoPath`, so a bare
# `var/tmp/x.iso` becomes `/findisovar/tmp/x.iso` and never matches.
mnt_point="$(findmnt -no TARGET --target "$stagedir")"
iso_relpath="$(printf '/%s/%s' "${stagedir#"$mnt_point"}" homelab-installer.iso | tr -s /)"
# Identical either way — only the mechanism that gets the kernel booted with
# it differs.
# root=LABEL=<volumeID> matches what the ISO menu passes; findiso overwrites
# /dev/root with the loop-mounted iso regardless, but keep it honest.
# boot.shell_on_fail gives a shell instead of the reboot/ignore prompt if
# stage 1 ever fails again. init= is the one that actually made this work.
# root=LABEL=<volumeID> matches what the ISO menu passes (findiso overwrites
# /dev/root regardless); boot.shell_on_fail gives a shell instead of a
# reboot/ignore prompt if stage 1 fails again.
local cmdline volumeID
volumeID="$(nix eval --raw .#nixosConfigurations.installer-iso.config.isoImage.volumeID)"
cmdline="init=$toplevel/init nohibernate root=LABEL=$volumeID boot.shell_on_fail loglevel=4 lsm=landlock,yama,bpf findiso=$iso_relpath homelab.install=$config homelab.keypart=$boot_partuuid"
@@ -483,31 +366,24 @@ EOF
require_tracked() {
local config="$1" cfgfile="hosts/$1/configuration.nix" f
[ -e "$cfgfile" ] || die "no $cfgfile in the repo"
# No .git at all (e.g. a tarball export of the repo, no working tree), or no
# git binary, means there's nothing that CAN be untracked — nothing to check.
# Only skip on that, not on any other git failure.
# No .git or no working tree (e.g. a tarball export) means nothing CAN be
# untracked — skip only on that, not on any other git failure.
command -v git >/dev/null 2>&1 || return 0
git -C "$REPO" rev-parse --is-inside-work-tree >/dev/null 2>&1 || return 0
# Every .nix in hosts/<config>/, not just configuration.nix: an untracked
# disk-config.nix is exactly as invisible to the flake, and it is the file
# that decides which disk gets wiped.
# Every .nix in hosts/<config>/, not just configuration.nix — an untracked
# disk-config.nix decides which disk gets wiped and is just as invisible.
for f in "hosts/$config"/*.nix; do
git -C "$REPO" ls-files --error-unmatch "$f" >/dev/null 2>&1 \
|| die "$f is untracked — 'git add hosts/$config' first (flakes ignore untracked files)"
done
}
# The password field of a Proton Pass item ("--field password" prints the bare
# value, one line), or empty if pass-cli is missing / logged out / has no such
# item — every caller then falls back to the normal interactive prompt.
#
# Resolve the title to an item id among ACTIVE items first, because `item view
# --item-title` has no state filter: Proton Pass keeps deleted items in the
# trash, and if a trashed item shares the title, view can match THAT one and
# return an empty password with exit 0. Empty is indistinguishable from "no such
# item", so the only symptom is a silent fall back to the interactive prompt
# even though the vault clearly holds the entry. (Hit for real on darman@neptun,
# which had an Active and a Trashed copy.)
# The password field of a Proton Pass item, or empty if pass-cli is missing /
# logged out / has no such item — callers then fall back to an interactive
# prompt. Resolves the title among ACTIVE items first, because `item view
# --item-title` has no state filter and can silently match a trashed item of
# the same title instead, returning an empty password with exit 0 (hit for
# real on darman@neptun, which had both an Active and a Trashed copy).
proton_pass_password() {
local title="$1" vault="${HOMELAB_PASS_VAULT:-HomeLab}" id pw
command -v pass-cli >/dev/null 2>&1 || return 0
@@ -564,15 +440,10 @@ case "$cmd" in
o=(-o ControlMaster=auto -o "ControlPath=$cm" -o ControlPersist=300 \
-o StrictHostKeyChecking=accept-new)
# Root's password from Proton Pass, fed to ssh/scp via sshpass -e. Only the
# first (master) connection authenticates; the rest ride the control socket.
#
# SSHPASS is exported here rather than passed as `env SSHPASS=... sshpass`.
# Both end up equally safe at rest: `env` execs its target immediately, so
# the assignment is only in argv for the sub-millisecond before exec, after
# which /proc/PID/cmdline reads plain `sshpass -e`. Exporting just closes
# that race window and drops a process. Either way the secret lives in the
# child's environ, which is readable by the owner and root only.
# Root's password from Proton Pass, fed to ssh/scp via sshpass -e; only the
# first (master) connection authenticates, the rest ride the control socket.
# Exported rather than `env SSHPASS=... sshpass` to close the sub-millisecond
# argv-exposure race before exec (either way the secret only lives in environ).
sp=()
root_item="${HOMELAB_PASS_ROOT_ITEM:-root@$config}"
root_pw="$(proton_pass_password "$root_item" || true)"
@@ -613,24 +484,18 @@ case "$cmd" in
ssh "${o[@]}" -O exit "root@$host" 2>/dev/null || true # close control socket
unset SSHPASS
# NB: no ssh-keygen -R here on purpose. kexec-run.sh copies /etc/ssh/ssh_host_*
# into the appended initrd and restore-remote-access.nix installs them back
# into the installer's /etc/ssh, so the host key SURVIVES the jump. Clearing
# known_hosts would just throw away the TOFU record for no reason.
# NB: no ssh-keygen -R here on purpose — the kexec installer keeps the box's
# ssh host key (restore-remote-access.nix), so known_hosts is still valid.
echo ">> box is kexec-ing. Wait ~1-2 min for the installer + network, then:"
echo " ./deploy install $config $host"
;;
kexec-local)
# No ssh, no second machine: build the same RAM installer as `kexec`, but
# run it directly on this box (you're sitting at it). The current shell
# drops when the kernel switches, same as any reboot — that's expected,
# not a failure. Disks are untouched; only the running kernel changes.
#
# This is a one-way trip on the machine you are typing at, so every check
# that can fail is done BEFORE the point of no return, and nothing that the
# jump depends on is cleaned up behind it (see the trap discussion below).
# Build the same RAM installer as `kexec`, but run it directly on this box
# (no ssh/second machine). One-way trip on the machine you're typing at, so
# every check that can fail runs BEFORE the point of no return (see the
# trap discussion below).
require_root "kexec-local"
assume_yes=""
@@ -719,11 +584,9 @@ case "$cmd" in
[ "$(cat /sys/kernel/kexec_loaded 2>/dev/null || echo 0)" = 1 ] \
|| { rm -rf "$stage"; die "kexec reported success but no image is loaded — aborting"; }
# THE trap MUST GO NOW. kexec-run.sh backgrounds `nohup sh -c "sleep 6 &&
# $SCRIPT_DIR/kexec -e"` and returns immediately, so the binary that
# performs the jump still has to exist ~6s after this script would normally
# exit. Letting the EXIT trap rm -rf "$stage" deletes it out from under that
# sleeping shell and the machine silently never jumps.
# THE trap MUST GO NOW: kexec-run.sh backgrounds the actual jump ~6s in the
# future, so an EXIT trap rm -rf'ing $stage here would delete the binary
# that performs it and the machine would silently never jump.
trap - EXIT
sync
@@ -737,9 +600,8 @@ case "$cmd" in
install)
config="${2:-}"; host="${3:-}"; assume_yes="${4:-}"
{ [ -n "$config" ] && [ -n "$host" ]; } || die "usage: ./deploy install <config> <host> [--yes]"
# $KEYDIR, not a bare $HOME — see its definition. This same check runs
# inside installer-iso, where homelab-auto-install.service has no $HOME and
# has just dropped the key into /root/.config/homelab/<config>/.
# $KEYDIR, not a bare $HOME — see its definition (also runs inside
# installer-iso, which has no $HOME).
hostkey="$KEYDIR/$config/ssh_host_ed25519_key"
[ -f "$hostkey" ] || die "missing host key: $hostkey"
[ -d "./hosts/$config" ] || die "no ./hosts/$config directory in the repo"
@@ -761,9 +623,8 @@ case "$cmd" in
[ -f "./hosts/$config/disk-config.nix" ] || die "no ./hosts/$config/disk-config.nix"
echo ">> disko .#$config onto this box's OS disk (WILL be wiped)"
# `.#disko`, not github:nix-community/disko — the revision comes from this
# repo's flake.lock rather than upstream master-of-the-day, and resolves
# from the local store. See the nixos-anywhere input in flake.nix.
# `.#disko`, not github:nix-community/disko: pins to this repo's
# flake.lock revision instead of upstream master-of-the-day.
nix run ".#disko" -- \
--mode disko "./hosts/$config/disk-config.nix"
@@ -787,9 +648,7 @@ case "$cmd" in
--target-host "root@$host")
# nixos-anywhere's --env-password reads root's ssh password from $SSHPASS
# (it ships its own sshpass), so a vault hit skips the ssh-copy-id prompt.
# Exported rather than `env SSHPASS=...` for consistency with `kexec`;
# see the note there — it's a marginal win, not a leak fix.
# (its own bundled sshpass), so a vault hit skips the ssh-copy-id prompt.
root_item="${HOMELAB_PASS_ROOT_ITEM:-root@$config}"
root_pw="$(proton_pass_password "$root_item" || true)"
if [ -n "$root_pw" ]; then
@@ -812,9 +671,7 @@ case "$cmd" in
echo ">> nixos-rebuild $cmd .#$config on darman@$host"
# --ask-sudo-password, not the deprecated --use-remote-sudo: common.nix sets
# security.sudo.wheelNeedsPassword = true, and --use-remote-sudo only
# prefixes with sudo without ever prompting. Asks for darman's password
# (the darman_password hash in each host's sops file).
# wheelNeedsPassword = true, and --use-remote-sudo never actually prompts.
rebuild=(nix run nixpkgs#nixos-rebuild -- "$cmd"
--flake ".#$config"
--target-host "darman@$host"
@@ -823,30 +680,19 @@ case "$cmd" in
item="${HOMELAB_PASS_ITEM:-darman@$config}"
pw="$(proton_pass_password "$item" || true)"
if [ -n "$pw" ] && command -v setsid >/dev/null 2>&1; then
# nixos-rebuild prompts with getpass(), which reads /dev/tty and ignores a
# piped stdin. setsid drops the controlling terminal, so getpass falls back
# to stdin and takes the vault password (it warns about echo — harmless,
# nothing is echoed since the password never reaches the terminal).
#
# Caveat of dropping the tty: EVERY prompt in the subtree now reads this
# stdin, not just the sudo one. Feed the line a few times so a retry or a
# second sudo ask doesn't hit EOF and hang. Anything else that prompts
# (an ssh key passphrase, a host-key confirmation) will still fail — fix
# those out of band rather than by feeding more lines here.
# nixos-rebuild's getpass() reads /dev/tty and ignores piped stdin; setsid
# drops the controlling terminal so it falls back to stdin instead. Every
# prompt in the subtree now reads that stdin, so the password line is fed
# a few times to survive a retry — anything else that prompts still fails.
echo ">> sudo password from Proton Pass ($item)"
printf '%s\n%s\n%s\n' "$pw" "$pw" "$pw" | setsid -w "${rebuild[@]}"
else
"${rebuild[@]}"
fi
# jupiter's 29G eMMC has no room to just let generations pile up between
# gc.dates=weekly runs (common.nix) — that's exactly how it filled up
# once already. configurationLimit=5 (also common.nix) makes
# switch-to-configuration prune generations beyond 5 as part of the
# switch above, but pruning a generation only drops it as a GC root —
# the store paths themselves still need an actual collect to free the
# disk. So do that here, right after every switch, rather than waiting
# up to a week for it to matter again.
# jupiter's 29G eMMC has already filled up once waiting for the weekly gc
# (common.nix). configurationLimit=5 only drops old generations as GC
# roots, so collect explicitly here rather than waiting up to a week.
if [ "$cmd" = switch ] && [ "$config" = jupiter ]; then
echo ">> jupiter: collecting garbage post-switch (keeps the eMMC under the 5-generation cap)"
need ssh
@@ -889,9 +735,8 @@ case "$cmd" in
sync
# If this config has a dedicated sops age key, drop it on the ROOT ext4
# partition at /var/lib/sops-nix/age.txt so sops decrypts on first boot.
# (The Pi's vfat partition isn't mounted at runtime, so the key can't live
# there.) Key stays off-repo, out of the nix store, and out of the image.
# partition (the Pi's vfat one isn't mounted at runtime) so sops decrypts
# on first boot. Key stays off-repo, out of the nix store and the image.
keyfile="$KEYDIR/$config/age.txt"
if [ -f "$keyfile" ]; then
echo ">> installing sops age key onto the root partition"
scripts/edit_secrets
+3 -4
View File
@@ -36,10 +36,9 @@ if [ "$show" -eq 1 ]; then
exec nix shell nixpkgs#sops -c sops --decrypt "$file"
fi
# sops opens $EDITOR on a temp file and re-encrypts only if it changed.
# Pitfalls that cause "File has not changed, exiting":
# - $EDITOR unset: no editor is on the `nix shell` PATH -> bundle one.
# - GUI editor (code/zed) forks and returns instantly -> force --wait.
# sops re-encrypts only if the $EDITOR session actually changed the temp file.
# GUI editors (code/zed) return instantly unless forced to --wait, and if
# $EDITOR is unset no editor exists on the `nix shell` PATH, so bundle one.
editor="${VISUAL:-${EDITOR:-}}"
extra=()
case "$editor" in
scripts/immich-import-legacy-db
+28 -55
View File
@@ -1,50 +1,28 @@
#!/usr/bin/env bash
# Import the OLD ZimaOS/CasaOS Immich database into the NixOS-managed one.
# Run this ON jupiter, as root, ONCE, AFTER the first `./deploy switch jupiter`
# that ships services/media/immich.nix (the empty `immich` DB must exist).
#
# The media files are moved separately — do that FIRST, it is a rename on the
# same filesystem, so instant even at 9.1G. Move the CONTENTS, not the dir:
# systemd.tmpfiles already created /mnt/data/AppData/immich on the first
# deploy, so `mv <src> <dst>` would nest it as .../immich/upload/ and every
# thumbnail lookup would ENOENT.
# Run ONCE on jupiter, as root, after the first `./deploy switch jupiter` that
# ships services/media/immich.nix (the empty `immich` DB must already exist).
#
# Move the media files separately FIRST (a same-filesystem rename, instant
# even at 9.1G) — move the CONTENTS of /mnt/data/Immich/upload into
# /mnt/data/AppData/immich, not the directory itself, or it nests under
# .../immich/upload and every thumbnail lookup ENOENTs:
# systemctl stop immich-server immich-machine-learning
# mv /mnt/data/Immich/upload/* /mnt/data/AppData/immich/
# chown -R immich:immich /mnt/data/AppData/immich
# chmod 700 /mnt/data/AppData/immich
# chown -R immich:immich /mnt/data/AppData/immich && chmod 700 /mnt/data/AppData/immich
#
# Expected afterwards: library/ upload/ thumbs/ encoded-video/ profile/ backups/
#
# The legacy cluster turned out to be Postgres 14 running VectorChord 0.3.0 +
# pgvector 0.8.1 (NOT pgvecto.rs), the same extensions nixpkgs ships — so this
# is a plain version-upgrade dump/restore and the smart-search and face
# embeddings come across intact. No re-running the ML jobs over the library.
# Upstream's accepted VectorChord range is >= 0.3, < 2.0, so 0.3.0 -> 1.1.1 is
# a supported jump; the REINDEX at the end is what upstream asks for after a
# version change.
#
# What this script does:
# 1. cp -a the legacy PGDATA to a scratch dir (the original is never touched,
# never even mounted rw — postgres would replay WAL into it).
# 2. Boots that copy under immich's own PG14 image, pinned to the SAME
# VectorChord version nixpkgs has (1.1.1), and runs `ALTER EXTENSION
# vchord UPDATE` so the catalog matches the loaded library.
# 3. Dumps it with the LOCAL pg_dump (17.x) over TCP, not the container's
# pg_dump (14.x) — dumping with the newer tool is the supported direction.
# 4. Restores into a scratch DB, hands ownership to the immich role, shows
# you the row counts, and only swaps it into place after you confirm.
#
# Afterwards Immich runs its own schema migrations up to 2.7.5 on first start.
# The legacy cluster is Postgres 14 + VectorChord 0.3.0 + pgvector 0.8.1 (the
# same extensions nixpkgs ships), so this is a plain version-upgrade
# dump/restore — smart-search and face embeddings come across intact with no
# ML rerun needed.
set -euo pipefail
LEGACY="${LEGACY:-/mnt/data/Immich/pg-data}"
WORK="${WORK:-/var/tmp/immich-import}"
# Pinned to EXACTLY what the legacy cluster records in pg_extension —
# vchord 0.3.0 + pgvector 0.8.1 — so the old server reads its own indexes
# without any in-place extension upgrade. The target side is vchord 1.1.1 /
# pgvector 0.8.2, which is fine: a dump/restore rebuilds every index from
# scratch, so only the index DEFINITION has to still be valid there.
# Pinned to exactly what the legacy cluster's pg_extension records (vchord
# 0.3.0/pgvector 0.8.1) so it reads its own indexes unmodified; the dump/
# restore rebuilds indexes from scratch on the target's newer versions, so
# only the index definitions need to stay valid.
IMAGE="${IMAGE:-ghcr.io/immich-app/postgres:14-vectorchord0.3.0-pgvector0.8.1}"
CTR=immich-legacy-pg
PORT="${PORT:-15432}"
@@ -71,13 +49,10 @@ cp -a "$LEGACY" "$WORK/pgdata"
# A crashed cluster leaves this behind; it makes the container refuse to start.
rm -f "$WORK/pgdata/postmaster.pid"
# The dump runs over TCP (local pg_dump 17 -> published port), and this
# cluster's own pg_hba wants a password for host connections — the marketplace
# app's POSTGRES_PASSWORD is long gone, and POSTGRES_HOST_AUTH_METHOD only
# applies when the image INITIALISES a cluster, not to an existing one. This is
# a scratch copy bound to 127.0.0.1 for the length of one dump, so trust it.
# REPLACE the file rather than appending: pg_hba is first-match-wins, and the
# image's existing scram-sha-256 line would shadow anything added below it.
# The marketplace app's original POSTGRES_PASSWORD is long gone, and
# POSTGRES_HOST_AUTH_METHOD only applies when the image initializes a cluster
# (not an existing one) — so pg_hba is REPLACED outright (not appended, since
# it's first-match-wins) to trust this scratch copy while it's dumped.
cat > "$WORK/pgdata/pg_hba.conf" <<'EOF'
local all all trust
host all all 0.0.0.0/0 trust
@@ -101,10 +76,10 @@ for _ in $(seq 1 60); do
done
[ "${ready:-}" = 1 ] || { podman logs --tail 30 "$CTR"; die "legacy postgres never became ready"; }
# The compose stack's POSTGRES_USER is not recorded anywhere on disk and is NOT
# necessarily "postgres" — the ZimaOS/CasaOS marketplace app used "casaos".
# pg_isready reports "accepting connections" even for a role that doesn't
# exist, so probe for one that can actually log in.
# The original POSTGRES_USER isn't recorded on disk and wasn't necessarily
# "postgres" (this marketplace app used "casaos"), and pg_isready reports
# ready even for a role that doesn't exist — so probe for one that can
# actually log in.
if [ -z "$LEGACY_USER" ] || ! podman exec "$CTR" psql -U "$LEGACY_USER" -lqt >/dev/null 2>&1; then
for candidate in casaos immich postgres; do
if podman exec "$CTR" psql -U "$candidate" -lqt >/dev/null 2>&1; then
@@ -160,13 +135,11 @@ echo ">> errors logged: $(grep -c '^ERROR' "$WORK/restore.log" || true) (see $W
grep '^ERROR' "$WORK/restore.log" | sort -u | head -10 | sed 's/^/ /' || true
step "handing ownership to the immich role"
# --no-owner made everything owned by the restoring role (postgres); immich
# connects as "immich" and its startup migrations run ALTER TABLE, so it must
# own its own schema. NOT `REASSIGN OWNED BY postgres` — that also sweeps up
# system objects and fails with "cannot reassign ownership of objects owned by
# role postgres because they are required by the database system". Extension-
# owned routines/types are excluded for the same reason; immich never alters
# those, and they correctly stay with postgres.
# immich's own ALTER TABLE migrations need it to own its schema, but plain
# `REASSIGN OWNED BY postgres` also sweeps up system objects and fails on ones
# the database system requires — so ownership is walked table-by-table
# instead, skipping extension-owned routines/types, which correctly stay with
# postgres.
sudo -u postgres psql -qd "$STAGING_DB" <<'SQL'
ALTER SCHEMA public OWNER TO immich;
DO $$