The staged installer booted, the auto-install service picked up terra's host
key and removed its temporary UEFI entry — then died before running anything:
env: 'bash': No such file or directory (status 127)
The service ran with the restricted PATH a `path = [...]` list produces, which
has no bash — so `./scripts/deploy`'s `#!/usr/bin/env bash` shebang could not
resolve, let alone the nix / nixos-install / git / sudo it then calls.
Point the unit's PATH at /run/current-system/sw/bin (+ /run/wrappers/bin for
sudo), which carries the whole installer toolset. mkForce because NixOS
otherwise derives environment.PATH from `path` and that line would win. HOME
moves into the same environment attr.
Verified: environment renders {HOME=/root,
PATH=/run/current-system/sw/bin:/run/wrappers/bin}, and sw/bin contains bash,
nix, nixos-install, git, sudo, efibootmgr, mount, grep, sed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
terra booted the staged installer this time but dropped to an emergency
shell: stage 1 mounted /sysroot, then timed out on /sysroot/nix/.ro-store
waiting for /dev/disk/by-label/nixos-minimal-26.05-x86_64.
findiso= is handled only in the scripted stage-1-init.sh, which loop-mounts
the file the param points at and symlinks it to /dev/root. The systemd initrd
— the default since 26.05 — has no findiso handling at all: iso-image.nix
mounts /iso directly from /dev/disk/by-label/<volumeID>, a label that only
exists when the ISO is the physical boot medium. Booted as kernel + initrd
off the ESP with the iso as a plain file on another partition, that label
never appears, so the store squashfs never mounts.
The entire `install <config> localhost` path is built on findiso, so pin the
installer to script stage 1. Verified: /iso device flips to /dev/root,
root=LABEL=... is added to the params, the rebuilt initrd's /init is
stage-1-init.sh and carries the findiso logic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`install <config> localhost` assumed systemd-boot. terra's CachyOS boots
Limine, so it stopped at "/boot/loader/entries doesn't exist" — the check
added in e538788 doing its job, but with nowhere to go.
Limine cannot help here at all: `bootctl status` lists it as
`✗ One-shot entry control`, and CachyOS's pacman hooks regenerate
limine.conf regardless. So drop below the bootloader entirely and use the
firmware's own BootNext, pointing at a temporary UEFI entry that
EFI-stub-boots the installer kernel straight off the ESP. That keeps the
property which makes this safe to attempt: BootNext is spent by that one
boot, so a failed try still comes back on the normal bootloader.
- picked at runtime: systemd-boot loader entry when $BOOT/loader/entries
exists, else arm_efi_bootnext(). jupiter/neptun and terra-after-install
keep the systemd-boot path.
- `efibootmgr --create-only`, NOT `--create`: the latter pushes the entry
to the front of BootOrder, which would make a wiped installer the
permanent default if the install died halfway.
- the EFI stub loads initrd= from the volume it was loaded from, so this
mode stages on --print-esp-path rather than --print-boot-path.
- stale entries from an earlier attempt are removed before adding one, and
homelab-auto-install.service deletes the entry as soon as it boots, so
nothing lingers in NVRAM pointing at a reformatted partition.
- label matching is EXACT ("Homelab Installer"); a prefix match would have
deleted this box's Windows or Limine entry.
Verified against terra's real NVRAM (read-only): the label parser picks out
Limine/UEFI OS/Windows by exact name and rejects prefixes, and both branches
run end-to-end under stubs — BootNext mode emits the right --disk/--part,
loader path and initrd= cmdline, systemd-boot mode still writes its entry and
never calls efibootmgr.
README/CLAUDE.md corrected: terra runs Limine, not systemd-boot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The `install <config> localhost` auto-path added in 0ea9020 could not have
completed. Fixed three blockers plus the guard that was silently not guarding.
Inside installer-iso the run died before doing anything:
- systemd sets no $HOME for a service without User= (SetLoginEnvironment=
defaults to false), and this script runs under `set -u`, so it aborted on
the bare $HOME with "unbound variable". Added $KEYDIR + Environment=HOME.
- the host key it needs to seed /etc/ssh isn't on the ISO at all — that is
built from a public repo and carries no credentials on purpose. It now
travels on the boot partition, located via homelab.keypart=<PARTUUID> on
the kernel cmdline, and dies with the disko wipe minutes later. Without
it sops can't decrypt on boot #1 and mutableUsers locks darman for good.
- installation-cd-minimal leaves experimental-features unset, so both
`nix run` and `nixos-install --flake` failed. (The nixos-images kexec
installer sets them itself, which is why the same branch worked after
kexec-local but not from the ISO.)
The staging-dir guard passed everything on btrfs: findmnt prints the
subvolume as /dev/sdb2[/@], lsblk can't open that, and an empty parent was
treated as "different disk" — so it allowed staging the iso on the very disk
disko then wiped. terra's current CachyOS root is exactly that layout. Now
uses --nofsroot, resolves EVERY whole-disk ancestor (LVM/RAID span several:
/mnt/ssd_01 -> sdd+sde), and treats "can't tell" as a hard error. btrfs
staging is refused outright — stage-1 mounts a btrfs volume's top level, so
an iso inside a subvolume is unreachable.
findiso= lost its leading slash whenever the staging mountpoint was /,
giving /findisovar/tmp/x.iso and an emergency shell after the reboot.
Also:
- confirm before rebooting, like flash/kexec-local already do; --yes skips
it and is what the ISO passes itself
- $BOOT from `bootctl --print-boot-path`, not a hardcoded /boot
- free-space checks on both target partitions before the ~1GB copy
- `nix run .#disko` / `.#nixos-anywhere` from locked inputs instead of
github:... master-of-the-day, resolved while a disk is being wiped
- one_match warns instead of silently taking [0]; require_tracked covers
every hosts/<config>/*.nix; flash traps its mount
- drop nixos-images' `inputs.nixpkgs.follows` — that input doesn't exist,
it only printed a warning on every nix command
Verified: the prepare path exercised under stubs against this box's real
disks (btrfs-on-OS-disk, tmpfs, LVM, subdirectory), shellcheck clean, all
six configs evaluate, checks.kexec-local still passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
./scripts/deploy install <config> localhost now branches on is_live_installer()
(checks uname -n): outside a live installer it builds installer-iso, stages
its kernel/initrd on the ESP and the iso file on a disk the caller picks
(never auto-picked — the wrong disk here is destroyed mid-install), writes a
systemd-boot one-shot findiso= entry with homelab.install=<config> on the
kernel cmdline, and does a real systemctl reboot (not kexec — terra's
kexec-local hang is specifically in kexec's device-shutdown pass, a real ACPI
reboot never runs that code at all).
installer-iso gains homelab-auto-install.service: once homelab-checkout.service
clones the repo, it reads homelab.install= back off /proc/cmdline and re-runs
the identical deploy command itself, now genuinely inside the installer, so
it takes the disko+nixos-install branch instead of preparing again. The whole
reinstall is one command and unattended after the first reboot.
Also: every root-requiring path (kexec-local, the new prepare-and-reboot
branch, the disko+nixos-install branch) self-elevates via a require_root()
helper that re-execs the original invocation under sudo -E, instead of dying
and asking the caller to prefix sudo themselves. Uses an absolute script path
captured before the script's own cd, so the re-exec is correct regardless of
how it was invoked.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
require_tracked() in scripts/deploy now skips its git-tracked-file check
when there's no .git at all (nothing can be untracked in that case) — needed
for an earlier baked-in-`self` approach and kept as a generic fallback.
Since the repo is public now, installer-iso instead clones current master
via a homelab-checkout.service (after network-online.target) on every boot,
to /root/homelab. One ISO build stays useful indefinitely instead of going
stale, and there's still no rsync-the-repo-over step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed on real hardware: kexec's device_shutdown() pass runs (SCSI disks
sync fine in the log) then the machine goes dark for good — journalctl
--list-boots showed a ~15min gap before the next boot, a genuine hang needing
a manual power cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT):
discrete AMD GPUs are known to hang during kexec's device-shutdown pass with
no clean handoff before the jump, same class of issue as jupiter's
reboot=pci warm-reboot workaround, just fatal here instead of slow.
README's terra install section now leads with the USB installer path instead
(build ISO, dd to USB, rsync the repo over, disko + nixos-install locally).
CLAUDE.md's gotchas list gets the same warning. installer-iso is renamed from
jupiter-installer to homelab-installer since it's genuinely host-agnostic,
and now ships git.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Tome (née AudibleLibrary) is darman's own Photino/.NET desktop app, private
repo on our own gitea. Fetched as a flake input over ssh with darman's
ambient key — same mechanism as any other git input, private or not.
buildDotnetModule package: the Preact/Vite frontend (Tome.App/ClientApp)
builds as its own buildNpmPackage derivation and gets copied into the
published app's wwwroot, since upstream's in-project MSBuild npm target has
no network access in the Nix sandbox. Photino.Native's runtime deps (gtk3,
webkitgtk_4_1, libnotify) are wrapped in — confirmed via readelf/ldd that
this Photino build already targets webkit2gtk-4.1, not the now-removed 4.0.
Also added dotnet-sdk + nodejs to terra for developing Tome locally, and
vivaldi (unfree, extends the existing allowUnfreePredicate).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Not in nixpkgs; packaged by github:tomsch/proton-pass-cli-nix. Used by
./scripts/deploy to autofill sudo/ssh passwords from the "HomeLab" vault.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kexec-local could never actually jump. nixos-images' kexec-run.sh ends with
`nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &` and returns immediately, so
the EXIT trap's `rm -rf "$stage"` deleted the kexec binary out from under the
sleeping shell. The box stayed on the old kernel and it looked like a slow boot.
Clear the trap before jumping, verify /sys/kernel/kexec_loaded, then sleep past
the timer.
Preflight everything before the point of no return, since this jumps the machine
you are typing at: CONFIG_KEXEC, kernel lockdown, exec-capable staging dir, free
space, RAM vs image size, and that the tarball holds all five expected files.
Stage on /var/tmp rather than /tmp because kexec-run.sh appends to initrd in
place and execs from that directory. sync before jumping (kexec -e skips
unmount). Confirmation prompt naming the host, since run in the wrong terminal
this kexecs the laptop; --yes skips it.
Drop the ssh-keygen -R added to the remote kexec path: kexec-run.sh copies
/etc/ssh/ssh_host_* into the appended initrd and restore-remote-access.nix
installs them back, so the host key survives the jump.
Proton Pass items are now keyed by <config> instead of <host>, since the address
is incidental and the config name is stable. kexec therefore takes <config>
<host>. Resolve titles among --filter-state active items first: a trashed item
with the same title shadowed the active one and returned an empty password,
which is indistinguishable from "no entry" and silently fell back to prompting
(hit on darman@neptun).
Other fixes: replace `ls glob | head -1` (returns empty with exit 0 on no match)
with a helper that dies; guard against untracked hosts/<config> since flakes
ignore untracked files; feed the sudo password more than once under setsid;
handle empty arrays under set -u; tolerate empty FSTYPE in the SD-card root
partition lookup; preflight zstdcat/dd/lsblk before the destructive dd; list
image and flash in the usage strings.
Add checks.x86_64-linux.kexec-local, a VM test driving the real script. It is
the only way to exercise kexec-local, which cannot be rehearsed on hardware. It
asserts the box left the old kernel, returned as nixos-installer, lost its old
/run, and kept its ssh host key. HOMELAB_KEXEC_TARBALL lets it reuse a prebuilt
installer instead of building ~500MB inside the guest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jupiter had a leftover docker-compose Immich on the RAID (/mnt/data/Immich,
9.9G) that survived the NixOS install. Native module now, media at
/mnt/data/AppData/immich, caddy vhost on 2283 with a 50GB body limit
(caddy's default rejects video uploads).
The package comes from nixpkgs-unstable, the module from the 26.05 pin:
26.05 ships immich 2.7.5, but that database was last written by 3.0.0 and
migrations only run forward --
corrupted migrations: previously executed migration
1776217577402-DropAuditTable is missing
Safe because the two module files are byte-identical at these revisions;
services/media/immich.nix carries the diff command to re-check on a bump.
Drop the input once the stable pin ships >= 3.0.0.
immich needs group "users" only to traverse /mnt/data/AppData (drwx--x---);
its own dir stays 0700 immich:immich. mediaLocation is outside /var/lib, so
the module's tmpfiles entry only ADJUSTS it -- add a rule that creates it.
scripts/immich-import-legacy-db does the database half: boots a copy of the
legacy PGDATA under the matching image (PG14 + vchord 0.3.0 + pgvector
0.8.1), dumps it with the local pg_dump 17, restores into a scratch DB,
fixes ownership, and only swaps after confirmation. Never touches the
original. The old cluster ran VectorChord, not pgvecto.rs, so the smart
search and face embeddings survive -- no ML re-run.
Imported: 666 assets, 25 people, 647 clip + 359 face embeddings, 2 users.
nixpkgs only carries Zitadel 2.71, which predates the login-v2 split and
cannot take a v3/v4 database (its migrations are forward-only), so the
instance running on the old Debian VPS could never have moved onto it.
authentik-nix ships 2026.5.4 and tracks upstream closely.
The authentik-nix input deliberately does not follow our nixpkgs, per
upstream's warning that overriding it breaks their pinned python
dependency set. That costs a second nixpkgs in the lock, so add
nix-community's Cachix to common.nix -- without it the closure is ~400
local derivations (npm, rust, python). The laptop that runs
scripts/deploy needs the same two lines in /etc/nix/nix.custom.conf.
Authentik's own module creates the database and orders its units against
postgresql.target, and recent versions need no redis, so the wiring is
just the module plus a secret. Pin postgresql explicitly so that editing
system.stateVersion can never silently demand a pg_upgrade of the
identity store.
Secret ownership is not uniform and the difference matters: authentik
and caddy take a systemd EnvironmentFile, which PID 1 reads as root
before dropping privileges, so root:root 0400 is correct. Headplane
opens its secret paths itself while already running as the headscale
user, so those three need an explicit owner or they fail to start.
Also on neptun:
- Pass Caddy's ACME account email through the same EnvironmentFile
mechanism and reference it with the Caddyfile {$VAR} placeholder.
services.caddy.email would render the address into the world-readable
store.
- Stop accepting MagicDNS from our own control server. headscale pushes
override_local_dns, so joining the tailnet would point neptun's
resolv.conf at a MagicDNS served by the tailscaled neptun itself hosts
-- a tailscaled failure would then also take out DNS, ACME renewal and
finally the certs for the control server every other node needs in
order to recover.
- Give headplane a writable DNS extra-records file. Its view of
headscale's config stays read-only, which is the right outcome for a
declarative box; records are data rather than config.
- Require a password for sudo. Deploys become interactive, but darman's
key is otherwise the only thing between the public internet and root.
- Enable zram (8 GB, and disko leaves no room for a swap device), and let
tailscaled-autoconnect retry instead of failing permanently when the
control server isn't up yet on a first boot.
networking.hosts still carries a PLACEHOLDER address for jupiter --
replace it from `headscale nodes list` once jupiter first enrols.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Group service modules by category (media, network, vpn, identity,
dev, desktop) to make the growing services/ dir easier to navigate.
containers.nix stays at the top level since it's a shared backend,
not a single-category service.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- sabnzbd, prowlarr, sonarr, radarr, clonarr, seerr, cinephage, mediamanager
services, wired into jupiter with LAN Caddy vhosts.
- Gitea: migrated the old ZimaOS docker instance's data (sqlite db, 4 repos,
no LFS objects) into the NixOS module's default stateDir layout. HTTP via
Caddy; git SSH on its own built-in server at :2222 (not :222 - the unpriv
gitea user can't bind <1024).
- mediamanager-nix flake input for the mediamanager service.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- services/pihole.nix: official pihole/pihole:2026.07.2 via podman, host net,
caps NET_ADMIN/NET_RAW/SYS_NICE/CHOWN; FTLCONF_* env config (upstream unbound,
DHCP 50-200, static lease jupiter, .sol domain, local records)
- unbound: resolveLocalQueries=false (was hijacking resolv.conf to :53 -> boot
DNS deadlock; the real root cause of the earlier failures too)
- password via sops FTLCONF env file; /var/lib/pihole created via tmpfiles
- VM-verified: mercury.sol/jupiter.sol/external all resolve, 0 restarts
- per-host darman_password (distinct hash) in secrets/{jupiter,vps,mercury}.yaml
-> hashedPasswordFile; different console password per host (ssh still key-only)
- mercury: dedicated age key (on boot partition post-flash), sops-nix wired
- AdGuard: module has no secret hook + writable config -> mutableSettings=true,
admin password set via web setup on first boot (never in repo/store)
- nixosConfigurations.mercury: aarch64, sd-image-aarch64, imports common.nix
- static net placeholders (CHANGE-ME), hostname mercury
- DNS service left undecided: commented pihole (services.pihole-ftl) + adguardhome
- no sops yet (add with the service if it needs a secret)
- add nixos-images input; nixosConfigurations.kexec bakes in the ssh login key
- build via config.system.build.kexecInstallerTarball
- deploy: ./deploy kexec <host> streams the installer to /tmp and kexecs
- works around ZimaOS RO root where nixos-anywhere ssh-copy-id fails
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- add sops-nix input + module (jupiter only, not the VM)
- secrets/jupiter.yaml: age-encrypted samba_password (safe to commit)
- .sops.yaml: encryption rule for admin age key
- secrets.nix: decrypt samba_password to /run/secrets on the host
- provisioning oneshot reads sops secret (host) or plaintext (VM), single value
- .sops private key stays off-repo (~/.config, /var/lib/sops-nix on host)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- add disko input; jupiter partitions/formats OS disk declaratively
- hardware-configuration.nix carries kernel modules only (disko owns fileSystems)
- data disk stays a plain unformatted mount, out of disko
- vbox unchanged (virtualbox-image supplies its own disk)
- README: nixos-anywhere remote install + daily rebuild loop
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
darman
