homelab/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Flake-based NixOS config for a homelab. Hosts: jupiter (ZimaBlade NAS, x86_64), neptun (netcup public reverse proxy + tailnet node, x86_64), mercury (Raspberry Pi 3B+ DNS/DHCP, aarch64). See README.md for the full install/deploy walkthrough.

Layout

flake.nix          # nixosConfigurations: real hosts + test/util targets
common.nix         # shared base: user darman (key-only ssh), nix settings, firewall :22, tz
services/<cat>/*.nix   # one reusable NixOS module per service, grouped by category
                       # (media, network, vpn, identity, dev, desktop); each opens
                       # ITS OWN firewall ports. services/containers.nix (podman
                       # backend) stays at the top level, shared across categories.
hosts/<h>/         # configuration.nix + disk-config.nix (disko) + hardware-configuration.nix + secrets.nix
secrets/<h>.yaml   # sops-nix, age-encrypted per host
scripts/deploy     # config-agnostic deploy wrapper (all args mandatory)
scripts/edit_secrets
.sops.yaml         # per-host encryption rules (admin key + each host's key)

A host = common.nix + the services/** modules it imports + its hosts/<h>/configuration.nix. services/ modules are engine-agnostic and shared across hosts (e.g. services/vpn/tailscale.nix, services/network/caddy.nix used by jupiter and neptun).

Commands

Eval/verify a config before deploying (eval only checks the module tree, not freeform config like pihole's TOML or a container's runtime):

nix eval --raw .#nixosConfigurations.<host>.config.system.build.toplevel.drvPath

Deploy (from a non-NixOS laptop too — runs nixos-rebuild/nixos-anywhere via nix run):

./scripts/deploy switch  <config> <host>     # daily rebuild + activate
./scripts/deploy install <config> <host>     # first install (nixos-anywhere, wipes OS disk)
./scripts/deploy kexec   <config> <host>     # RO-root box (ZimaOS): kexec into a RAM installer first
sudo ./scripts/deploy kexec-local [--yes]    # kexec THIS box (no ssh); confirm prompt unless --yes
./scripts/deploy image   mercury             # build the aarch64 SD image
./scripts/deploy flash   mercury /dev/sdX    # build + write SD + drop the sops age key

Both password prompts are auto-filled from the "HomeLab" Proton Pass vault, keyed by <config>, not <host> — items darman@<config> (sudo) and root@<config> (ssh). A trashed Proton Pass item with the same title shadows the active one and yields an empty password, so the script resolves the title among --filter-state active items first; a plain pass-cli item view --item-title silently returns the trashed copy and you get an interactive prompt with no explanation.

HOMELAB_KEXEC_TARBALL (+ _CPIO / _GZIP) makes kexec/kexec-local reuse a prebuilt installer instead of rebuilding ~500MB. The VM test below uses this.

Secrets (needs the admin age key at ~/.config/sops/age/keys.txt):

./scripts/edit_secrets secrets/<host>.yaml

Test a service config BEFORE touching hardware — always do this for nontrivial changes:

# x86 QEMU VM of mercury's DNS/DHCP stack (fast; validates pihole/unbound at runtime)
nix build .#nixosConfigurations.mercury-vm.config.system.build.vm -o result
./result/bin/run-mercury-vm-vm            # ssh -p 2223 darman@localhost (pw: test)
# jupiter services as a VirtualBox OVA
nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
# end-to-end VM test of `deploy kexec-local` (~45s once the tarball is built)
nix build .#checks.x86_64-linux.kexec-local -L

checks.kexec-local is the only way to exercise kexec-local at all: it jumps the machine you are typing at, so it cannot be rehearsed on real hardware and a failure looks exactly like a slow boot. It asserts the box actually left the old kernel (SSH drops then returns), came back as nixos-installer, lost its old /run, and kept its ssh host key. Run it after ANY change to the kexec paths.

Secrets (sops-nix)

• Each secrets/<host>.yaml is encrypted to the admin key (edit) + that host's key (runtime decrypt); rules in .sops.yaml. Private keys live OFF-repo: ~/.config/sops/age/keys.txt (admin), ~/.config/homelab/<host>/ (host keys).
• jupiter/neptun decrypt with their ssh host key (ssh-to-age recipient), shipped at install via nixos-anywhere --extra-files.
• mercury (SD image, no --extra-files) uses a dedicated age key at /var/lib/sops-nix/age.txt — ./scripts/deploy flash writes it to the ext4 root partition.
• A service password that must come from sops but whose module has no passwordFile hook (pihole, adguard) is injected via sops.templates → an env file → the service (FTLCONF_* for pihole). See hosts/mercury/configuration.nix.

Non-obvious gotchas (all learned the hard way)

• aarch64 (mercury): the x86 laptop needs extra-platforms = aarch64-linux in /etc/nix/nix.custom.conf (NOT /etc/nix/nix.conf — Determinate Nix regenerates that)
  • qemu-user-static-binfmt, else emulated builds fail with "platform mismatch". Or build on the Pi with --build-host darman@<ip>.
• pihole on mercury is a CONTAINER (services/network/pihole.nix, official image, host networking, caps NET_ADMIN/NET_RAW/SYS_NICE/CHOWN, FTLCONF_* env config). The native services.pihole-ftl module segfaults on the Pi 3B+ aarch64 — do not switch back.
• services.unbound.resolveLocalQueries = false is required: unbound listens on :5335, so leaving it true points the host's resolv.conf at 127.0.0.1:53 with nothing there → boot-time DNS deadlock (starves image pulls / list downloads). Host resolves via upstream networking.nameservers; pihole forwards to unbound explicitly at 127.0.0.1#5335.
• Remote deploy pushes unsigned closures: hosts set nix.settings.trusted-users = [ "root" "@wheel" ] (in common.nix) so a laptop-built closure is accepted by the target.
• jupiter: boot.kernelParams = [ "reboot=pci" ] (warm reboot hangs on that board); eMMC initrd modules pinned in configuration.nix (generate-config misses them); the 16TB×2 RAID0 data lives on /mnt/data with nofail, kept OUT of disko (never wiped).
• terra: ./scripts/deploy kexec-local hangs hard — do not use it there. Confirmed on real hardware: kexec's device_shutdown() pass runs (SCSI disks sync fine in the log), then the machine goes dark and never comes back — journalctl --list-boots showed a ~15min gap before the next boot, i.e. a genuine hang needing a manual power cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT): discrete AMD GPUs are known to hang during kexec's device-shutdown pass with no clean handoff before the jump — same class of issue as jupiter's reboot=pci workaround, just fatal here instead of slow. Use ./scripts/deploy install terra localhost instead (README's "First install on terra" section) — it detects it isn't inside a live installer yet and reboots via a real systemctl reboot + systemd-boot one-shot findiso= entry, not kexec.
• ./scripts/deploy install <config> localhost's behavior depends on uname -n (is_live_installer()): on a real running OS it builds installer-iso, stages it locally, and reboots into it (local_install_prepare_and_reboot()); only inside nixos-installer (kexec) or homelab-installer (installer-iso) does it actually run disko + nixos-install. installer-iso's homelab-auto-install.service closes the loop: it reads homelab.install=<config> back off /proc/cmdline (set by the prepare step) and re-runs the identical command itself once homelab-checkout.service has cloned the repo — the whole reinstall is one command and unattended after the first reboot. It confirms (type yes) before rebooting, like flash/kexec-local; --yes skips that and is what the ISO passes itself. It always ASKS where to stage the iso file (never auto-picks — the wrong disk here is destroyed mid-install); HOMELAB_INSTALLER_STAGE_DIR skips the prompt for scripted use. Both this and kexec-local self-elevate via sudo (require_root()) rather than requiring you to prefix the command yourself.
• terra runs Limine, not systemd-boot — bootctl set-oneshot is useless there (bootctl status lists ✗ One-shot entry control, and CachyOS's pacman hooks regenerate limine.conf anyway). local_install_prepare_and_reboot() therefore picks its one-shot mechanism at runtime: a systemd-boot loader entry when $BOOT/loader/entries exists, otherwise arm_efi_bootnext() — a temporary UEFI entry that EFI-stub-boots the kernel off the ESP, armed via the firmware's BootNext. Created with efibootmgr --create-only (NOT --create, which pushes it to the front of BootOrder and would make a wiped installer the permanent default if anything went wrong). BootNext is spent by that one boot, so a failed attempt still comes back on the normal bootloader. The EFI-stub path needs the kernel on the ESP itself, not on a separate XBOOTLDR — hence --print-esp-path rather than --print-boot-path in that mode. homelab-auto-install.service deletes the leftover NVRAM entry as soon as it boots; both it and the script match the label Homelab Installer EXACTLY (a prefix match would delete the Windows or Limine entry).
• installer-iso must force boot.initrd.systemd.enable = false. findiso= is a SCRIPT-stage-1 feature (stage-1-init.sh loop-mounts the file it points at and symlinks it to /dev/root). The systemd initrd — default since 26.05 — has NO findiso path: it mounts /iso straight from /dev/disk/by-label/<volumeID> (iso-image.nix), a label that only exists when the ISO is the physical boot medium. EFI-stub-booted off the ESP with the iso as a plain file on another fs, that label never appears; stage 1 mounts /sysroot fine, then times out on /sysroot/nix/.ro-store waiting for /dev/disk/by-label/nixos-minimal-… and drops to an emergency shell. The whole install <config> localhost findiso path depends on script stage 1.
• The staging-dir guard must fail CLOSED, and findmnt needs --nofsroot: on btrfs findmnt -no SOURCE prints /dev/sdb2[/@], which lsblk cannot open, so a naive parent-device lookup comes back empty. Treating empty as "different disk" silently allowed staging the iso on the very disk disko then wiped — terra's CachyOS root is exactly that layout, so it hit the live case. disks_backing() (lsblk -rnso NAME,TYPE) returns EVERY whole-disk ancestor because LVM/RAID can span several (/mnt/ssd_01 → sdd + sde), and an empty result is a hard error, not a pass. btrfs staging is refused outright: stage-1 mounts a btrfs volume's TOP level, so an iso inside a subvolume is unreachable via findiso=.
• findiso= must keep its leading slash: stage-1 tests [ -e "/findiso$isoPath" ], so stripping the mountpoint prefix off a stagedir whose mountpoint is / yields var/tmp/x.iso → /findisovar/tmp/x.iso → emergency shell, after you have already rebooted out of the working OS.
• The auto-install needs the host key shipped to it, and a $HOME: the ISO is built from a public repo with no credentials, so local_install_prepare_and_reboot() copies the key onto the boot partition and passes that partition's PARTUUID as homelab.keypart=; the service mounts it and drops the key in /root/.config/homelab/<config>/ before running the install. Also, systemd does NOT set $HOME for a system service without User= (SetLoginEnvironment= defaults to false), and scripts/deploy runs under set -u — hence $KEYDIR instead of a bare $HOME, plus Environment=HOME=/root on the unit.
• installer-iso must enable experimental-features itself. installation-cd-minimal leaves them unset, so nix run and nixos-install --flake both die with "experimental Nix feature 'nix-command' is disabled". The nixos-images kexec installer sets them itself, which is why the same install <config> localhost branch worked after kexec-local but not from the ISO.
• disko/nixos-anywhere run as nix run .#disko / .#nixos-anywhere, from this flake's locked inputs — not nix run github:.... They execute while a disk is being wiped, so the revision must be the reviewed one in flake.lock, and it has to resolve without network.
• disko wipes only the OS disk named in hosts/<h>/disk-config.nix; data disks are plain fileSystems in configuration.nix.
• nixos-anywhere/kexec needs a writable root; ZimaOS root is read-only, hence the ./scripts/deploy kexec step that streams a RAM installer (with static cpio/gzip since ZimaOS lacks them).
• kexec/run jumps ~6s AFTER it returns: nixos-images' kexec-run.sh ends with nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &. So the staging dir must OUTLIVE the script — an rm -rf in an EXIT trap deletes the binary that performs the jump and the box silently stays on the old kernel. kexec-local clears its trap before jumping and then sleeps 60s on purpose. Covered by checks.kexec-local.
• The kexec installer KEEPS the box's ssh host key: kexec-run.sh copies /etc/ssh/ssh_host_* into the appended initrd and restore-remote-access.nix installs them back. So do NOT ssh-keygen -R after a kexec — the key does not change, and clearing it just throws away the known_hosts record.
• kexec-local stages on /var/tmp, not /tmp: kexec-run.sh appends a fresh cpio to kexec/initrd in place and execs binaries from that dir, so a size-capped or noexec tmpfs gives a half-written initrd or a bare "Permission denied".