# homelab Flake-based NixOS config. Hosts: `jupiter` (ZimaBlade, NAS + services), `neptun` (netcup VPS: public reverse proxy, Authentik, headscale), `mercury` (Raspberry Pi 3B+, DNS/DHCP), `terra` (desktop), `mars` (on-site, single-purpose: Hermes Agent only). ## Structure ``` flake.nix # inputs + nixosConfigurations (jupiter, neptun, kexec, ...) common.nix # shared base: user, ssh, nix, firewall, timezone services/ # one reusable module per service, by category media/ jellyfin, audiobookshelf, the *arrs, sabnzbd, seerr, ... network/ caddy, samba, avahi, pihole, unbound vpn/ tailscale, headscale (control server), headplane (its web UI) identity/ authentik (OIDC provider, from the authentik-nix flake) dev/ gitea desktop/ hyprland containers.nix # podman backend, shared across categories hosts/ jupiter/ # ZimaBlade NAS configuration.nix # host bits + imports common + the services it runs disk-config.nix # disko: eMMC partitions hardware-configuration.nix secrets.nix # sops-nix wiring vm.nix # VirtualBox test image (jupiter-vbox) neptun/ # netcup public reverse proxy + tailnet node configuration.nix disk-config.nix hardware-configuration.nix secrets.nix mars/ # on-site, single-purpose: Hermes Agent only configuration.nix disk-config.nix hardware-configuration.nix secrets.nix hermes-agent.nix # Hermes Agent (moved here from jupiter) secrets/ # age-encrypted sops files, one per host scripts/ # deploy, edit_secrets ``` Hosts compose by importing `common.nix` + whichever `services/*` modules they run. Each service module opens its own firewall ports. ## Gitea event relay Mars includes a small HMAC-validating relay for Gitea webhooks. It forwards the authenticated request body and Gitea's own signature to Hermes over localhost completely unchanged, and copies `X-Gitea-Event` into `X-GitHub-Event`. It has no event, repository, action, payload, or prompt policy; Hermes owns interpretation and response behavior. Jupiter's Gitea provisioning service registers the webhook idempotently at `http://mars.orbit.sol:8645/gitea/gitea-pr-comments`. The path after `/gitea/` names the Hermes route to forward into, so the relay is not tied to any one subscription: another Hermes route needs a `hermes webhook subscribe ` and a Gitea hook pointing at `/gitea/`, and no relay change. Route names are validated against a strict charset before being used in the outbound URL. Neither provisioning unit deletes anything: Jupiter's only creates or updates its own hook, and Mars's only removes the route it is about to re-subscribe. Retiring the pre-rename `gitea-events` route is therefore a one-off, done by hand after the first deploy of both hosts: ``` # on mars — drop the old subscription sudo podman exec hermes-agent hermes webhook remove gitea-events # on jupiter — delete the old hook (it posts to the relay's bare /gitea path) api=http://127.0.0.1:3000/api/v1; repo=darman/homelab tok="$(sudo cat /run/secrets/gitea_provisioning_token)" old="$(curl -fsS -H "Authorization: token $tok" "$api/repos/$repo/hooks" \ | jq -r '.[] | select(.config.url == "http://mars.orbit.sol:8645/gitea") | .id')" for id in $old; do curl -fsS -H "Authorization: token $tok" -X DELETE "$api/repos/$repo/hooks/$id" done ``` Check `hermes webhook list` and the repo's webhook page afterwards; until the old hook is gone both it and the new one fire, so events arrive twice. That one header copy is the entire reason the relay exists. Gitea signs every webhook with `X-Hub-Signature-256` in GitHub's exact format, which Hermes already accepts on any route — so authentication would work pointing Gitea straight at Hermes on 8644. But Hermes reads the event name only from `X-GitHub-Event`/`X-GitLab-Event` (then `event_type`/`type` in the payload, then the literal `"unknown"`), and Gitea sends none of those. Without the copy every delivery arrives as `unknown` and `hermes webhook subscribe --events ...` can never match anything. Run `python3 services/dev/gitea-hermes-webhook-relay-test.py` to exercise the relay end to end (signature acceptance and rejection, byte-identical body forwarding, and the event-header copy). Before deploying either host, add the same random `gitea_hermes_webhook_secret` value to both `secrets/mars.yaml` and `secrets/jupiter.yaml` using `scripts/edit_secrets`, with no trailing newline — the value reaches Hermes through an env-file template, where a newline both corrupts the file and changes the key the HMAC is computed with. The value is intentionally not included in the repository. ## Test in VirtualBox (no hardware needed) ``` nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA VBoxManage import result/*.ova --vsys 0 --vmname jupiter-vbox VBoxManage startvm jupiter-vbox --type headless ``` Login `darman` / `test`. Forward ports with `VBoxManage modifyvm ... --natpf1`. ## First install on the ZimaBlade — nixos-anywhere + disko Wipes the OS disk and installs the flake over SSH. No USB needed if the box already runs Linux (ZimaOS) reachable by root SSH — nixos-anywhere kexecs into an installer, partitions via disko, installs. > ⚠️ The OS disk in `disk-config.nix` is WIPED. Set `device` to the OS disk > ONLY (by-id). Back up / physically identify the NAS data disk first — it must > NOT appear in disko. `lsblk -o NAME,SERIAL,SIZE,MODEL` to identify. 1. Set the real OS disk id in `hosts/jupiter/disk-config.nix` (`ls -l /dev/disk/by-id`), and the data-disk mount in `configuration.nix`. 2. Add your login SSH pubkey to `users.users.darman.openssh.authorizedKeys.keys`. 3. Set the real samba password: ``` export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt nix shell nixpkgs#sops -c sops secrets/jupiter.yaml # edit, commit ``` 4. Stage the pre-generated host key so sops can decrypt on boot #1 (private key lives off-repo in `~/.config/homelab/jupiter/`): ``` install -Dm600 ~/.config/homelab/jupiter/ssh_host_ed25519_key \ /tmp/extra/etc/ssh/ssh_host_ed25519_key install -Dm644 ~/.config/homelab/jupiter/ssh_host_ed25519_key.pub \ /tmp/extra/etc/ssh/ssh_host_ed25519_key.pub ``` 5. Run from your laptop: ``` nix run github:nix-community/nixos-anywhere -- \ --flake .#jupiter \ --extra-files /tmp/extra \ --generate-hardware-config nixos-generate-config ./hosts/jupiter/hardware-configuration.nix \ --target-host root@ ``` `--extra-files` plants the host key before first boot (its age identity is already a recipient in `.sops.yaml`, so `/run/secrets/samba_password` decrypts on boot #1). `--generate-hardware-config` pulls the target's real kernel modules into the placeholder. Commit the result. Reboot into NixOS. Manual alternative (USB ISO): boot installer, `disko` the disk, then `nixos-install --flake .#jupiter`. ## First install on mars mars is an older x86_64 box (unknown provenance, "got from work"), on-site, running Hermes Agent only (see `hosts/mars/hermes-agent.nix` — moved there from jupiter). Its age recipient, host key (`~/.config/homelab/mars/ssh_host_ed25519_key`), and `secrets/mars.yaml` are already set up, with `darman_password`/`samba_password`/`opencode_go_api_key`/ `telegram_bot_token`/`hermes_dashboard_oidc_client_secret` carried over from jupiter's old instance. Two things are still placeholders and MUST be filled in before installing: 1. **OS disk id** in `hosts/mars/disk-config.nix` (`ls -l /dev/disk/by-id` once you have console/installer access on the box) — same `REPLACE-ME` in `hosts/mars/configuration.nix`'s comment refers to the same disk, but only `disk-config.nix`'s `device` actually needs editing (grub's own device list comes from disko, see that file's comment). 2. **`tailscale_authkey`** in `secrets/mars.yaml` — generate a fresh one (see "Bootstrap the tailnet" under neptun below) rather than reusing an old key; reusable pre-auth keys still expire. Boot mode is assumed **legacy BIOS** (grub, not systemd-boot) — unconfirmed; check `[ -d /sys/firmware/efi ]` once you're at the machine and see `hosts/mars/disk-config.nix`'s header comment if it turns out to be UEFI. Otherwise the flow is identical to the ZimaBlade steps above: ``` nix run github:nix-community/nixos-anywhere -- \ --flake .#mars \ --extra-files /tmp/extra \ --generate-hardware-config nixos-generate-config ./hosts/mars/hardware-configuration.nix \ --target-host root@ ``` (stage the host key into `/tmp/extra/etc/ssh/` first, same as step 4 there). Manual alternative (USB ISO): boot installer, `disko` the disk, then `nixos-install --flake .#mars`. ## First install on terra — no-USB findiso reinstall (replacing CachyOS) terra is a Ryzen 9 5900X / Radeon RX 6800 XT desktop, currently running CachyOS with a writable root and **Limine** as its bootloader (not systemd-boot — see step 4). Everything is already prepped in this repo: real OS-disk id in `disk-config.nix`, real login pubkey in `common.nix`, terra's age recipient in `.sops.yaml`, its host key pre-generated at `~/.config/homelab/terra/`, and `secrets/terra.yaml` already holds real `darman_password` / `tailscale_authkey` values. Nothing to fill in — just run it. > ⚠️ **`./scripts/deploy kexec-local` does NOT work on terra — do not use it.** > Confirmed on real hardware: the jump hangs completely (kexec's own > `device_shutdown()` pass runs — SCSI disks sync fine — then the machine goes > dark and never comes back; `journalctl --list-boots` showed a **~15 minute** > gap before the next boot, i.e. a hard hang needing a manual power cycle, not > a slow jump). Near-certainly amdgpu: discrete AMD GPUs are known to hang > during kexec's device-shutdown pass with no clean way to hand control back > before the jump — same class of issue as jupiter's `reboot=pci` warm-reboot > workaround, just fatal here instead of merely slow. The path below instead > triggers a real ACPI reboot through firmware POST — a materially different > code path that never runs kexec's device-shutdown pass at all. > ⚠️ The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The > dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs > disks are not in disko and are untouched — but double check with > `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have > changed since `disk-config.nix` was written. One command does the whole thing — no need to `sudo` it yourself, it self-elevates: ``` ./scripts/deploy install terra localhost ``` `scripts/deploy` detects it isn't already inside a live installer (checks `uname -n`) and instead: 1. Asks where to stage the iso file — never auto-picks, because the wrong disk here is destroyed mid-install (`HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for scripted use). It **refuses** if that path resolves to a disk `disk-config.nix` is about to wipe, if it can't work out which physical disk the path is on at all (fail-closed — LVM and RAID can span several), or if it's on btrfs (stage-1 mounts a btrfs volume's *top level*, so a path inside a subvolume never resolves and you boot to an emergency shell). On terra, `/mnt/hdd_01` is the right answer; the CachyOS root is btrfs on the OS disk and is rejected on both counts. 2. Prints exactly what it is about to do — OS disk, staging disk, boot entry — and waits for you to type `yes`. `--yes` skips it; that is what the ISO passes when it re-runs the command itself. 3. Builds `installer-iso`'s kernel + initrd + iso image, checks both target partitions have room, then copies the kernel/initrd **and terra's pre-generated ssh host key** to the boot partition (found via `bootctl --print-boot-path`, not assumed to be `/boot`) and the iso to the staging dir. 4. Arms a **one-shot** boot of it with `findiso=` + `homelab.install=terra` + `homelab.keypart=` on the kernel cmdline, and reboots — a real `systemctl reboot`, not kexec. Two mechanisms, picked automatically: - **systemd-boot** (jupiter, neptun, and terra once NixOS is on it): a `bootctl set-oneshot` loader entry. - **anything else** — terra today runs Limine, which reports `One-shot entry control: ✗` and has no equivalent: the firmware's own **`BootNext`**, pointing at a temporary UEFI entry that EFI-stub-boots the kernel straight off the ESP. Created with `--create-only` so it never enters `BootOrder`, which means it is reachable exactly once and nothing else changes. Either way the box falls back to its normal bootloader if the attempt fails — nothing is made permanent before the install succeeds. The temporary UEFI entry is deleted by the installer as soon as it boots. The booted installer clones the repo (`homelab-checkout.service`, needs network — it's public now, no credentials involved) and then `homelab-auto-install.service` reads `homelab.install=terra` back off `/proc/cmdline`, mounts `homelab.keypart=` to pick terra's host key back up into `/root/.config/homelab/terra/`, and re-runs the exact same `./scripts/deploy install terra localhost` itself — now genuinely inside the installer, so it takes the disko + `nixos-install` branch instead of preparing again. That key is what seeds `/etc/ssh` on the new system, which is what lets `/run/secrets/*` decrypt on boot #1; it has to travel this way because the ISO is built from a **public** repo and deliberately carries no credentials. The copy on the boot partition dies with the disko wipe minutes later. The whole thing is unattended after the initial reboot; ssh into `homelab-installer` (same pubkey as the ISO everywhere else) to watch progress — `journalctl -u homelab-checkout -u homelab-auto-install -f`. > The checkout step resolves `git.mgaction.town`, which goes through mercury's > pihole on the LAN. If mercury is down, the installer boots fine but never > gets the repo — fix DNS and `systemctl restart homelab-checkout`. When it's done, reboot again into the freshly installed NixOS. Then, same as any other host: ``` ssh darman@terra sudo -v # DO NOT SKIP — see below ``` `darman` is created with `mutableUsers = true`, so `/etc/shadow` is written **once**. If the sops secret wasn't readable at that moment the account gets `!` (locked) permanently and no `deploy switch` will fix it — verify sudo works while you still have physical console access as a fallback. ## Deploy (the `./deploy` wrapper) All arguments mandatory — no default host, no default config. ``` ./deploy kexec # headless kexec into a RAM installer (RO-root box) ./deploy install [--yes] # first install; wipes OS disk, ships host key ./deploy switch # rebuild + activate on a running host ./deploy boot|test # stage for next boot / activate without boot entry ./deploy image # build an SD-card image (mercury) ./deploy flash # build SD image, write it, drop the sops age key ``` `switch`/`boot`/`test` prompt for darman's password (`wheelNeedsPassword`). `` is a `nixosConfigurations` name (`jupiter`, `neptun`). Its pre-generated SSH host key lives at `~/.config/homelab//ssh_host_ed25519_key`. Examples: ``` ./deploy switch jupiter jupiter.sol ./deploy install neptun 159.195.64.117 ``` Rollback: `nixos-rebuild switch --rollback` on the host, or pick a prior generation at boot. ## Post-deploy steps (per host) Things the flake cannot do for you. Skipping these leaves a host that builds and boots but doesn't work. ### Every host, immediately after a first install ``` ssh darman@ sudo -v # DO NOT SKIP ``` `users.mutableUsers` is `true`, so `/etc/shadow` is written **once**, when the user is created. If the sops secret wasn't readable at that moment the account gets `!` (locked) permanently — `deploy switch` will never fix it, because the activation script only sets a password for users not already in `/etc/shadow`. Combined with `wheelNeedsPassword = true` and `PermitRootLogin = "no"` that means no way to escalate, and recovery is physical: netcup's rescue system for neptun, or pulling the SD card for mercury. Verify sudo while you still have another way in. ### neptun (netcup VPS) 1. **Edge firewall.** In netcup's panel, inbound `ACCEPT` for TCP 22/80/443/2222 **and a rule accepting inbound UDP**. The firewall is stateless: without the UDP rule every DNS and NTP *reply* is dropped, and nothing on the box reports an error — it looks like headscale crash-looping on its DERP fetch and Caddy failing ACME. `grep -A1 '^Udp:' /proc/net/snmp` showing `InDatagrams 0` is the tell. Rules apply on VM restart, not on save. This is safe: `nixos-fw` is stateful and default-deny, so it remains the real policy. Also open UDP 3478 (STUN) and 41641 (tailscale direct). 2. **Authentik** creates `akadmin` on first start; log in at `https://auth.mgaction.town` with `authentik_bootstrap_password` from sops. The username is hardcoded upstream and the bootstrap runs once — later changes to the env vars are ignored. To use your own admin instead: create a user, add it to the **`authentik Admins`** group (superuser is a *group* flag in Authentik, there is no per-user one), verify it works in a private window, then **deactivate** `akadmin` — do not rename or delete it. The bootstrap blueprint keys on `username: akadmin` with `state: created`, so if no user by that name exists it simply makes a new one on the next reconcile. 3. **Bootstrap the tailnet** (headscale starts with an empty database): ``` sudo headscale users create darman sudo headscale preauthkeys create --user darman --reusable --expiration 24h ``` Put that key in **every** host's sops file as `tailscale_authkey` and rebuild. 4. **Headplane API key** — defaults to 90d, after which headplane silently stops listing nodes: ``` sudo headscale apikeys create --expiration 999d # -> headplane_headscale_api_key ``` 5. **Headplane OIDC.** In Authentik create an OAuth2/OpenID provider (confidential, redirect `https://vpn.mgaction.town/admin/oidc/callback`, **a signing key must be selected** or discovery exposes no JWKS) and an application with slug **`headplane`** — the slug is what makes the issuer `.../application/o/headplane/` in `services/vpn/headplane.nix`. Client ID goes in that file, client secret into sops. 6. **Headscale OIDC** (optional — pre-auth keys work without it). A *second* Authentik provider/application, slug **`headscale`**, redirect `https://vpn.mgaction.town/oidc/callback` (headscale's own, not headplane's under `/admin`). Client ID in `services/vpn/headscale.nix`, secret into sops as `headscale_oidc_client_secret`. ⚠️ headscale runs OIDC discovery **at startup and a failure is fatal** — an issuer pointing at an application that doesn't exist yet means the control server won't boot, taking the whole tailnet's control plane with it. Always verify first: ``` curl -s https://auth.mgaction.town/application/o/headscale/.well-known/openid-configuration ``` Users created by OIDC login are distinct from `headscale users create` ones: headplane matches the OIDC `sub` claim against the user's `providerId`, CLI-made users have none, and 0.28 dropped both `map_legacy_users` and node reassignment — so moving an existing node to an OIDC user means re-enrolling it. ### jupiter - `chown -R gitea:gitea /mnt/data/AppData/gitea` after the first deploy (the repos were copied in over CIFS as `darman:users`). - **Re-enrolling after the headscale database was recreated:** `tailscaled` keeps its old node key and reports `Running`, and the autoconnect unit exits early on that state without ever sending the new pre-auth key. Force it: ``` sudo tailscale logout && sudo systemctl restart tailscaled-autoconnect ``` ### mars - **Confirm the OIDC redirect still resolves.** hermes-agent.nix reuses jupiter's old Authentik application (slug `hermes`, redirect `https://hermes.mgaction.town/auth/callback`) unchanged — nothing to reconfigure in Authentik, just verify `neptun`'s `hermes.mgaction.town` vhost (now pointed at `mars.orbit.sol:9119`) actually reaches the dashboard once mars is up and joined the tailnet. - **Carrying forward old chat history/memories:** mars starts with a fresh Hermes state dir (`/var/lib/hermes/.hermes`). jupiter's old instance data is backed up at `/mnt/data/AppData/hermes.bak-2026-08-21` — rsync it over (via the `/mnt/jupiter` samba mount) before the first switch if you want it preserved instead of starting clean. ### mercury (Raspberry Pi 3B+) - `./deploy flash mercury /dev/sdX` writes the dedicated age key to the root partition. Without `~/.config/homelab/mercury/age.txt` it silently skips that step and **no secret decrypts on the box** — check `ls /run/secrets` after first boot. - It boots from an SD card, so config changes are `./deploy switch mercury ` (an aarch64 build — needs `extra-platforms` + binfmt on the laptop, see the gotchas in `CLAUDE.md`) rather than a reflash. - **Suspect the card first** when binaries crash with `Illegal instruction` or services fail inexplicably. Failing flash returns corrupt data with no I/O errors in `dmesg`: ``` sudo nix-store --verify --check-contents # add --repair to fix ``` A card that has corrupted one path will corrupt more. Replace it and reflash. - **A reflash wipes `/var/lib/pihole`**, taking the gravity database with it. The blocklists themselves are declared in `services/network/pihole.nix`, and the `pihole-adlists` unit re-seeds them on boot and rebuilds gravity when it finds it empty — so this heals itself, but the first boot after a reflash spends several minutes downloading lists. Query history and dynamic DHCP leases are genuinely lost (static leases are declarative). Check with: ``` systemctl status pihole-adlists sudo podman exec pihole pihole-FTL sqlite3 /etc/pihole/gravity.db \ "SELECT address,enabled FROM adlist; SELECT COUNT(*) FROM gravity;" ``` `Blocked DNS queries: 0` in the pihole logs means gravity is empty — DNS resolves fine, nothing is filtered. - mercury's own `resolv.conf` is deliberately public resolvers, not its own pihole (`resolveLocalQueries = false`, see `CLAUDE.md`) — so `.sol` names do not resolve *on mercury itself*. That is expected, not a fault. - **mercury is load-bearing for the whole tailnet's DNS.** headscale sets `override_local_dns = true` with pihole as the only global nameserver, so every node — including a phone on mobile data — resolves through it and gets ad blocking and `.sol` names anywhere. The flip side is that mercury (or the home connection) going down costs name resolution on every device, not just `.sol`. Recovery on a stranded device is turning Tailscale off. There is deliberately no public fallback in `nameservers.global`: tailscale treats that list as a set, so a second entry would let queries slip past the filter whenever mercury is slow. neptun and mercury opt out with `--accept-dns=false` — mercury because it would otherwise resolve through itself, neptun because a public reverse proxy must not depend on a Pi at home to renew its certificates. - Tailnet names are `*.orbit.sol`, LAN names are `*.sol`. Both work everywhere on the tailnet because tailscale matches DNS routes by **longest suffix**, so `orbit.sol` reaches MagicDNS even though everything else goes to pihole. Never name a LAN host `orbit`: pihole's `address=/.sol/` lines match a name *and everything beneath it*, which would swallow the entire tailnet zone. ## Adding a service Copy the `whoami` block in `oci-containers.containers`, swap image/ports/volumes. Native NixOS module exists for many apps (Nextcloud, Jellyfin, Grafana...) — prefer `services.` over a container when available. Add a `caddy` `virtualHosts` block to expose it. ## Notes - Backend is Podman with `dockerCompat` — `docker` CLI works, no daemon. - Samba keeps its own password DB. `services.samba` never sets it; a systemd oneshot (`samba-smbpasswd`) provisions it. Host reads the password from `/run/secrets/samba_password` (**sops-nix**); the VM falls back to plaintext `/etc/samba/smb-password`. - Secrets: `secrets/jupiter.yaml` is age-encrypted (safe to commit) to two recipients in `.sops.yaml` — the **admin** key (edit on laptop, `~/.config/sops/age/keys.txt`) and the **jupiter host** key (derived from its SSH host key via `ssh-to-age`, decrypts at runtime). Private keys live off-repo and are gitignored. Rotate/add recipients with `sops updatekeys`. - Data disk: plain `fileSystems."/mnt/data"` in configuration.nix — kept out of disko so it is never formatted. Reference by `by-id` / `by-uuid`. - `system.stateVersion` = `26.05`, install-time schema. Do NOT bump on upgrades. - Terraform is not used: a single bare-metal box has no provider API. disko + nixos-anywhere cover provisioning natively.