homelab/README.md

# homelab

Flake-based NixOS config. Hosts: `jupiter` (ZimaBlade, NAS + services),
`neptun` (netcup VPS: public reverse proxy, Authentik, headscale),
`mercury` (Raspberry Pi 3B+, DNS/DHCP), `terra` (desktop), `mars` (on-site,
single-purpose: Hermes Agent only).

## Structure

```
flake.nix                  # inputs + nixosConfigurations (jupiter, neptun, kexec, ...)
common.nix                 # shared base: user, ssh, nix, firewall, timezone
services/                  # one reusable module per service, by category
  media/     jellyfin, audiobookshelf, the *arrs, sabnzbd, seerr, ...
  network/   caddy, samba, avahi, pihole, unbound
  vpn/       tailscale, headscale (control server), headplane (its web UI)
  identity/  authentik (OIDC provider, from the authentik-nix flake)
  dev/       gitea
  desktop/   hyprland
  containers.nix           # podman backend, shared across categories
hosts/
  jupiter/                 # ZimaBlade NAS
    configuration.nix      #   host bits + imports common + the services it runs
    disk-config.nix        #   disko: eMMC partitions
    hardware-configuration.nix
    secrets.nix            #   sops-nix wiring
    vm.nix                 #   VirtualBox test image (jupiter-vbox)
  neptun/                     # netcup public reverse proxy + tailnet node
    configuration.nix  disk-config.nix  hardware-configuration.nix  secrets.nix
  mars/                     # on-site, single-purpose: Hermes Agent only
    configuration.nix  disk-config.nix  hardware-configuration.nix  secrets.nix
    hermes-agent.nix          #   Hermes Agent (moved here from jupiter)
secrets/                   # age-encrypted sops files, one per host
scripts/                   # deploy, edit_secrets
```

Hosts compose by importing `common.nix` + whichever `services/*` modules they
run. Each service module opens its own firewall ports.

## Gitea event relay

Mars includes a small HMAC-validating relay for Gitea webhooks. It forwards the
authenticated request body and Gitea's own signature to Hermes over localhost
completely unchanged, and copies `X-Gitea-Event` into `X-GitHub-Event`. It has
no event, repository, action, payload, or prompt policy; Hermes owns
interpretation and response behavior. Jupiter's Gitea provisioning service
registers the webhook idempotently at `http://mars.orbit.sol:8645/gitea`.

That one header copy is the entire reason the relay exists. Gitea signs every
webhook with `X-Hub-Signature-256` in GitHub's exact format, which Hermes
already accepts on any route — so authentication would work pointing Gitea
straight at Hermes on 8644. But Hermes reads the event name only from
`X-GitHub-Event`/`X-GitLab-Event` (then `event_type`/`type` in the payload,
then the literal `"unknown"`), and Gitea sends none of those. Without the copy
every delivery arrives as `unknown` and `hermes webhook subscribe --events ...`
can never match anything.

Run `python3 services/dev/gitea-hermes-webhook-relay-test.py` to exercise the
relay end to end (signature acceptance and rejection, byte-identical body
forwarding, and the event-header copy).

Before deploying either host, add the same random
`gitea_hermes_webhook_secret` value to both `secrets/mars.yaml` and
`secrets/jupiter.yaml` using `scripts/edit_secrets`, with no trailing newline
— the value reaches Hermes through an env-file template, where a newline both
corrupts the file and changes the key the HMAC is computed with. The value is
intentionally not included in the repository.

## Test in VirtualBox (no hardware needed)

```
nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
VBoxManage import result/*.ova --vsys 0 --vmname jupiter-vbox
VBoxManage startvm jupiter-vbox --type headless
```
Login `darman` / `test`. Forward ports with `VBoxManage modifyvm ... --natpf1`.

## First install on the ZimaBlade — nixos-anywhere + disko

Wipes the OS disk and installs the flake over SSH. No USB needed if the box
already runs Linux (ZimaOS) reachable by root SSH — nixos-anywhere kexecs into
an installer, partitions via disko, installs.

> ⚠️  The OS disk in `disk-config.nix` is WIPED. Set `device` to the OS disk
> ONLY (by-id). Back up / physically identify the NAS data disk first — it must
> NOT appear in disko. `lsblk -o NAME,SERIAL,SIZE,MODEL` to identify.

1. Set the real OS disk id in `hosts/jupiter/disk-config.nix`
   (`ls -l /dev/disk/by-id`), and the data-disk mount in `configuration.nix`.
2. Add your login SSH pubkey to `users.users.darman.openssh.authorizedKeys.keys`.
3. Set the real samba password:
   ```
   export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt
   nix shell nixpkgs#sops -c sops secrets/jupiter.yaml   # edit, commit
   ```
4. Stage the pre-generated host key so sops can decrypt on boot #1
   (private key lives off-repo in `~/.config/homelab/jupiter/`):
   ```
   install -Dm600 ~/.config/homelab/jupiter/ssh_host_ed25519_key \
     /tmp/extra/etc/ssh/ssh_host_ed25519_key
   install -Dm644 ~/.config/homelab/jupiter/ssh_host_ed25519_key.pub \
     /tmp/extra/etc/ssh/ssh_host_ed25519_key.pub
   ```
5. Run from your laptop:
   ```
   nix run github:nix-community/nixos-anywhere -- \
     --flake .#jupiter \
     --extra-files /tmp/extra \
     --generate-hardware-config nixos-generate-config ./hosts/jupiter/hardware-configuration.nix \
     --target-host root@<zimablade-ip>
   ```
   `--extra-files` plants the host key before first boot (its age identity is
   already a recipient in `.sops.yaml`, so `/run/secrets/samba_password`
   decrypts on boot #1). `--generate-hardware-config` pulls the target's real
   kernel modules into the placeholder. Commit the result. Reboot into NixOS.

Manual alternative (USB ISO): boot installer, `disko` the disk, then
`nixos-install --flake .#jupiter`.

## First install on mars

mars is an older x86_64 box (unknown provenance, "got from work"), on-site,
running Hermes Agent only (see `hosts/mars/hermes-agent.nix` — moved there
from jupiter). Its age recipient, host key
(`~/.config/homelab/mars/ssh_host_ed25519_key`), and `secrets/mars.yaml` are
already set up, with `darman_password`/`samba_password`/`opencode_go_api_key`/
`telegram_bot_token`/`hermes_dashboard_oidc_client_secret` carried over from
jupiter's old instance. Two things are still placeholders and MUST be filled
in before installing:

1. **OS disk id** in `hosts/mars/disk-config.nix` (`ls -l /dev/disk/by-id`
   once you have console/installer access on the box) — same `REPLACE-ME` in
   `hosts/mars/configuration.nix`'s comment refers to the same disk, but only
   `disk-config.nix`'s `device` actually needs editing (grub's own device list
   comes from disko, see that file's comment).
2. **`tailscale_authkey`** in `secrets/mars.yaml` — generate a fresh one
   (see "Bootstrap the tailnet" under neptun below) rather than reusing an
   old key; reusable pre-auth keys still expire.

Boot mode is assumed **legacy BIOS** (grub, not systemd-boot) — unconfirmed;
check `[ -d /sys/firmware/efi ]` once you're at the machine and see
`hosts/mars/disk-config.nix`'s header comment if it turns out to be UEFI.

Otherwise the flow is identical to the ZimaBlade steps above:
```
nix run github:nix-community/nixos-anywhere -- \
  --flake .#mars \
  --extra-files /tmp/extra \
  --generate-hardware-config nixos-generate-config ./hosts/mars/hardware-configuration.nix \
  --target-host root@<mars-ip>
```
(stage the host key into `/tmp/extra/etc/ssh/` first, same as step 4 there).
Manual alternative (USB ISO): boot installer, `disko` the disk, then
`nixos-install --flake .#mars`.

## First install on terra — no-USB findiso reinstall (replacing CachyOS)

terra is a Ryzen 9 5900X / Radeon RX 6800 XT desktop, currently running
CachyOS with a writable root and **Limine** as its bootloader (not
systemd-boot — see step 4). Everything is already prepped
in this repo: real OS-disk id in `disk-config.nix`, real login pubkey in
`common.nix`, terra's age recipient in `.sops.yaml`, its host key
pre-generated at `~/.config/homelab/terra/`, and `secrets/terra.yaml` already
holds real `darman_password` / `tailscale_authkey` values. Nothing to fill
in — just run it.

> ⚠️  **`./scripts/deploy kexec-local` does NOT work on terra — do not use it.**
> Confirmed on real hardware: the jump hangs completely (kexec's own
> `device_shutdown()` pass runs — SCSI disks sync fine — then the machine goes
> dark and never comes back; `journalctl --list-boots` showed a **~15 minute**
> gap before the next boot, i.e. a hard hang needing a manual power cycle, not
> a slow jump). Near-certainly amdgpu: discrete AMD GPUs are known to hang
> during kexec's device-shutdown pass with no clean way to hand control back
> before the jump — same class of issue as jupiter's `reboot=pci` warm-reboot
> workaround, just fatal here instead of merely slow. The path below instead
> triggers a real ACPI reboot through firmware POST — a materially different
> code path that never runs kexec's device-shutdown pass at all.

> ⚠️  The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The
> dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs
> disks are not in disko and are untouched — but double check with
> `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have
> changed since `disk-config.nix` was written.

One command does the whole thing — no need to `sudo` it yourself, it
self-elevates:
```
./scripts/deploy install terra localhost
```
`scripts/deploy` detects it isn't already inside a live installer (checks
`uname -n`) and instead:

1. Asks where to stage the iso file — never auto-picks, because the wrong disk
   here is destroyed mid-install (`HOMELAB_INSTALLER_STAGE_DIR` skips the
   prompt for scripted use). It **refuses** if that path resolves to a disk
   `disk-config.nix` is about to wipe, if it can't work out which physical disk
   the path is on at all (fail-closed — LVM and RAID can span several), or if
   it's on btrfs (stage-1 mounts a btrfs volume's *top level*, so a path inside
   a subvolume never resolves and you boot to an emergency shell). On terra,
   `/mnt/hdd_01` is the right answer; the CachyOS root is btrfs on the OS disk
   and is rejected on both counts.
2. Prints exactly what it is about to do — OS disk, staging disk, boot entry —
   and waits for you to type `yes`. `--yes` skips it; that is what the ISO
   passes when it re-runs the command itself.
3. Builds `installer-iso`'s kernel + initrd + iso image, checks both target
   partitions have room, then copies the kernel/initrd **and terra's
   pre-generated ssh host key** to the boot partition (found via
   `bootctl --print-boot-path`, not assumed to be `/boot`) and the iso to the
   staging dir.
4. Arms a **one-shot** boot of it with `findiso=` + `homelab.install=terra` +
   `homelab.keypart=<PARTUUID>` on the kernel cmdline, and reboots — a real
   `systemctl reboot`, not kexec. Two mechanisms, picked automatically:
   - **systemd-boot** (jupiter, neptun, and terra once NixOS is on it): a
     `bootctl set-oneshot` loader entry.
   - **anything else** — terra today runs Limine, which reports `One-shot entry
     control: ✗` and has no equivalent: the firmware's own **`BootNext`**,
     pointing at a temporary UEFI entry that EFI-stub-boots the kernel straight
     off the ESP. Created with `--create-only` so it never enters `BootOrder`,
     which means it is reachable exactly once and nothing else changes.

   Either way the box falls back to its normal bootloader if the attempt
   fails — nothing is made permanent before the install succeeds. The
   temporary UEFI entry is deleted by the installer as soon as it boots.

The booted installer clones the repo (`homelab-checkout.service`, needs
network — it's public now, no credentials involved) and then
`homelab-auto-install.service` reads `homelab.install=terra` back off
`/proc/cmdline`, mounts `homelab.keypart=` to pick terra's host key back up
into `/root/.config/homelab/terra/`, and re-runs the exact same
`./scripts/deploy install terra localhost` itself — now genuinely inside the
installer, so it takes the disko + `nixos-install` branch instead of preparing
again. That key is what seeds `/etc/ssh` on the new system, which is what lets
`/run/secrets/*` decrypt on boot #1; it has to travel this way because the ISO
is built from a **public** repo and deliberately carries no credentials. The
copy on the boot partition dies with the disko wipe minutes later.

The whole thing is unattended after the initial reboot; ssh into
`homelab-installer` (same pubkey as the ISO everywhere else) to watch
progress — `journalctl -u homelab-checkout -u homelab-auto-install -f`.

> The checkout step resolves `git.mgaction.town`, which goes through mercury's
> pihole on the LAN. If mercury is down, the installer boots fine but never
> gets the repo — fix DNS and `systemctl restart homelab-checkout`.

When it's done, reboot again into the freshly installed NixOS. Then, same as
any other host:
```
ssh darman@terra sudo -v          # DO NOT SKIP — see below
```

`darman` is created with `mutableUsers = true`, so `/etc/shadow` is written
**once**. If the sops secret wasn't readable at that moment the account gets
`!` (locked) permanently and no `deploy switch` will fix it — verify sudo
works while you still have physical console access as a fallback.

## Deploy (the `./deploy` wrapper)

All arguments mandatory — no default host, no default config.

```
./deploy kexec   <config> <host>   # headless kexec into a RAM installer (RO-root box)
./deploy install <config> <host> [--yes]   # first install; wipes OS disk, ships host key
./deploy switch  <config> <host>   # rebuild + activate on a running host
./deploy boot|test <config> <host> # stage for next boot / activate without boot entry
./deploy image   <config>          # build an SD-card image (mercury)
./deploy flash   <config> <dev>    # build SD image, write it, drop the sops age key
```
`switch`/`boot`/`test` prompt for darman's password (`wheelNeedsPassword`).
`<config>` is a `nixosConfigurations` name (`jupiter`, `neptun`). Its pre-generated
SSH host key lives at `~/.config/homelab/<config>/ssh_host_ed25519_key`.

Examples:
```
./deploy switch jupiter jupiter.sol
./deploy install neptun 159.195.64.117
```
Rollback: `nixos-rebuild switch --rollback` on the host, or pick a prior
generation at boot.

## Post-deploy steps (per host)

Things the flake cannot do for you. Skipping these leaves a host that builds
and boots but doesn't work.

### Every host, immediately after a first install

```
ssh darman@<host> sudo -v          # DO NOT SKIP
```

`users.mutableUsers` is `true`, so `/etc/shadow` is written **once**, when the
user is created. If the sops secret wasn't readable at that moment the account
gets `!` (locked) permanently — `deploy switch` will never fix it, because the
activation script only sets a password for users not already in `/etc/shadow`.
Combined with `wheelNeedsPassword = true` and `PermitRootLogin = "no"` that
means no way to escalate, and recovery is physical: netcup's rescue system for
neptun, or pulling the SD card for mercury. Verify sudo while you still have
another way in.

### neptun (netcup VPS)

1. **Edge firewall.** In netcup's panel, inbound `ACCEPT` for TCP 22/80/443/2222
   **and a rule accepting inbound UDP**. The firewall is stateless: without the
   UDP rule every DNS and NTP *reply* is dropped, and nothing on the box reports
   an error — it looks like headscale crash-looping on its DERP fetch and Caddy
   failing ACME. `grep -A1 '^Udp:' /proc/net/snmp` showing `InDatagrams 0` is the
   tell. Rules apply on VM restart, not on save. This is safe: `nixos-fw` is
   stateful and default-deny, so it remains the real policy.
   Also open UDP 3478 (STUN) and 41641 (tailscale direct).
2. **Authentik** creates `akadmin` on first start; log in at
   `https://auth.mgaction.town` with `authentik_bootstrap_password` from sops.
   The username is hardcoded upstream and the bootstrap runs once — later
   changes to the env vars are ignored.
   To use your own admin instead: create a user, add it to the **`authentik
   Admins`** group (superuser is a *group* flag in Authentik, there is no
   per-user one), verify it works in a private window, then **deactivate**
   `akadmin` — do not rename or delete it. The bootstrap blueprint keys on
   `username: akadmin` with `state: created`, so if no user by that name
   exists it simply makes a new one on the next reconcile.
3. **Bootstrap the tailnet** (headscale starts with an empty database):
   ```
   sudo headscale users create darman
   sudo headscale preauthkeys create --user darman --reusable --expiration 24h
   ```
   Put that key in **every** host's sops file as `tailscale_authkey` and rebuild.
4. **Headplane API key** — defaults to 90d, after which headplane silently stops
   listing nodes:
   ```
   sudo headscale apikeys create --expiration 999d      # -> headplane_headscale_api_key
   ```
5. **Headplane OIDC.** In Authentik create an OAuth2/OpenID provider
   (confidential, redirect `https://vpn.mgaction.town/admin/oidc/callback`,
   **a signing key must be selected** or discovery exposes no JWKS) and an
   application with slug **`headplane`** — the slug is what makes the issuer
   `.../application/o/headplane/` in `services/vpn/headplane.nix`. Client ID goes
   in that file, client secret into sops.
6. **Headscale OIDC** (optional — pre-auth keys work without it). A *second*
   Authentik provider/application, slug **`headscale`**, redirect
   `https://vpn.mgaction.town/oidc/callback` (headscale's own, not headplane's
   under `/admin`). Client ID in `services/vpn/headscale.nix`, secret into sops
   as `headscale_oidc_client_secret`.
   ⚠️  headscale runs OIDC discovery **at startup and a failure is fatal** —
   an issuer pointing at an application that doesn't exist yet means the
   control server won't boot, taking the whole tailnet's control plane with
   it. Always verify first:
   ```
   curl -s https://auth.mgaction.town/application/o/headscale/.well-known/openid-configuration
   ```
   Users created by OIDC login are distinct from `headscale users create`
   ones: headplane matches the OIDC `sub` claim against the user's
   `providerId`, CLI-made users have none, and 0.28 dropped both
   `map_legacy_users` and node reassignment — so moving an existing node to
   an OIDC user means re-enrolling it.

### jupiter

- `chown -R gitea:gitea /mnt/data/AppData/gitea` after the first deploy (the
  repos were copied in over CIFS as `darman:users`).
- **Re-enrolling after the headscale database was recreated:** `tailscaled`
  keeps its old node key and reports `Running`, and the autoconnect unit exits
  early on that state without ever sending the new pre-auth key. Force it:
  ```
  sudo tailscale logout && sudo systemctl restart tailscaled-autoconnect
  ```

### mars

- **Confirm the OIDC redirect still resolves.** hermes-agent.nix reuses
  jupiter's old Authentik application (slug `hermes`, redirect
  `https://hermes.mgaction.town/auth/callback`) unchanged — nothing to
  reconfigure in Authentik, just verify `neptun`'s `hermes.mgaction.town`
  vhost (now pointed at `mars.orbit.sol:9119`) actually reaches the
  dashboard once mars is up and joined the tailnet.
- **Carrying forward old chat history/memories:** mars starts with a fresh
  Hermes state dir (`/var/lib/hermes/.hermes`). jupiter's old instance data
  is backed up at `/mnt/data/AppData/hermes.bak-2026-08-21` — rsync it over
  (via the `/mnt/jupiter` samba mount) before the first switch if you want
  it preserved instead of starting clean.

### mercury (Raspberry Pi 3B+)

- `./deploy flash mercury /dev/sdX` writes the dedicated age key to the root
  partition. Without `~/.config/homelab/mercury/age.txt` it silently skips that
  step and **no secret decrypts on the box** — check `ls /run/secrets` after
  first boot.
- It boots from an SD card, so config changes are `./deploy switch mercury <ip>`
  (an aarch64 build — needs `extra-platforms` + binfmt on the laptop, see the
  gotchas in `CLAUDE.md`) rather than a reflash.
- **Suspect the card first** when binaries crash with `Illegal instruction` or
  services fail inexplicably. Failing flash returns corrupt data with no I/O
  errors in `dmesg`:
  ```
  sudo nix-store --verify --check-contents      # add --repair to fix
  ```
  A card that has corrupted one path will corrupt more. Replace it and reflash.
- **A reflash wipes `/var/lib/pihole`**, taking the gravity database with it.
  The blocklists themselves are declared in `services/network/pihole.nix`, and
  the `pihole-adlists` unit re-seeds them on boot and rebuilds gravity when it
  finds it empty — so this heals itself, but the first boot after a reflash
  spends several minutes downloading lists. Query history and dynamic DHCP
  leases are genuinely lost (static leases are declarative). Check with:
  ```
  systemctl status pihole-adlists
  sudo podman exec pihole pihole-FTL sqlite3 /etc/pihole/gravity.db \
    "SELECT address,enabled FROM adlist; SELECT COUNT(*) FROM gravity;"
  ```
  `Blocked DNS queries: 0` in the pihole logs means gravity is empty — DNS
  resolves fine, nothing is filtered.
- mercury's own `resolv.conf` is deliberately public resolvers, not its own
  pihole (`resolveLocalQueries = false`, see `CLAUDE.md`) — so `.sol` names do
  not resolve *on mercury itself*. That is expected, not a fault.
- **mercury is load-bearing for the whole tailnet's DNS.** headscale sets
  `override_local_dns = true` with pihole as the only global nameserver, so
  every node — including a phone on mobile data — resolves through it and gets
  ad blocking and `.sol` names anywhere. The flip side is that mercury (or the
  home connection) going down costs name resolution on every device, not just
  `.sol`. Recovery on a stranded device is turning Tailscale off.
  There is deliberately no public fallback in `nameservers.global`: tailscale
  treats that list as a set, so a second entry would let queries slip past the
  filter whenever mercury is slow.
  neptun and mercury opt out with `--accept-dns=false` — mercury because it
  would otherwise resolve through itself, neptun because a public reverse
  proxy must not depend on a Pi at home to renew its certificates.
- Tailnet names are `*.orbit.sol`, LAN names are `*.sol`. Both work everywhere
  on the tailnet because tailscale matches DNS routes by **longest suffix**, so
  `orbit.sol` reaches MagicDNS even though everything else goes to pihole.
  Never name a LAN host `orbit`: pihole's `address=/<host>.sol/<ip>` lines match
  a name *and everything beneath it*, which would swallow the entire tailnet
  zone.

## Adding a service

Copy the `whoami` block in `oci-containers.containers`, swap image/ports/volumes.
Native NixOS module exists for many apps (Nextcloud, Jellyfin, Grafana...) —
prefer `services.<app>` over a container when available. Add a `caddy`
`virtualHosts` block to expose it.

## Notes

- Backend is Podman with `dockerCompat` — `docker` CLI works, no daemon.
- Samba keeps its own password DB. `services.samba` never sets it; a systemd
  oneshot (`samba-smbpasswd`) provisions it. Host reads the password from
  `/run/secrets/samba_password` (**sops-nix**); the VM falls back to plaintext
  `/etc/samba/smb-password`.
- Secrets: `secrets/jupiter.yaml` is age-encrypted (safe to commit) to two
  recipients in `.sops.yaml` — the **admin** key (edit on laptop,
  `~/.config/sops/age/keys.txt`) and the **jupiter host** key (derived from its
  SSH host key via `ssh-to-age`, decrypts at runtime). Private keys live
  off-repo and are gitignored. Rotate/add recipients with `sops updatekeys`.
- Data disk: plain `fileSystems."/mnt/data"` in configuration.nix — kept out of
  disko so it is never formatted. Reference by `by-id` / `by-uuid`.
- `system.stateVersion` = `26.05`, install-time schema. Do NOT bump on upgrades.
- Terraform is not used: a single bare-metal box has no provider API. disko +
  nixos-anywhere cover provisioning natively.