Files
homelab/CLAUDE.md
T
darmanandClaude Opus 5.5 7a68716c1c feat(matrix): Synapse + MAS on jupiter, published by neptun
Homeserver for mgaction.town (@user:mgaction.town). Synapse and the
Matrix Authentication Service share jupiter's postgres; logins are
OIDC-only through Authentik via MAS, since Element X can't use
Synapse's own SSO. neptun serves the apex .well-known delegation plus
matrix. and account. vhosts; the admin API stays tailnet-only.

Synapse's signing key is a sops secret so the federation identity
survives losing the RAID0 array. Covered by checks.matrix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 13:04:52 +02:00

290 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
Flake-based NixOS config for a homelab. Hosts: **jupiter** (ZimaBlade NAS, x86_64),
**neptun** (netcup public reverse proxy + tailnet node, x86_64), **mercury** (Raspberry
Pi 3B+ DNS/DHCP, aarch64), **terra** (desktop), **titan** (Zenbook Pro Duo laptop),
**mars** (Hermes Agent + luna's apps). See `README.md` for the full install/deploy walkthrough.
## Layout
```
flake.nix # nixosConfigurations: real hosts + test/util targets
common.nix # shared base: user darman (key-only ssh), nix settings, firewall :22, tz
services/<cat>/*.nix # one reusable NixOS module per service, grouped by category
# (media, network, vpn, identity, dev, desktop, comms); each opens
# ITS OWN firewall ports. services/containers.nix (podman
# backend) stays at the top level, shared across categories.
hosts/<h>/ # configuration.nix + disk-config.nix (disko) + hardware-configuration.nix + secrets.nix
profiles/workstation.nix # Hyprland workstation shared by terra + titan (NixOS side)
home/desktop/ # its home-manager side; per-host monitors via homelab.desktop.*
secrets/<h>.yaml # sops-nix, age-encrypted per host
scripts/deploy # config-agnostic deploy wrapper (all args mandatory)
scripts/edit_secrets
.sops.yaml # per-host encryption rules (admin key + each host's key)
```
A host = `common.nix` + the `services/**` modules it imports + its `hosts/<h>/configuration.nix`.
`services/` modules are engine-agnostic and shared across hosts (e.g. `services/vpn/tailscale.nix`,
`services/network/caddy.nix` used by jupiter and neptun).
## Commands
Eval/verify a config before deploying (eval only checks the module tree, not
freeform config like pihole's TOML or a container's runtime):
```
nix eval --raw .#nixosConfigurations.<host>.config.system.build.toplevel.drvPath
```
Deploy (from a non-NixOS laptop too — runs nixos-rebuild/nixos-anywhere via `nix run`):
```
./scripts/deploy switch <config> <host> # daily rebuild + activate
./scripts/deploy install <config> <host> # first install (nixos-anywhere, wipes OS disk)
./scripts/deploy kexec <config> <host> # RO-root box (ZimaOS): kexec into a RAM installer first
sudo ./scripts/deploy kexec-local [--yes] # kexec THIS box (no ssh); confirm prompt unless --yes
./scripts/deploy image mercury # build the aarch64 SD image
./scripts/deploy flash mercury /dev/sdX # build + write SD + drop the sops age key
```
Both password prompts are auto-filled from the "HomeLab" Proton Pass vault, keyed by
**`<config>`, not `<host>`** — items `darman@<config>` (sudo) and `root@<config>` (ssh).
A *trashed* Proton Pass item with the same title shadows the active one and yields an
empty password, so the script resolves the title among `--filter-state active` items
first; a plain `pass-cli item view --item-title` silently returns the trashed copy and
you get an interactive prompt with no explanation.
On terra/titan the values come from **pass-broker** (`services/identity/pass-broker.nix`),
not from anyone's own pass-cli session: it runs as its own user with an agent PAT scoped to
the HomeLab vaults and releases an item only after a Telegram tap. Prompts go through the
**gateway on mars** (`hosts/mars/pass-broker-gateway.nix`, tailnet :8097), the bot's only
`getUpdates` consumer: a second poller would split the taps. A new host gets its own
`pass_broker_gateway_token` in its sops file and that token's sha256 in the gateway's
`hosts`. When the socket exists, deploy uses it exclusively; a refused tap falls back to the
interactive prompt.
**Claude: fetch Proton Pass items only via `pass-broker get --item <title> --reason <why>`**,
never `pass-cli login` yourself, and never print a value you fetched. Values go into
variables or pipes, as `scripts/deploy` does.
`HOMELAB_KEXEC_TARBALL` (+ `_CPIO` / `_GZIP`) makes `kexec`/`kexec-local` reuse a
prebuilt installer instead of rebuilding ~500MB. The VM test below uses this.
Secrets (needs the admin age key at `~/.config/sops/age/keys.txt`):
```
./scripts/edit_secrets secrets/<host>.yaml
```
**Claude: never run `sops --decrypt`/`edit_secrets --show` and print the result — that
puts every plaintext secret in the file into the conversation transcript, not just the
one you wanted.** To add or change a single value non-interactively, use
`sops --set '["key"] "value"' secrets/<host>.yaml` (quote the value as JSON), which
writes without ever displaying the file's existing contents.
Test a service config BEFORE touching hardware — always do this for nontrivial changes:
```
# x86 QEMU VM of mercury's DNS/DHCP stack (fast; validates pihole/unbound at runtime)
nix build .#nixosConfigurations.mercury-vm.config.system.build.vm -o result
./result/bin/run-mercury-vm-vm # ssh -p 2223 darman@localhost (pw: test)
# jupiter services as a VirtualBox OVA
nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
# end-to-end VM test of `deploy kexec-local` (~45s once the tarball is built)
nix build .#checks.x86_64-linux.kexec-local -L
# VM test of luna's app hosting on mars (hosts/mars/luna-sites.nix): podman socket
# proxy, registry validation, caddy routes, reboot persistence
nix build .#checks.x86_64-linux.luna-sites -L
# VM test of pass-broker with fake pass-cli + Telegram: approve/deny/expiry,
# grants, policy, uid check
nix build .#checks.x86_64-linux.pass-broker -L
# two-node VM test of the gateway on mars with a broker using it: labels,
# cancel, expiry, per-host token isolation, gateway down
nix build .#checks.x86_64-linux.pass-broker-gateway -L
# Synapse + MAS on jupiter behind neptun's caddy: C collation, delegation,
# SSO-only login flows, admin API not public, chat + media upload, restart
nix build .#checks.x86_64-linux.matrix -L
```
`checks.kexec-local` is the only way to exercise `kexec-local` at all: it jumps the
machine you are typing at, so it cannot be rehearsed on real hardware and a failure
looks exactly like a slow boot. It asserts the box actually left the old kernel
(SSH drops then returns), came back as `nixos-installer`, lost its old `/run`, and
kept its ssh host key. Run it after ANY change to the kexec paths.
## Secrets (sops-nix)
- Each `secrets/<host>.yaml` is encrypted to the **admin** key (edit) + that **host's**
key (runtime decrypt); rules in `.sops.yaml`. Private keys live OFF-repo:
`~/.config/sops/age/keys.txt` (admin), `~/.config/homelab/<host>/` (host keys).
- jupiter/neptun decrypt with their **ssh host key** (`ssh-to-age` recipient), shipped at
install via `nixos-anywhere --extra-files`.
- mercury (SD image, no `--extra-files`) uses a **dedicated age key** at
`/var/lib/sops-nix/age.txt` — `./scripts/deploy flash` writes it to the ext4 root partition.
- A service password that must come from sops but whose module has no `passwordFile`
hook (pihole, adguard) is injected via `sops.templates` → an env file → the service
(`FTLCONF_*` for pihole). See `hosts/mercury/configuration.nix`.
## Non-obvious gotchas (all learned the hard way)
- **aarch64 (mercury)**: the x86 laptop needs `extra-platforms = aarch64-linux` in
`/etc/nix/nix.custom.conf` (NOT `/etc/nix/nix.conf` — Determinate Nix regenerates that)
+ `qemu-user-static-binfmt`, else emulated builds fail with "platform mismatch". Or
build on the Pi with `--build-host darman@<ip>`.
- **pihole on mercury is a CONTAINER** (`services/network/pihole.nix`, official image, host
networking, caps NET_ADMIN/NET_RAW/SYS_NICE/CHOWN, `FTLCONF_*` env config). The native
`services.pihole-ftl` module **segfaults on the Pi 3B+ aarch64** — do not switch back.
- **`services.unbound.resolveLocalQueries = false`** is required: unbound listens on
:5335, so leaving it true points the host's resolv.conf at 127.0.0.1:**53** with nothing
there → boot-time DNS deadlock (starves image pulls / list downloads). Host resolves via
upstream `networking.nameservers`; pihole forwards to unbound explicitly at `127.0.0.1#5335`.
- **Remote deploy pushes unsigned closures**: hosts set `nix.settings.trusted-users =
[ "root" "@wheel" ]` (in common.nix) so a laptop-built closure is accepted by the target.
- **jupiter**: `boot.kernelParams = [ "reboot=pci" ]` (warm reboot hangs on that board);
eMMC initrd modules pinned in `configuration.nix` (generate-config misses them); the
16TB×2 **RAID0** data lives on `/mnt/data` with `nofail`, kept OUT of disko (never wiped).
- **terra: `./scripts/deploy kexec-local` hangs hard — do not use it there.** Confirmed
on real hardware: kexec's `device_shutdown()` pass runs (SCSI disks sync fine in the
log), then the machine goes dark and never comes back — `journalctl --list-boots`
showed a ~15min gap before the next boot, i.e. a genuine hang needing a manual power
cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT): discrete AMD GPUs are known
to hang during kexec's device-shutdown pass with no clean handoff before the jump —
same class of issue as jupiter's `reboot=pci` workaround, just fatal here instead of
slow. Use `./scripts/deploy install terra localhost` instead (README's "First install
on terra" section) — it detects it isn't inside a live installer yet and reboots via
a real `systemctl reboot` + systemd-boot one-shot `findiso=` entry, not kexec.
- **`./scripts/deploy install <config> localhost`'s behavior depends on `uname -n`**
(`is_live_installer()`): on a real running OS it builds `installer-iso`, stages it
locally, and reboots into it (`local_install_prepare_and_reboot()`); only inside
`nixos-installer` (kexec) or `homelab-installer` (installer-iso) does it actually run
disko + `nixos-install`. `installer-iso`'s `homelab-auto-install.service` closes the
loop: it reads `homelab.install=<config>` back off `/proc/cmdline` (set by the prepare
step) and re-runs the identical command itself once `homelab-checkout.service` has
cloned the repo — the whole reinstall is one command and unattended after the first
reboot. It confirms (type `yes`) before rebooting, like `flash`/`kexec-local`; `--yes`
skips that and is what the ISO passes itself. It always ASKS where to stage the iso
file (never auto-picks — the wrong disk here is destroyed mid-install);
`HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for scripted use. Both this and
`kexec-local` self-elevate via `sudo` (`require_root()`) rather than requiring you to
prefix the command yourself.
- **terra runs Limine, not systemd-boot** — `bootctl set-oneshot` is useless there
(`bootctl status` lists `✗ One-shot entry control`, and CachyOS's pacman hooks
regenerate `limine.conf` anyway). `local_install_prepare_and_reboot()` therefore
picks its one-shot mechanism at runtime: a systemd-boot loader entry when
`$BOOT/loader/entries` exists, otherwise `arm_efi_bootnext()` — a temporary UEFI
entry that EFI-stub-boots the kernel off the ESP, armed via the firmware's
`BootNext`. Created with `efibootmgr --create-only` (NOT `--create`, which pushes
it to the front of `BootOrder` and would make a wiped installer the permanent
default if anything went wrong). BootNext is spent by that one boot, so a failed
attempt still comes back on the normal bootloader. The EFI-stub path needs the
kernel on the **ESP** itself, not on a separate XBOOTLDR — hence `--print-esp-path`
rather than `--print-boot-path` in that mode. `homelab-auto-install.service` deletes
the leftover NVRAM entry as soon as it boots; both it and the script match the label
`Homelab Installer` EXACTLY (a prefix match would delete the Windows or Limine entry).
- **`installer-iso` must force `boot.initrd.systemd.enable = false`.** `findiso=` is a
SCRIPT-stage-1 feature (`stage-1-init.sh` loop-mounts the file it points at and
symlinks it to `/dev/root`). The systemd initrd — default since 26.05 — has NO findiso
path: it mounts `/iso` straight from `/dev/disk/by-label/<volumeID>` (iso-image.nix),
a label that only exists when the ISO is the physical boot medium. EFI-stub-booted off
the ESP with the iso as a plain file on another fs, that label never appears; stage 1
mounts `/sysroot` fine, then times out on `/sysroot/nix/.ro-store` waiting for
`/dev/disk/by-label/nixos-minimal-…` and drops to an emergency shell. The whole
`install <config> localhost` findiso path depends on script stage 1.
- **The staging-dir guard must fail CLOSED, and `findmnt` needs `--nofsroot`**: on btrfs
`findmnt -no SOURCE` prints `/dev/sdb2[/@]`, which `lsblk` cannot open, so a naive
parent-device lookup comes back empty. Treating empty as "different disk" silently
allowed staging the iso on the very disk disko then wiped — terra's CachyOS root is
exactly that layout, so it hit the live case. `disks_backing()` (`lsblk -rnso
NAME,TYPE`) returns EVERY whole-disk ancestor because LVM/RAID can span several
(`/mnt/ssd_01` → sdd + sde), and an empty result is a hard error, not a pass. btrfs
staging is refused outright: stage-1 mounts a btrfs volume's TOP level, so an iso
inside a subvolume is unreachable via `findiso=`.
- **`findiso=` must keep its leading slash**: stage-1 tests `[ -e "/findiso$isoPath" ]`,
so stripping the mountpoint prefix off a stagedir whose mountpoint is `/` yields
`var/tmp/x.iso` → `/findisovar/tmp/x.iso` → emergency shell, after you have already
rebooted out of the working OS.
- **The auto-install needs the host key shipped to it, and a `$HOME`**: the ISO is built
from a public repo with no credentials, so `local_install_prepare_and_reboot()` copies
the key onto the boot partition and passes that partition's PARTUUID as
`homelab.keypart=`; the service mounts it and drops the key in
`/root/.config/homelab/<config>/` before running the install. Also, systemd does NOT
set `$HOME` for a system service without `User=` (`SetLoginEnvironment=` defaults to
false), and `scripts/deploy` runs under `set -u` — hence `$KEYDIR` instead of a bare
`$HOME`, plus `Environment=HOME=/root` on the unit.
- **`installer-iso` must enable `experimental-features` itself.** `installation-cd-minimal`
leaves them unset, so `nix run` and `nixos-install --flake` both die with "experimental
Nix feature 'nix-command' is disabled". The nixos-images `kexec` installer sets them
itself, which is why the same `install <config> localhost` branch worked after
`kexec-local` but not from the ISO.
- **disko/nixos-anywhere run as `nix run .#disko` / `.#nixos-anywhere`**, from this
flake's locked inputs — not `nix run github:...`. They execute while a disk is being
wiped, so the revision must be the reviewed one in `flake.lock`, and it has to resolve
without network.
- **disko wipes only the OS disk** named in `hosts/<h>/disk-config.nix`; data disks are
plain `fileSystems` in `configuration.nix`.
- `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the
`./scripts/deploy kexec` step that streams a RAM installer (with static cpio/gzip since
ZimaOS lacks them).
- **`kexec/run` jumps ~6s AFTER it returns**: nixos-images' `kexec-run.sh` ends with
`nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &`. So the staging dir must OUTLIVE the
script — an `rm -rf` in an EXIT trap deletes the binary that performs the jump and the
box silently stays on the old kernel. `kexec-local` clears its trap before jumping and
then sleeps 60s on purpose. Covered by `checks.kexec-local`.
- **The kexec installer KEEPS the box's ssh host key**: `kexec-run.sh` copies
`/etc/ssh/ssh_host_*` into the appended initrd and `restore-remote-access.nix` installs
them back. So do NOT `ssh-keygen -R` after a kexec — the key does not change, and
clearing it just throws away the known_hosts record.
- **`kexec-local` stages on `/var/tmp`, not `/tmp`**: `kexec-run.sh` appends a fresh cpio
to `kexec/initrd` in place and execs binaries from that dir, so a size-capped or
`noexec` tmpfs gives a half-written initrd or a bare "Permission denied".
- **sops-nix values must be STRINGS.** `sops --set '["id"] 123'` stores `type:int`; the
build-time check only sees key names and passes, then activation's sops-install-secrets
fails with "is not a string" and installs NONE of the new secrets. The old generation
stays, so it looks like only the new keys are missing. Quote it: `'["id"] "123"'`.
- **A sops secret a USER service reads needs `owner`, not a mode.** `luna-voiced`
(`services/desktop/luna-voice.nix`) runs as darman's user unit, and a user manager
cannot read root:root 0400 — `LoadCredential` does not bridge that gap either.
Hence `sops.secrets.<x>.owner = "darman"` in `hosts/terra/secrets.nix`.
- **`elevenlabs_api_key` is in BOTH `secrets/mars.yaml` and `secrets/terra.yaml`** —
same value. terra calls ElevenLabs directly (the client-direct path Hermes
documents) rather than relaying audio through mars, so rotating that key means
editing two files. `hermes_api_server_key` is duplicated for the same reason:
mars serves the API, terra is its only client.
- **terra's mic is QUIET, and every voice threshold is sized for it.** Measured on
the AIR 192|4 (`luna-voice level`): ~0.001–0.002 RMS ambient, only 0.0045–0.005
for normal speech — about 6dB of separation. Consequences, all of which looked
like "the daemon is broken": a VAD multiplier above ~2 can never fire, an
absolute trigger floor of 0.006 sits *above* the user's voice, and whisper's own
gate at 0.015 discarded every word the VAD did capture. Hence `vadMult = 1.8`,
`vadFloor = 0.0025`, and `stt::normalize` scaling the clip up (capped at 30x)
before whisper sees it. Raising the gain on the interface is worth more than any
of these; tune via `homelab.lunaVoice.vad*` rather than editing Rust.
- **`try_recv()` POPS — an `if let Ok(Cancel)` destroys every other command.**
luna-voiced polls for a cancel while she speaks; written as a single `if let`,
any `say`/`toggle`/`mode off` arriving mid-reply was taken off the channel and
dropped, so SUPER+V during her answer did nothing at all and the next turn never
ran. Non-cancel commands are collected and re-sent into the channel after the
turn instead (`requeue` in `turn()`).
- **A rising sound must not update the noise floor.** Feeding every sample into the
floor EMA lets speech drag its own threshold up faster than it rises — measured,
a tone peaking at 0.029 was chased by a threshold that reached 0.018 and never
triggered. Only samples *below* the current trigger update the floor.
- **whisper invents words out of silence** — "you", "Thank you.", "Bye." — and a
held push-to-talk key over a quiet room is long AND empty, so duration alone
does not catch it. `stt::has_speech` gates on the loudest 100ms window instead;
without it every stray keypress sends a hallucinated sentence to luna.
- **terra's greeter is a throwaway Hyprland running `dotfiles/quickshell/greeter.qml`**
(`services/desktop/quickshell-greeter.nix`). It must exit after login or greetd never
starts the session, and with a Lua config `hyprctl dispatch exit` is REJECTED — it needs
`hyprctl dispatch 'hl.dsp.exit()'`. Test the flow without touching the real greetd by
running greetd's `fakegreet "qs -p …/greeter.qml"` inside a nested Hyprland
(user `user`, password `password`, then answer `9`).
- **Matrix logins go through MAS, not Synapse's `oidc_providers`** (`services/comms/matrix.nix`).
Element X only supports SSO via Matrix Authentication Service, and moving an existing
Synapse onto MAS is a one-way `syn2mas` migration — so it was there from day one. With
MAS enabled Synapse refuses to start if password login, registration or any SSO
section is on. Admin rights come from MAS:
`sudo -u matrix-authentication-service mas manage promote-admin <user>`.
- **Synapse's database needs `LC_COLLATE`/`LC_CTYPE` = C**, which `ensureDatabases`
can't give (cluster default locale) and `initialScript` never runs on jupiter's
existing immich cluster — hence the `\gexec` CREATE DATABASE in `postgresql-setup`'s
`ExecStartPost`. Synapse's signing key is a sops secret, not generated on first
start: it IS the server's federation identity, and the array it would live on is RAID0.