Also jellyfin-mpv-shim, mangohud logging, portal-backed xdg-open, systemd-resolved with fallback DNS, and keyring under the user manager. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
303 lines
21 KiB
Markdown
303 lines
21 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
Flake-based NixOS config for a homelab. Hosts: **jupiter** (ZimaBlade NAS, x86_64),
|
||
**neptun** (netcup public reverse proxy + tailnet node, x86_64), **mercury** (Raspberry
|
||
Pi 3B+ DNS/DHCP, aarch64), **terra** (desktop), **titan** (Zenbook Pro Duo laptop),
|
||
**mars** (Hermes Agent + luna's apps). See `README.md` for the full install/deploy walkthrough.
|
||
|
||
## Layout
|
||
|
||
```
|
||
flake.nix # nixosConfigurations: real hosts + test/util targets
|
||
common.nix # shared base: user darman (key-only ssh), nix settings, firewall :22, tz
|
||
services/<cat>/*.nix # one reusable NixOS module per service, grouped by category
|
||
# (media, network, vpn, identity, dev, desktop, comms); each opens
|
||
# ITS OWN firewall ports. services/containers.nix (podman
|
||
# backend) stays at the top level, shared across categories.
|
||
hosts/<h>/ # configuration.nix + disk-config.nix (disko) + hardware-configuration.nix + secrets.nix
|
||
profiles/workstation.nix # Hyprland workstation shared by terra + titan (NixOS side)
|
||
home/desktop/ # its home-manager side; per-host monitors via homelab.desktop.*
|
||
secrets/<h>.yaml # sops-nix, age-encrypted per host
|
||
scripts/deploy # config-agnostic deploy wrapper (all args mandatory)
|
||
scripts/edit_secrets
|
||
.sops.yaml # per-host encryption rules (admin key + each host's key)
|
||
```
|
||
|
||
A host = `common.nix` + the `services/**` modules it imports + its `hosts/<h>/configuration.nix`.
|
||
`services/` modules are engine-agnostic and shared across hosts (e.g. `services/vpn/tailscale.nix`,
|
||
`services/network/caddy.nix` used by jupiter and neptun).
|
||
|
||
## Commands
|
||
|
||
Eval/verify a config before deploying (eval only checks the module tree, not
|
||
freeform config like pihole's TOML or a container's runtime):
|
||
```
|
||
nix eval --raw .#nixosConfigurations.<host>.config.system.build.toplevel.drvPath
|
||
```
|
||
|
||
Deploy (from a non-NixOS laptop too — runs nixos-rebuild/nixos-anywhere via `nix run`):
|
||
```
|
||
./scripts/deploy switch <config> <host> # daily rebuild + activate
|
||
./scripts/deploy install <config> <host> # first install (nixos-anywhere, wipes OS disk)
|
||
./scripts/deploy kexec <config> <host> # RO-root box (ZimaOS): kexec into a RAM installer first
|
||
sudo ./scripts/deploy kexec-local [--yes] # kexec THIS box (no ssh); confirm prompt unless --yes
|
||
./scripts/deploy image mercury # build the aarch64 SD image
|
||
./scripts/deploy flash mercury /dev/sdX # build + write SD + drop the sops age key
|
||
```
|
||
|
||
Both password prompts are auto-filled from the "HomeLab" Proton Pass vault, keyed by
|
||
**`<config>`, not `<host>`** — items `darman@<config>` (sudo) and `root@<config>` (ssh).
|
||
A *trashed* Proton Pass item with the same title shadows the active one and yields an
|
||
empty password, so the script resolves the title among `--filter-state active` items
|
||
first; a plain `pass-cli item view --item-title` silently returns the trashed copy and
|
||
you get an interactive prompt with no explanation.
|
||
|
||
On terra/titan the values come from **pass-broker** (`services/identity/pass-broker.nix`),
|
||
not from anyone's own pass-cli session: it runs as its own user with an agent PAT scoped to
|
||
the HomeLab vaults and releases an item only after a Telegram tap. Prompts go through the
|
||
**gateway on mars** (`hosts/mars/pass-broker-gateway.nix`, tailnet :8097), the bot's only
|
||
`getUpdates` consumer: a second poller would split the taps. A new host gets its own
|
||
`pass_broker_gateway_token` in its sops file and that token's sha256 in the gateway's
|
||
`hosts`. When the socket exists, deploy uses it exclusively; a refused tap falls back to the
|
||
interactive prompt.
|
||
**Claude: fetch Proton Pass items only via `pass-broker get --item <title> --reason <why>`**,
|
||
never `pass-cli login` yourself, and never print a value you fetched. Values go into
|
||
variables or pipes, as `scripts/deploy` does.
|
||
|
||
`HOMELAB_KEXEC_TARBALL` (+ `_CPIO` / `_GZIP`) makes `kexec`/`kexec-local` reuse a
|
||
prebuilt installer instead of rebuilding ~500MB. The VM test below uses this.
|
||
|
||
Secrets (needs the admin age key at `~/.config/sops/age/keys.txt`):
|
||
```
|
||
./scripts/edit_secrets secrets/<host>.yaml
|
||
```
|
||
|
||
**Claude: never run `sops --decrypt`/`edit_secrets --show` and print the result — that
|
||
puts every plaintext secret in the file into the conversation transcript, not just the
|
||
one you wanted.** To add or change a single value non-interactively, use
|
||
`sops --set '["key"] "value"' secrets/<host>.yaml` (quote the value as JSON), which
|
||
writes without ever displaying the file's existing contents.
|
||
|
||
Test a service config BEFORE touching hardware — always do this for nontrivial changes:
|
||
```
|
||
# x86 QEMU VM of mercury's DNS/DHCP stack (fast; validates pihole/unbound at runtime)
|
||
nix build .#nixosConfigurations.mercury-vm.config.system.build.vm -o result
|
||
./result/bin/run-mercury-vm-vm # ssh -p 2223 darman@localhost (pw: test)
|
||
# jupiter services as a VirtualBox OVA
|
||
nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
|
||
# end-to-end VM test of `deploy kexec-local` (~45s once the tarball is built)
|
||
nix build .#checks.x86_64-linux.kexec-local -L
|
||
# VM test of luna's app hosting on mars (hosts/mars/luna-sites.nix): podman socket
|
||
# proxy, registry validation, caddy routes, reboot persistence
|
||
nix build .#checks.x86_64-linux.luna-sites -L
|
||
# VM test of pass-broker with fake pass-cli + Telegram: approve/deny/expiry,
|
||
# grants, policy, uid check
|
||
nix build .#checks.x86_64-linux.pass-broker -L
|
||
# two-node VM test of the gateway on mars with a broker using it: labels,
|
||
# cancel, expiry, per-host token isolation, gateway down
|
||
nix build .#checks.x86_64-linux.pass-broker-gateway -L
|
||
# Synapse + MAS on jupiter behind neptun's caddy: C collation, delegation,
|
||
# SSO-only login flows, admin API not public, chat + media upload, restart
|
||
nix build .#checks.x86_64-linux.matrix -L
|
||
```
|
||
|
||
`checks.kexec-local` is the only way to exercise `kexec-local` at all: it jumps the
|
||
machine you are typing at, so it cannot be rehearsed on real hardware and a failure
|
||
looks exactly like a slow boot. It asserts the box actually left the old kernel
|
||
(SSH drops then returns), came back as `nixos-installer`, lost its old `/run`, and
|
||
kept its ssh host key. Run it after ANY change to the kexec paths.
|
||
|
||
## Secrets (sops-nix)
|
||
|
||
- Each `secrets/<host>.yaml` is encrypted to the **admin** key (edit) + that **host's**
|
||
key (runtime decrypt); rules in `.sops.yaml`. Private keys live OFF-repo:
|
||
`~/.config/sops/age/keys.txt` (admin), `~/.config/homelab/<host>/` (host keys).
|
||
- jupiter/neptun decrypt with their **ssh host key** (`ssh-to-age` recipient), shipped at
|
||
install via `nixos-anywhere --extra-files`.
|
||
- mercury (SD image, no `--extra-files`) uses a **dedicated age key** at
|
||
`/var/lib/sops-nix/age.txt` — `./scripts/deploy flash` writes it to the ext4 root partition.
|
||
- A service password that must come from sops but whose module has no `passwordFile`
|
||
hook (pihole, adguard) is injected via `sops.templates` → an env file → the service
|
||
(`FTLCONF_*` for pihole). See `hosts/mercury/configuration.nix`.
|
||
|
||
## Non-obvious gotchas (all learned the hard way)
|
||
|
||
- **aarch64 (mercury)**: the x86 laptop needs `extra-platforms = aarch64-linux` in
|
||
`/etc/nix/nix.custom.conf` (NOT `/etc/nix/nix.conf` — Determinate Nix regenerates that)
|
||
+ `qemu-user-static-binfmt`, else emulated builds fail with "platform mismatch". Or
|
||
build on the Pi with `--build-host darman@<ip>`.
|
||
- **pihole on mercury is a CONTAINER** (`services/network/pihole.nix`, official image, host
|
||
networking, caps NET_ADMIN/NET_RAW/SYS_NICE/CHOWN, `FTLCONF_*` env config). The native
|
||
`services.pihole-ftl` module **segfaults on the Pi 3B+ aarch64** — do not switch back.
|
||
- **`services.unbound.resolveLocalQueries = false`** is required: unbound listens on
|
||
:5335, so leaving it true points the host's resolv.conf at 127.0.0.1:**53** with nothing
|
||
there → boot-time DNS deadlock (starves image pulls / list downloads). Host resolves via
|
||
upstream `networking.nameservers`; pihole forwards to unbound explicitly at `127.0.0.1#5335`.
|
||
- **Remote deploy pushes unsigned closures**: hosts set `nix.settings.trusted-users =
|
||
[ "root" "@wheel" ]` (in common.nix) so a laptop-built closure is accepted by the target.
|
||
- **jupiter**: `boot.kernelParams = [ "reboot=pci" ]` (warm reboot hangs on that board);
|
||
eMMC initrd modules pinned in `configuration.nix` (generate-config misses them); the
|
||
16TB×2 **RAID0** data lives on `/mnt/data` with `nofail`, kept OUT of disko (never wiped).
|
||
- **terra: `./scripts/deploy kexec-local` hangs hard — do not use it there.** Confirmed
|
||
on real hardware: kexec's `device_shutdown()` pass runs (SCSI disks sync fine in the
|
||
log), then the machine goes dark and never comes back — `journalctl --list-boots`
|
||
showed a ~15min gap before the next boot, i.e. a genuine hang needing a manual power
|
||
cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT): discrete AMD GPUs are known
|
||
to hang during kexec's device-shutdown pass with no clean handoff before the jump —
|
||
same class of issue as jupiter's `reboot=pci` workaround, just fatal here instead of
|
||
slow. Use `./scripts/deploy install terra localhost` instead (README's "First install
|
||
on terra" section) — it detects it isn't inside a live installer yet and reboots via
|
||
a real `systemctl reboot` + systemd-boot one-shot `findiso=` entry, not kexec.
|
||
- **`./scripts/deploy install <config> localhost`'s behavior depends on `uname -n`**
|
||
(`is_live_installer()`): on a real running OS it builds `installer-iso`, stages it
|
||
locally, and reboots into it (`local_install_prepare_and_reboot()`); only inside
|
||
`nixos-installer` (kexec) or `homelab-installer` (installer-iso) does it actually run
|
||
disko + `nixos-install`. `installer-iso`'s `homelab-auto-install.service` closes the
|
||
loop: it reads `homelab.install=<config>` back off `/proc/cmdline` (set by the prepare
|
||
step) and re-runs the identical command itself once `homelab-checkout.service` has
|
||
cloned the repo — the whole reinstall is one command and unattended after the first
|
||
reboot. It confirms (type `yes`) before rebooting, like `flash`/`kexec-local`; `--yes`
|
||
skips that and is what the ISO passes itself. It always ASKS where to stage the iso
|
||
file (never auto-picks — the wrong disk here is destroyed mid-install);
|
||
`HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for scripted use. Both this and
|
||
`kexec-local` self-elevate via `sudo` (`require_root()`) rather than requiring you to
|
||
prefix the command yourself.
|
||
- **terra runs Limine, not systemd-boot** — `bootctl set-oneshot` is useless there
|
||
(`bootctl status` lists `✗ One-shot entry control`, and CachyOS's pacman hooks
|
||
regenerate `limine.conf` anyway). `local_install_prepare_and_reboot()` therefore
|
||
picks its one-shot mechanism at runtime: a systemd-boot loader entry when
|
||
`$BOOT/loader/entries` exists, otherwise `arm_efi_bootnext()` — a temporary UEFI
|
||
entry that EFI-stub-boots the kernel off the ESP, armed via the firmware's
|
||
`BootNext`. Created with `efibootmgr --create-only` (NOT `--create`, which pushes
|
||
it to the front of `BootOrder` and would make a wiped installer the permanent
|
||
default if anything went wrong). BootNext is spent by that one boot, so a failed
|
||
attempt still comes back on the normal bootloader. The EFI-stub path needs the
|
||
kernel on the **ESP** itself, not on a separate XBOOTLDR — hence `--print-esp-path`
|
||
rather than `--print-boot-path` in that mode. `homelab-auto-install.service` deletes
|
||
the leftover NVRAM entry as soon as it boots; both it and the script match the label
|
||
`Homelab Installer` EXACTLY (a prefix match would delete the Windows or Limine entry).
|
||
- **`installer-iso` must force `boot.initrd.systemd.enable = false`.** `findiso=` is a
|
||
SCRIPT-stage-1 feature (`stage-1-init.sh` loop-mounts the file it points at and
|
||
symlinks it to `/dev/root`). The systemd initrd — default since 26.05 — has NO findiso
|
||
path: it mounts `/iso` straight from `/dev/disk/by-label/<volumeID>` (iso-image.nix),
|
||
a label that only exists when the ISO is the physical boot medium. EFI-stub-booted off
|
||
the ESP with the iso as a plain file on another fs, that label never appears; stage 1
|
||
mounts `/sysroot` fine, then times out on `/sysroot/nix/.ro-store` waiting for
|
||
`/dev/disk/by-label/nixos-minimal-…` and drops to an emergency shell. The whole
|
||
`install <config> localhost` findiso path depends on script stage 1.
|
||
- **The staging-dir guard must fail CLOSED, and `findmnt` needs `--nofsroot`**: on btrfs
|
||
`findmnt -no SOURCE` prints `/dev/sdb2[/@]`, which `lsblk` cannot open, so a naive
|
||
parent-device lookup comes back empty. Treating empty as "different disk" silently
|
||
allowed staging the iso on the very disk disko then wiped — terra's CachyOS root is
|
||
exactly that layout, so it hit the live case. `disks_backing()` (`lsblk -rnso
|
||
NAME,TYPE`) returns EVERY whole-disk ancestor because LVM/RAID can span several
|
||
(`/mnt/ssd_01` → sdd + sde), and an empty result is a hard error, not a pass. btrfs
|
||
staging is refused outright: stage-1 mounts a btrfs volume's TOP level, so an iso
|
||
inside a subvolume is unreachable via `findiso=`.
|
||
- **`findiso=` must keep its leading slash**: stage-1 tests `[ -e "/findiso$isoPath" ]`,
|
||
so stripping the mountpoint prefix off a stagedir whose mountpoint is `/` yields
|
||
`var/tmp/x.iso` → `/findisovar/tmp/x.iso` → emergency shell, after you have already
|
||
rebooted out of the working OS.
|
||
- **The auto-install needs the host key shipped to it, and a `$HOME`**: the ISO is built
|
||
from a public repo with no credentials, so `local_install_prepare_and_reboot()` copies
|
||
the key onto the boot partition and passes that partition's PARTUUID as
|
||
`homelab.keypart=`; the service mounts it and drops the key in
|
||
`/root/.config/homelab/<config>/` before running the install. Also, systemd does NOT
|
||
set `$HOME` for a system service without `User=` (`SetLoginEnvironment=` defaults to
|
||
false), and `scripts/deploy` runs under `set -u` — hence `$KEYDIR` instead of a bare
|
||
`$HOME`, plus `Environment=HOME=/root` on the unit.
|
||
- **`installer-iso` must enable `experimental-features` itself.** `installation-cd-minimal`
|
||
leaves them unset, so `nix run` and `nixos-install --flake` both die with "experimental
|
||
Nix feature 'nix-command' is disabled". The nixos-images `kexec` installer sets them
|
||
itself, which is why the same `install <config> localhost` branch worked after
|
||
`kexec-local` but not from the ISO.
|
||
- **disko/nixos-anywhere run as `nix run .#disko` / `.#nixos-anywhere`**, from this
|
||
flake's locked inputs — not `nix run github:...`. They execute while a disk is being
|
||
wiped, so the revision must be the reviewed one in `flake.lock`, and it has to resolve
|
||
without network.
|
||
- **disko wipes only the OS disk** named in `hosts/<h>/disk-config.nix`; data disks are
|
||
plain `fileSystems` in `configuration.nix`.
|
||
- `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the
|
||
`./scripts/deploy kexec` step that streams a RAM installer (with static cpio/gzip since
|
||
ZimaOS lacks them).
|
||
- **`kexec/run` jumps ~6s AFTER it returns**: nixos-images' `kexec-run.sh` ends with
|
||
`nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &`. So the staging dir must OUTLIVE the
|
||
script — an `rm -rf` in an EXIT trap deletes the binary that performs the jump and the
|
||
box silently stays on the old kernel. `kexec-local` clears its trap before jumping and
|
||
then sleeps 60s on purpose. Covered by `checks.kexec-local`.
|
||
- **The kexec installer KEEPS the box's ssh host key**: `kexec-run.sh` copies
|
||
`/etc/ssh/ssh_host_*` into the appended initrd and `restore-remote-access.nix` installs
|
||
them back. So do NOT `ssh-keygen -R` after a kexec — the key does not change, and
|
||
clearing it just throws away the known_hosts record.
|
||
- **`kexec-local` stages on `/var/tmp`, not `/tmp`**: `kexec-run.sh` appends a fresh cpio
|
||
to `kexec/initrd` in place and execs binaries from that dir, so a size-capped or
|
||
`noexec` tmpfs gives a half-written initrd or a bare "Permission denied".
|
||
- **sops-nix values must be STRINGS.** `sops --set '["id"] 123'` stores `type:int`; the
|
||
build-time check only sees key names and passes, then activation's sops-install-secrets
|
||
fails with "is not a string" and installs NONE of the new secrets. The old generation
|
||
stays, so it looks like only the new keys are missing. Quote it: `'["id"] "123"'`.
|
||
- **A sops secret a USER service reads needs `owner`, not a mode.** `luna-voiced`
|
||
(`services/desktop/luna-voice.nix`) runs as darman's user unit, and a user manager
|
||
cannot read root:root 0400 — `LoadCredential` does not bridge that gap either.
|
||
Hence `sops.secrets.<x>.owner = "darman"` in `hosts/terra/secrets.nix`.
|
||
- **`elevenlabs_api_key` is in BOTH `secrets/mars.yaml` and `secrets/terra.yaml`** —
|
||
same value. terra calls ElevenLabs directly (the client-direct path Hermes
|
||
documents) rather than relaying audio through mars, so rotating that key means
|
||
editing two files. `hermes_api_server_key` is duplicated for the same reason:
|
||
mars serves the API, terra is its only client.
|
||
- **terra's mic is QUIET, and every voice threshold is sized for it.** Measured on
|
||
the AIR 192|4 (`luna-voice level`): ~0.001–0.002 RMS ambient, only 0.0045–0.005
|
||
for normal speech — about 6dB of separation. Consequences, all of which looked
|
||
like "the daemon is broken": a VAD multiplier above ~2 can never fire, an
|
||
absolute trigger floor of 0.006 sits *above* the user's voice, and whisper's own
|
||
gate at 0.015 discarded every word the VAD did capture. Hence `vadMult = 1.8`,
|
||
`vadFloor = 0.0025`, and `stt::normalize` scaling the clip up (capped at 30x)
|
||
before whisper sees it. Raising the gain on the interface is worth more than any
|
||
of these; tune via `homelab.lunaVoice.vad*` rather than editing Rust.
|
||
- **`try_recv()` POPS — an `if let Ok(Cancel)` destroys every other command.**
|
||
luna-voiced polls for a cancel while she speaks; written as a single `if let`,
|
||
any `say`/`toggle`/`mode off` arriving mid-reply was taken off the channel and
|
||
dropped, so SUPER+V during her answer did nothing at all and the next turn never
|
||
ran. Non-cancel commands are collected and re-sent into the channel after the
|
||
turn instead (`requeue` in `turn()`).
|
||
- **A rising sound must not update the noise floor.** Feeding every sample into the
|
||
floor EMA lets speech drag its own threshold up faster than it rises — measured,
|
||
a tone peaking at 0.029 was chased by a threshold that reached 0.018 and never
|
||
triggered. Only samples *below* the current trigger update the floor.
|
||
- **whisper invents words out of silence** — "you", "Thank you.", "Bye." — and a
|
||
held push-to-talk key over a quiet room is long AND empty, so duration alone
|
||
does not catch it. `stt::has_speech` gates on the loudest 100ms window instead;
|
||
without it every stray keypress sends a hallucinated sentence to luna.
|
||
- **terra's greeter is a throwaway Hyprland running `dotfiles/quickshell/greeter.qml`**
|
||
(`services/desktop/quickshell-greeter.nix`). It must exit after login or greetd never
|
||
starts the session, and with a Lua config `hyprctl dispatch exit` is REJECTED — it needs
|
||
`hyprctl dispatch 'hl.dsp.exit()'`. Test the flow without touching the real greetd by
|
||
running greetd's `fakegreet "qs -p …/greeter.qml"` inside a nested Hyprland
|
||
(user `user`, password `password`, then answer `9`).
|
||
- **Matrix logins go through MAS, not Synapse's `oidc_providers`** (`services/comms/matrix.nix`).
|
||
Element X only supports SSO via Matrix Authentication Service, and moving an existing
|
||
Synapse onto MAS is a one-way `syn2mas` migration — so it was there from day one. With
|
||
MAS enabled Synapse refuses to start if password login, registration or any SSO
|
||
section is on. Admin rights come from MAS:
|
||
`sudo -u matrix-authentication-service mas manage promote-admin <user>`.
|
||
- **Synapse's database needs `LC_COLLATE`/`LC_CTYPE` = C**, which `ensureDatabases`
|
||
can't give (cluster default locale) and `initialScript` never runs on jupiter's
|
||
existing immich cluster — hence the `\gexec` CREATE DATABASE in `postgresql-setup`'s
|
||
`ExecStartPost`. Synapse's signing key is a sops secret, not generated on first
|
||
start: it IS the server's federation identity, and the array it would live on is RAID0.
|
||
- **A MAS compat token is "not active" until Synapse has its device.** Synapse rejects
|
||
any token whose device it doesn't know, and `mas manage register-user` immediately
|
||
followed by `issue-compatibility-token` races: the new user's provisioning job syncs
|
||
devices before the token's session is committed and deletes the device just created.
|
||
`mas manage provision-all-users` re-syncs it. luna's Matrix bot token
|
||
(`hosts/mars/secrets.nix`) hit this.
|
||
- **Run the Hermes CLI on mars as `hermes`, never as root** (`podman exec -u hermes`, or
|
||
the `hermes` zsh alias). Since v0.22, optional deps (ElevenLabs `tts-premium`, plugin
|
||
deps like Honcho's `honcho-ai`) live in PM generations under `/opt/data/installs`; a
|
||
root `hermes pm status` created a root-owned `.leases` dir there and every later
|
||
install failed with "install state is not writable by this user". Those deps are
|
||
luna's volume state, not Nix: after a reset, `hermes pm install --extra tts-premium`
|
||
and `hermes plugins install honcho --force --yes-deps --enable`.
|