A Telegram bot has a single getUpdates consumer, so a broker per host would split approval taps between them. mars now runs the gateway: the bot's only poller, relaying prompts and taps for terra's and titan's brokers over the tailnet (:8097) without ever seeing an item. Each broker authenticates with its own pass_broker_gateway_token; mars holds only the sha256, and the prompt is labelled with the host that token belongs to. The bot token and chat id move to mars and are dropped from the other hosts' sops files (terra.yaml also loses its leftover LibreChat keys and gains protonvpn_wg_private_key, which a later commit uses). Covered by checks.pass-broker-gateway (two nodes: labels, cancel, expiry, per-host token isolation, gateway down); the fakes are shared with checks.pass-broker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
276 lines
19 KiB
Markdown
276 lines
19 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
Flake-based NixOS config for a homelab. Hosts: **jupiter** (ZimaBlade NAS, x86_64),
|
||
**neptun** (netcup public reverse proxy + tailnet node, x86_64), **mercury** (Raspberry
|
||
Pi 3B+ DNS/DHCP, aarch64), **terra** (desktop), **titan** (Zenbook Pro Duo laptop),
|
||
**mars** (Hermes Agent + luna's apps). See `README.md` for the full install/deploy walkthrough.
|
||
|
||
## Layout
|
||
|
||
```
|
||
flake.nix # nixosConfigurations: real hosts + test/util targets
|
||
common.nix # shared base: user darman (key-only ssh), nix settings, firewall :22, tz
|
||
services/<cat>/*.nix # one reusable NixOS module per service, grouped by category
|
||
# (media, network, vpn, identity, dev, desktop); each opens
|
||
# ITS OWN firewall ports. services/containers.nix (podman
|
||
# backend) stays at the top level, shared across categories.
|
||
hosts/<h>/ # configuration.nix + disk-config.nix (disko) + hardware-configuration.nix + secrets.nix
|
||
profiles/workstation.nix # Hyprland workstation shared by terra + titan (NixOS side)
|
||
home/desktop/ # its home-manager side; per-host monitors via homelab.desktop.*
|
||
secrets/<h>.yaml # sops-nix, age-encrypted per host
|
||
scripts/deploy # config-agnostic deploy wrapper (all args mandatory)
|
||
scripts/edit_secrets
|
||
.sops.yaml # per-host encryption rules (admin key + each host's key)
|
||
```
|
||
|
||
A host = `common.nix` + the `services/**` modules it imports + its `hosts/<h>/configuration.nix`.
|
||
`services/` modules are engine-agnostic and shared across hosts (e.g. `services/vpn/tailscale.nix`,
|
||
`services/network/caddy.nix` used by jupiter and neptun).
|
||
|
||
## Commands
|
||
|
||
Eval/verify a config before deploying (eval only checks the module tree, not
|
||
freeform config like pihole's TOML or a container's runtime):
|
||
```
|
||
nix eval --raw .#nixosConfigurations.<host>.config.system.build.toplevel.drvPath
|
||
```
|
||
|
||
Deploy (from a non-NixOS laptop too — runs nixos-rebuild/nixos-anywhere via `nix run`):
|
||
```
|
||
./scripts/deploy switch <config> <host> # daily rebuild + activate
|
||
./scripts/deploy install <config> <host> # first install (nixos-anywhere, wipes OS disk)
|
||
./scripts/deploy kexec <config> <host> # RO-root box (ZimaOS): kexec into a RAM installer first
|
||
sudo ./scripts/deploy kexec-local [--yes] # kexec THIS box (no ssh); confirm prompt unless --yes
|
||
./scripts/deploy image mercury # build the aarch64 SD image
|
||
./scripts/deploy flash mercury /dev/sdX # build + write SD + drop the sops age key
|
||
```
|
||
|
||
Both password prompts are auto-filled from the "HomeLab" Proton Pass vault, keyed by
|
||
**`<config>`, not `<host>`** — items `darman@<config>` (sudo) and `root@<config>` (ssh).
|
||
A *trashed* Proton Pass item with the same title shadows the active one and yields an
|
||
empty password, so the script resolves the title among `--filter-state active` items
|
||
first; a plain `pass-cli item view --item-title` silently returns the trashed copy and
|
||
you get an interactive prompt with no explanation.
|
||
|
||
On terra/titan the values come from **pass-broker** (`services/identity/pass-broker.nix`),
|
||
not from anyone's own pass-cli session: it runs as its own user with an agent PAT scoped to
|
||
the HomeLab vaults and releases an item only after a Telegram tap. Prompts go through the
|
||
**gateway on mars** (`hosts/mars/pass-broker-gateway.nix`, tailnet :8097), the bot's only
|
||
`getUpdates` consumer: a second poller would split the taps. A new host gets its own
|
||
`pass_broker_gateway_token` in its sops file and that token's sha256 in the gateway's
|
||
`hosts`. When the socket exists, deploy uses it exclusively; a refused tap falls back to the
|
||
interactive prompt.
|
||
**Claude: fetch Proton Pass items only via `pass-broker get --item <title> --reason <why>`**,
|
||
never `pass-cli login` yourself, and never print a value you fetched. Values go into
|
||
variables or pipes, as `scripts/deploy` does.
|
||
|
||
`HOMELAB_KEXEC_TARBALL` (+ `_CPIO` / `_GZIP`) makes `kexec`/`kexec-local` reuse a
|
||
prebuilt installer instead of rebuilding ~500MB. The VM test below uses this.
|
||
|
||
Secrets (needs the admin age key at `~/.config/sops/age/keys.txt`):
|
||
```
|
||
./scripts/edit_secrets secrets/<host>.yaml
|
||
```
|
||
|
||
**Claude: never run `sops --decrypt`/`edit_secrets --show` and print the result — that
|
||
puts every plaintext secret in the file into the conversation transcript, not just the
|
||
one you wanted.** To add or change a single value non-interactively, use
|
||
`sops --set '["key"] "value"' secrets/<host>.yaml` (quote the value as JSON), which
|
||
writes without ever displaying the file's existing contents.
|
||
|
||
Test a service config BEFORE touching hardware — always do this for nontrivial changes:
|
||
```
|
||
# x86 QEMU VM of mercury's DNS/DHCP stack (fast; validates pihole/unbound at runtime)
|
||
nix build .#nixosConfigurations.mercury-vm.config.system.build.vm -o result
|
||
./result/bin/run-mercury-vm-vm # ssh -p 2223 darman@localhost (pw: test)
|
||
# jupiter services as a VirtualBox OVA
|
||
nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
|
||
# end-to-end VM test of `deploy kexec-local` (~45s once the tarball is built)
|
||
nix build .#checks.x86_64-linux.kexec-local -L
|
||
# VM test of luna's app hosting on mars (hosts/mars/luna-sites.nix): podman socket
|
||
# proxy, registry validation, caddy routes, reboot persistence
|
||
nix build .#checks.x86_64-linux.luna-sites -L
|
||
# VM test of pass-broker with fake pass-cli + Telegram: approve/deny/expiry,
|
||
# grants, policy, uid check
|
||
nix build .#checks.x86_64-linux.pass-broker -L
|
||
# two-node VM test of the gateway on mars with a broker using it: labels,
|
||
# cancel, expiry, per-host token isolation, gateway down
|
||
nix build .#checks.x86_64-linux.pass-broker-gateway -L
|
||
```
|
||
|
||
`checks.kexec-local` is the only way to exercise `kexec-local` at all: it jumps the
|
||
machine you are typing at, so it cannot be rehearsed on real hardware and a failure
|
||
looks exactly like a slow boot. It asserts the box actually left the old kernel
|
||
(SSH drops then returns), came back as `nixos-installer`, lost its old `/run`, and
|
||
kept its ssh host key. Run it after ANY change to the kexec paths.
|
||
|
||
## Secrets (sops-nix)
|
||
|
||
- Each `secrets/<host>.yaml` is encrypted to the **admin** key (edit) + that **host's**
|
||
key (runtime decrypt); rules in `.sops.yaml`. Private keys live OFF-repo:
|
||
`~/.config/sops/age/keys.txt` (admin), `~/.config/homelab/<host>/` (host keys).
|
||
- jupiter/neptun decrypt with their **ssh host key** (`ssh-to-age` recipient), shipped at
|
||
install via `nixos-anywhere --extra-files`.
|
||
- mercury (SD image, no `--extra-files`) uses a **dedicated age key** at
|
||
`/var/lib/sops-nix/age.txt` — `./scripts/deploy flash` writes it to the ext4 root partition.
|
||
- A service password that must come from sops but whose module has no `passwordFile`
|
||
hook (pihole, adguard) is injected via `sops.templates` → an env file → the service
|
||
(`FTLCONF_*` for pihole). See `hosts/mercury/configuration.nix`.
|
||
|
||
## Non-obvious gotchas (all learned the hard way)
|
||
|
||
- **aarch64 (mercury)**: the x86 laptop needs `extra-platforms = aarch64-linux` in
|
||
`/etc/nix/nix.custom.conf` (NOT `/etc/nix/nix.conf` — Determinate Nix regenerates that)
|
||
+ `qemu-user-static-binfmt`, else emulated builds fail with "platform mismatch". Or
|
||
build on the Pi with `--build-host darman@<ip>`.
|
||
- **pihole on mercury is a CONTAINER** (`services/network/pihole.nix`, official image, host
|
||
networking, caps NET_ADMIN/NET_RAW/SYS_NICE/CHOWN, `FTLCONF_*` env config). The native
|
||
`services.pihole-ftl` module **segfaults on the Pi 3B+ aarch64** — do not switch back.
|
||
- **`services.unbound.resolveLocalQueries = false`** is required: unbound listens on
|
||
:5335, so leaving it true points the host's resolv.conf at 127.0.0.1:**53** with nothing
|
||
there → boot-time DNS deadlock (starves image pulls / list downloads). Host resolves via
|
||
upstream `networking.nameservers`; pihole forwards to unbound explicitly at `127.0.0.1#5335`.
|
||
- **Remote deploy pushes unsigned closures**: hosts set `nix.settings.trusted-users =
|
||
[ "root" "@wheel" ]` (in common.nix) so a laptop-built closure is accepted by the target.
|
||
- **jupiter**: `boot.kernelParams = [ "reboot=pci" ]` (warm reboot hangs on that board);
|
||
eMMC initrd modules pinned in `configuration.nix` (generate-config misses them); the
|
||
16TB×2 **RAID0** data lives on `/mnt/data` with `nofail`, kept OUT of disko (never wiped).
|
||
- **terra: `./scripts/deploy kexec-local` hangs hard — do not use it there.** Confirmed
|
||
on real hardware: kexec's `device_shutdown()` pass runs (SCSI disks sync fine in the
|
||
log), then the machine goes dark and never comes back — `journalctl --list-boots`
|
||
showed a ~15min gap before the next boot, i.e. a genuine hang needing a manual power
|
||
cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT): discrete AMD GPUs are known
|
||
to hang during kexec's device-shutdown pass with no clean handoff before the jump —
|
||
same class of issue as jupiter's `reboot=pci` workaround, just fatal here instead of
|
||
slow. Use `./scripts/deploy install terra localhost` instead (README's "First install
|
||
on terra" section) — it detects it isn't inside a live installer yet and reboots via
|
||
a real `systemctl reboot` + systemd-boot one-shot `findiso=` entry, not kexec.
|
||
- **`./scripts/deploy install <config> localhost`'s behavior depends on `uname -n`**
|
||
(`is_live_installer()`): on a real running OS it builds `installer-iso`, stages it
|
||
locally, and reboots into it (`local_install_prepare_and_reboot()`); only inside
|
||
`nixos-installer` (kexec) or `homelab-installer` (installer-iso) does it actually run
|
||
disko + `nixos-install`. `installer-iso`'s `homelab-auto-install.service` closes the
|
||
loop: it reads `homelab.install=<config>` back off `/proc/cmdline` (set by the prepare
|
||
step) and re-runs the identical command itself once `homelab-checkout.service` has
|
||
cloned the repo — the whole reinstall is one command and unattended after the first
|
||
reboot. It confirms (type `yes`) before rebooting, like `flash`/`kexec-local`; `--yes`
|
||
skips that and is what the ISO passes itself. It always ASKS where to stage the iso
|
||
file (never auto-picks — the wrong disk here is destroyed mid-install);
|
||
`HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for scripted use. Both this and
|
||
`kexec-local` self-elevate via `sudo` (`require_root()`) rather than requiring you to
|
||
prefix the command yourself.
|
||
- **terra runs Limine, not systemd-boot** — `bootctl set-oneshot` is useless there
|
||
(`bootctl status` lists `✗ One-shot entry control`, and CachyOS's pacman hooks
|
||
regenerate `limine.conf` anyway). `local_install_prepare_and_reboot()` therefore
|
||
picks its one-shot mechanism at runtime: a systemd-boot loader entry when
|
||
`$BOOT/loader/entries` exists, otherwise `arm_efi_bootnext()` — a temporary UEFI
|
||
entry that EFI-stub-boots the kernel off the ESP, armed via the firmware's
|
||
`BootNext`. Created with `efibootmgr --create-only` (NOT `--create`, which pushes
|
||
it to the front of `BootOrder` and would make a wiped installer the permanent
|
||
default if anything went wrong). BootNext is spent by that one boot, so a failed
|
||
attempt still comes back on the normal bootloader. The EFI-stub path needs the
|
||
kernel on the **ESP** itself, not on a separate XBOOTLDR — hence `--print-esp-path`
|
||
rather than `--print-boot-path` in that mode. `homelab-auto-install.service` deletes
|
||
the leftover NVRAM entry as soon as it boots; both it and the script match the label
|
||
`Homelab Installer` EXACTLY (a prefix match would delete the Windows or Limine entry).
|
||
- **`installer-iso` must force `boot.initrd.systemd.enable = false`.** `findiso=` is a
|
||
SCRIPT-stage-1 feature (`stage-1-init.sh` loop-mounts the file it points at and
|
||
symlinks it to `/dev/root`). The systemd initrd — default since 26.05 — has NO findiso
|
||
path: it mounts `/iso` straight from `/dev/disk/by-label/<volumeID>` (iso-image.nix),
|
||
a label that only exists when the ISO is the physical boot medium. EFI-stub-booted off
|
||
the ESP with the iso as a plain file on another fs, that label never appears; stage 1
|
||
mounts `/sysroot` fine, then times out on `/sysroot/nix/.ro-store` waiting for
|
||
`/dev/disk/by-label/nixos-minimal-…` and drops to an emergency shell. The whole
|
||
`install <config> localhost` findiso path depends on script stage 1.
|
||
- **The staging-dir guard must fail CLOSED, and `findmnt` needs `--nofsroot`**: on btrfs
|
||
`findmnt -no SOURCE` prints `/dev/sdb2[/@]`, which `lsblk` cannot open, so a naive
|
||
parent-device lookup comes back empty. Treating empty as "different disk" silently
|
||
allowed staging the iso on the very disk disko then wiped — terra's CachyOS root is
|
||
exactly that layout, so it hit the live case. `disks_backing()` (`lsblk -rnso
|
||
NAME,TYPE`) returns EVERY whole-disk ancestor because LVM/RAID can span several
|
||
(`/mnt/ssd_01` → sdd + sde), and an empty result is a hard error, not a pass. btrfs
|
||
staging is refused outright: stage-1 mounts a btrfs volume's TOP level, so an iso
|
||
inside a subvolume is unreachable via `findiso=`.
|
||
- **`findiso=` must keep its leading slash**: stage-1 tests `[ -e "/findiso$isoPath" ]`,
|
||
so stripping the mountpoint prefix off a stagedir whose mountpoint is `/` yields
|
||
`var/tmp/x.iso` → `/findisovar/tmp/x.iso` → emergency shell, after you have already
|
||
rebooted out of the working OS.
|
||
- **The auto-install needs the host key shipped to it, and a `$HOME`**: the ISO is built
|
||
from a public repo with no credentials, so `local_install_prepare_and_reboot()` copies
|
||
the key onto the boot partition and passes that partition's PARTUUID as
|
||
`homelab.keypart=`; the service mounts it and drops the key in
|
||
`/root/.config/homelab/<config>/` before running the install. Also, systemd does NOT
|
||
set `$HOME` for a system service without `User=` (`SetLoginEnvironment=` defaults to
|
||
false), and `scripts/deploy` runs under `set -u` — hence `$KEYDIR` instead of a bare
|
||
`$HOME`, plus `Environment=HOME=/root` on the unit.
|
||
- **`installer-iso` must enable `experimental-features` itself.** `installation-cd-minimal`
|
||
leaves them unset, so `nix run` and `nixos-install --flake` both die with "experimental
|
||
Nix feature 'nix-command' is disabled". The nixos-images `kexec` installer sets them
|
||
itself, which is why the same `install <config> localhost` branch worked after
|
||
`kexec-local` but not from the ISO.
|
||
- **disko/nixos-anywhere run as `nix run .#disko` / `.#nixos-anywhere`**, from this
|
||
flake's locked inputs — not `nix run github:...`. They execute while a disk is being
|
||
wiped, so the revision must be the reviewed one in `flake.lock`, and it has to resolve
|
||
without network.
|
||
- **disko wipes only the OS disk** named in `hosts/<h>/disk-config.nix`; data disks are
|
||
plain `fileSystems` in `configuration.nix`.
|
||
- `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the
|
||
`./scripts/deploy kexec` step that streams a RAM installer (with static cpio/gzip since
|
||
ZimaOS lacks them).
|
||
- **`kexec/run` jumps ~6s AFTER it returns**: nixos-images' `kexec-run.sh` ends with
|
||
`nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &`. So the staging dir must OUTLIVE the
|
||
script — an `rm -rf` in an EXIT trap deletes the binary that performs the jump and the
|
||
box silently stays on the old kernel. `kexec-local` clears its trap before jumping and
|
||
then sleeps 60s on purpose. Covered by `checks.kexec-local`.
|
||
- **The kexec installer KEEPS the box's ssh host key**: `kexec-run.sh` copies
|
||
`/etc/ssh/ssh_host_*` into the appended initrd and `restore-remote-access.nix` installs
|
||
them back. So do NOT `ssh-keygen -R` after a kexec — the key does not change, and
|
||
clearing it just throws away the known_hosts record.
|
||
- **`kexec-local` stages on `/var/tmp`, not `/tmp`**: `kexec-run.sh` appends a fresh cpio
|
||
to `kexec/initrd` in place and execs binaries from that dir, so a size-capped or
|
||
`noexec` tmpfs gives a half-written initrd or a bare "Permission denied".
|
||
- **sops-nix values must be STRINGS.** `sops --set '["id"] 123'` stores `type:int`; the
|
||
build-time check only sees key names and passes, then activation's sops-install-secrets
|
||
fails with "is not a string" and installs NONE of the new secrets. The old generation
|
||
stays, so it looks like only the new keys are missing. Quote it: `'["id"] "123"'`.
|
||
- **A sops secret a USER service reads needs `owner`, not a mode.** `luna-voiced`
|
||
(`services/desktop/luna-voice.nix`) runs as darman's user unit, and a user manager
|
||
cannot read root:root 0400 — `LoadCredential` does not bridge that gap either.
|
||
Hence `sops.secrets.<x>.owner = "darman"` in `hosts/terra/secrets.nix`.
|
||
- **`elevenlabs_api_key` is in BOTH `secrets/mars.yaml` and `secrets/terra.yaml`** —
|
||
same value. terra calls ElevenLabs directly (the client-direct path Hermes
|
||
documents) rather than relaying audio through mars, so rotating that key means
|
||
editing two files. `hermes_api_server_key` is duplicated for the same reason:
|
||
mars serves the API, terra is its only client.
|
||
- **terra's mic is QUIET, and every voice threshold is sized for it.** Measured on
|
||
the AIR 192|4 (`luna-voice level`): ~0.001–0.002 RMS ambient, only 0.0045–0.005
|
||
for normal speech — about 6dB of separation. Consequences, all of which looked
|
||
like "the daemon is broken": a VAD multiplier above ~2 can never fire, an
|
||
absolute trigger floor of 0.006 sits *above* the user's voice, and whisper's own
|
||
gate at 0.015 discarded every word the VAD did capture. Hence `vadMult = 1.8`,
|
||
`vadFloor = 0.0025`, and `stt::normalize` scaling the clip up (capped at 30x)
|
||
before whisper sees it. Raising the gain on the interface is worth more than any
|
||
of these; tune via `homelab.lunaVoice.vad*` rather than editing Rust.
|
||
- **`try_recv()` POPS — an `if let Ok(Cancel)` destroys every other command.**
|
||
luna-voiced polls for a cancel while she speaks; written as a single `if let`,
|
||
any `say`/`toggle`/`mode off` arriving mid-reply was taken off the channel and
|
||
dropped, so SUPER+V during her answer did nothing at all and the next turn never
|
||
ran. Non-cancel commands are collected and re-sent into the channel after the
|
||
turn instead (`requeue` in `turn()`).
|
||
- **A rising sound must not update the noise floor.** Feeding every sample into the
|
||
floor EMA lets speech drag its own threshold up faster than it rises — measured,
|
||
a tone peaking at 0.029 was chased by a threshold that reached 0.018 and never
|
||
triggered. Only samples *below* the current trigger update the floor.
|
||
- **whisper invents words out of silence** — "you", "Thank you.", "Bye." — and a
|
||
held push-to-talk key over a quiet room is long AND empty, so duration alone
|
||
does not catch it. `stt::has_speech` gates on the loudest 100ms window instead;
|
||
without it every stray keypress sends a hallucinated sentence to luna.
|
||
- **terra's greeter is a throwaway Hyprland running `dotfiles/quickshell/greeter.qml`**
|
||
(`services/desktop/quickshell-greeter.nix`). It must exit after login or greetd never
|
||
starts the session, and with a Lua config `hyprctl dispatch exit` is REJECTED — it needs
|
||
`hyprctl dispatch 'hl.dsp.exit()'`. Test the flow without touching the real greetd by
|
||
running greetd's `fakegreet "qs -p …/greeter.qml"` inside a nested Hyprland
|
||
(user `user`, password `password`, then answer `9`).
|