A Telegram bot has a single getUpdates consumer, so a broker per host would split approval taps between them. mars now runs the gateway: the bot's only poller, relaying prompts and taps for terra's and titan's brokers over the tailnet (:8097) without ever seeing an item. Each broker authenticates with its own pass_broker_gateway_token; mars holds only the sha256, and the prompt is labelled with the host that token belongs to. The bot token and chat id move to mars and are dropped from the other hosts' sops files (terra.yaml also loses its leftover LibreChat keys and gains protonvpn_wg_private_key, which a later commit uses). Covered by checks.pass-broker-gateway (two nodes: labels, cancel, expiry, per-host token isolation, gateway down); the fakes are shared with checks.pass-broker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
19 KiB
CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
Flake-based NixOS config for a homelab. Hosts: jupiter (ZimaBlade NAS, x86_64),
neptun (netcup public reverse proxy + tailnet node, x86_64), mercury (Raspberry
Pi 3B+ DNS/DHCP, aarch64), terra (desktop), titan (Zenbook Pro Duo laptop),
mars (Hermes Agent + luna's apps). See README.md for the full install/deploy walkthrough.
Layout
flake.nix # nixosConfigurations: real hosts + test/util targets
common.nix # shared base: user darman (key-only ssh), nix settings, firewall :22, tz
services/<cat>/*.nix # one reusable NixOS module per service, grouped by category
# (media, network, vpn, identity, dev, desktop); each opens
# ITS OWN firewall ports. services/containers.nix (podman
# backend) stays at the top level, shared across categories.
hosts/<h>/ # configuration.nix + disk-config.nix (disko) + hardware-configuration.nix + secrets.nix
profiles/workstation.nix # Hyprland workstation shared by terra + titan (NixOS side)
home/desktop/ # its home-manager side; per-host monitors via homelab.desktop.*
secrets/<h>.yaml # sops-nix, age-encrypted per host
scripts/deploy # config-agnostic deploy wrapper (all args mandatory)
scripts/edit_secrets
.sops.yaml # per-host encryption rules (admin key + each host's key)
A host = common.nix + the services/** modules it imports + its hosts/<h>/configuration.nix.
services/ modules are engine-agnostic and shared across hosts (e.g. services/vpn/tailscale.nix,
services/network/caddy.nix used by jupiter and neptun).
Commands
Eval/verify a config before deploying (eval only checks the module tree, not freeform config like pihole's TOML or a container's runtime):
nix eval --raw .#nixosConfigurations.<host>.config.system.build.toplevel.drvPath
Deploy (from a non-NixOS laptop too — runs nixos-rebuild/nixos-anywhere via nix run):
./scripts/deploy switch <config> <host> # daily rebuild + activate
./scripts/deploy install <config> <host> # first install (nixos-anywhere, wipes OS disk)
./scripts/deploy kexec <config> <host> # RO-root box (ZimaOS): kexec into a RAM installer first
sudo ./scripts/deploy kexec-local [--yes] # kexec THIS box (no ssh); confirm prompt unless --yes
./scripts/deploy image mercury # build the aarch64 SD image
./scripts/deploy flash mercury /dev/sdX # build + write SD + drop the sops age key
Both password prompts are auto-filled from the "HomeLab" Proton Pass vault, keyed by
<config>, not <host> — items darman@<config> (sudo) and root@<config> (ssh).
A trashed Proton Pass item with the same title shadows the active one and yields an
empty password, so the script resolves the title among --filter-state active items
first; a plain pass-cli item view --item-title silently returns the trashed copy and
you get an interactive prompt with no explanation.
On terra/titan the values come from pass-broker (services/identity/pass-broker.nix),
not from anyone's own pass-cli session: it runs as its own user with an agent PAT scoped to
the HomeLab vaults and releases an item only after a Telegram tap. Prompts go through the
gateway on mars (hosts/mars/pass-broker-gateway.nix, tailnet :8097), the bot's only
getUpdates consumer: a second poller would split the taps. A new host gets its own
pass_broker_gateway_token in its sops file and that token's sha256 in the gateway's
hosts. When the socket exists, deploy uses it exclusively; a refused tap falls back to the
interactive prompt.
Claude: fetch Proton Pass items only via pass-broker get --item <title> --reason <why>,
never pass-cli login yourself, and never print a value you fetched. Values go into
variables or pipes, as scripts/deploy does.
HOMELAB_KEXEC_TARBALL (+ _CPIO / _GZIP) makes kexec/kexec-local reuse a
prebuilt installer instead of rebuilding ~500MB. The VM test below uses this.
Secrets (needs the admin age key at ~/.config/sops/age/keys.txt):
./scripts/edit_secrets secrets/<host>.yaml
Claude: never run sops --decrypt/edit_secrets --show and print the result — that
puts every plaintext secret in the file into the conversation transcript, not just the
one you wanted. To add or change a single value non-interactively, use
sops --set '["key"] "value"' secrets/<host>.yaml (quote the value as JSON), which
writes without ever displaying the file's existing contents.
Test a service config BEFORE touching hardware — always do this for nontrivial changes:
# x86 QEMU VM of mercury's DNS/DHCP stack (fast; validates pihole/unbound at runtime)
nix build .#nixosConfigurations.mercury-vm.config.system.build.vm -o result
./result/bin/run-mercury-vm-vm # ssh -p 2223 darman@localhost (pw: test)
# jupiter services as a VirtualBox OVA
nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
# end-to-end VM test of `deploy kexec-local` (~45s once the tarball is built)
nix build .#checks.x86_64-linux.kexec-local -L
# VM test of luna's app hosting on mars (hosts/mars/luna-sites.nix): podman socket
# proxy, registry validation, caddy routes, reboot persistence
nix build .#checks.x86_64-linux.luna-sites -L
# VM test of pass-broker with fake pass-cli + Telegram: approve/deny/expiry,
# grants, policy, uid check
nix build .#checks.x86_64-linux.pass-broker -L
# two-node VM test of the gateway on mars with a broker using it: labels,
# cancel, expiry, per-host token isolation, gateway down
nix build .#checks.x86_64-linux.pass-broker-gateway -L
checks.kexec-local is the only way to exercise kexec-local at all: it jumps the
machine you are typing at, so it cannot be rehearsed on real hardware and a failure
looks exactly like a slow boot. It asserts the box actually left the old kernel
(SSH drops then returns), came back as nixos-installer, lost its old /run, and
kept its ssh host key. Run it after ANY change to the kexec paths.
Secrets (sops-nix)
- Each
secrets/<host>.yamlis encrypted to the admin key (edit) + that host's key (runtime decrypt); rules in.sops.yaml. Private keys live OFF-repo:~/.config/sops/age/keys.txt(admin),~/.config/homelab/<host>/(host keys). - jupiter/neptun decrypt with their ssh host key (
ssh-to-agerecipient), shipped at install vianixos-anywhere --extra-files. - mercury (SD image, no
--extra-files) uses a dedicated age key at/var/lib/sops-nix/age.txt—./scripts/deploy flashwrites it to the ext4 root partition. - A service password that must come from sops but whose module has no
passwordFilehook (pihole, adguard) is injected viasops.templates→ an env file → the service (FTLCONF_*for pihole). Seehosts/mercury/configuration.nix.
Non-obvious gotchas (all learned the hard way)
- aarch64 (mercury): the x86 laptop needs
extra-platforms = aarch64-linuxin/etc/nix/nix.custom.conf(NOT/etc/nix/nix.conf— Determinate Nix regenerates that)qemu-user-static-binfmt, else emulated builds fail with "platform mismatch". Or build on the Pi with--build-host darman@<ip>.
- pihole on mercury is a CONTAINER (
services/network/pihole.nix, official image, host networking, caps NET_ADMIN/NET_RAW/SYS_NICE/CHOWN,FTLCONF_*env config). The nativeservices.pihole-ftlmodule segfaults on the Pi 3B+ aarch64 — do not switch back. services.unbound.resolveLocalQueries = falseis required: unbound listens on :5335, so leaving it true points the host's resolv.conf at 127.0.0.1:53 with nothing there → boot-time DNS deadlock (starves image pulls / list downloads). Host resolves via upstreamnetworking.nameservers; pihole forwards to unbound explicitly at127.0.0.1#5335.- Remote deploy pushes unsigned closures: hosts set
nix.settings.trusted-users = [ "root" "@wheel" ](in common.nix) so a laptop-built closure is accepted by the target. - jupiter:
boot.kernelParams = [ "reboot=pci" ](warm reboot hangs on that board); eMMC initrd modules pinned inconfiguration.nix(generate-config misses them); the 16TB×2 RAID0 data lives on/mnt/datawithnofail, kept OUT of disko (never wiped). - terra:
./scripts/deploy kexec-localhangs hard — do not use it there. Confirmed on real hardware: kexec'sdevice_shutdown()pass runs (SCSI disks sync fine in the log), then the machine goes dark and never comes back —journalctl --list-bootsshowed a ~15min gap before the next boot, i.e. a genuine hang needing a manual power cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT): discrete AMD GPUs are known to hang during kexec's device-shutdown pass with no clean handoff before the jump — same class of issue as jupiter'sreboot=pciworkaround, just fatal here instead of slow. Use./scripts/deploy install terra localhostinstead (README's "First install on terra" section) — it detects it isn't inside a live installer yet and reboots via a realsystemctl reboot+ systemd-boot one-shotfindiso=entry, not kexec. ./scripts/deploy install <config> localhost's behavior depends onuname -n(is_live_installer()): on a real running OS it buildsinstaller-iso, stages it locally, and reboots into it (local_install_prepare_and_reboot()); only insidenixos-installer(kexec) orhomelab-installer(installer-iso) does it actually run disko +nixos-install.installer-iso'shomelab-auto-install.servicecloses the loop: it readshomelab.install=<config>back off/proc/cmdline(set by the prepare step) and re-runs the identical command itself oncehomelab-checkout.servicehas cloned the repo — the whole reinstall is one command and unattended after the first reboot. It confirms (typeyes) before rebooting, likeflash/kexec-local;--yesskips that and is what the ISO passes itself. It always ASKS where to stage the iso file (never auto-picks — the wrong disk here is destroyed mid-install);HOMELAB_INSTALLER_STAGE_DIRskips the prompt for scripted use. Both this andkexec-localself-elevate viasudo(require_root()) rather than requiring you to prefix the command yourself.- terra runs Limine, not systemd-boot —
bootctl set-oneshotis useless there (bootctl statuslists✗ One-shot entry control, and CachyOS's pacman hooks regeneratelimine.confanyway).local_install_prepare_and_reboot()therefore picks its one-shot mechanism at runtime: a systemd-boot loader entry when$BOOT/loader/entriesexists, otherwisearm_efi_bootnext()— a temporary UEFI entry that EFI-stub-boots the kernel off the ESP, armed via the firmware'sBootNext. Created withefibootmgr --create-only(NOT--create, which pushes it to the front ofBootOrderand would make a wiped installer the permanent default if anything went wrong). BootNext is spent by that one boot, so a failed attempt still comes back on the normal bootloader. The EFI-stub path needs the kernel on the ESP itself, not on a separate XBOOTLDR — hence--print-esp-pathrather than--print-boot-pathin that mode.homelab-auto-install.servicedeletes the leftover NVRAM entry as soon as it boots; both it and the script match the labelHomelab InstallerEXACTLY (a prefix match would delete the Windows or Limine entry). installer-isomust forceboot.initrd.systemd.enable = false.findiso=is a SCRIPT-stage-1 feature (stage-1-init.shloop-mounts the file it points at and symlinks it to/dev/root). The systemd initrd — default since 26.05 — has NO findiso path: it mounts/isostraight from/dev/disk/by-label/<volumeID>(iso-image.nix), a label that only exists when the ISO is the physical boot medium. EFI-stub-booted off the ESP with the iso as a plain file on another fs, that label never appears; stage 1 mounts/sysrootfine, then times out on/sysroot/nix/.ro-storewaiting for/dev/disk/by-label/nixos-minimal-…and drops to an emergency shell. The wholeinstall <config> localhostfindiso path depends on script stage 1.- The staging-dir guard must fail CLOSED, and
findmntneeds--nofsroot: on btrfsfindmnt -no SOURCEprints/dev/sdb2[/@], whichlsblkcannot open, so a naive parent-device lookup comes back empty. Treating empty as "different disk" silently allowed staging the iso on the very disk disko then wiped — terra's CachyOS root is exactly that layout, so it hit the live case.disks_backing()(lsblk -rnso NAME,TYPE) returns EVERY whole-disk ancestor because LVM/RAID can span several (/mnt/ssd_01→ sdd + sde), and an empty result is a hard error, not a pass. btrfs staging is refused outright: stage-1 mounts a btrfs volume's TOP level, so an iso inside a subvolume is unreachable viafindiso=. findiso=must keep its leading slash: stage-1 tests[ -e "/findiso$isoPath" ], so stripping the mountpoint prefix off a stagedir whose mountpoint is/yieldsvar/tmp/x.iso→/findisovar/tmp/x.iso→ emergency shell, after you have already rebooted out of the working OS.- The auto-install needs the host key shipped to it, and a
$HOME: the ISO is built from a public repo with no credentials, solocal_install_prepare_and_reboot()copies the key onto the boot partition and passes that partition's PARTUUID ashomelab.keypart=; the service mounts it and drops the key in/root/.config/homelab/<config>/before running the install. Also, systemd does NOT set$HOMEfor a system service withoutUser=(SetLoginEnvironment=defaults to false), andscripts/deployruns underset -u— hence$KEYDIRinstead of a bare$HOME, plusEnvironment=HOME=/rooton the unit. installer-isomust enableexperimental-featuresitself.installation-cd-minimalleaves them unset, sonix runandnixos-install --flakeboth die with "experimental Nix feature 'nix-command' is disabled". The nixos-imageskexecinstaller sets them itself, which is why the sameinstall <config> localhostbranch worked afterkexec-localbut not from the ISO.- disko/nixos-anywhere run as
nix run .#disko/.#nixos-anywhere, from this flake's locked inputs — notnix run github:.... They execute while a disk is being wiped, so the revision must be the reviewed one inflake.lock, and it has to resolve without network. - disko wipes only the OS disk named in
hosts/<h>/disk-config.nix; data disks are plainfileSystemsinconfiguration.nix. nixos-anywhere/kexec needs a writable root; ZimaOS root is read-only, hence the./scripts/deploy kexecstep that streams a RAM installer (with static cpio/gzip since ZimaOS lacks them).kexec/runjumps ~6s AFTER it returns: nixos-images'kexec-run.shends withnohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &. So the staging dir must OUTLIVE the script — anrm -rfin an EXIT trap deletes the binary that performs the jump and the box silently stays on the old kernel.kexec-localclears its trap before jumping and then sleeps 60s on purpose. Covered bychecks.kexec-local.- The kexec installer KEEPS the box's ssh host key:
kexec-run.shcopies/etc/ssh/ssh_host_*into the appended initrd andrestore-remote-access.nixinstalls them back. So do NOTssh-keygen -Rafter a kexec — the key does not change, and clearing it just throws away the known_hosts record. kexec-localstages on/var/tmp, not/tmp:kexec-run.shappends a fresh cpio tokexec/initrdin place and execs binaries from that dir, so a size-capped ornoexectmpfs gives a half-written initrd or a bare "Permission denied".- sops-nix values must be STRINGS.
sops --set '["id"] 123'storestype:int; the build-time check only sees key names and passes, then activation's sops-install-secrets fails with "is not a string" and installs NONE of the new secrets. The old generation stays, so it looks like only the new keys are missing. Quote it:'["id"] "123"'. - A sops secret a USER service reads needs
owner, not a mode.luna-voiced(services/desktop/luna-voice.nix) runs as darman's user unit, and a user manager cannot read root:root 0400 —LoadCredentialdoes not bridge that gap either. Hencesops.secrets.<x>.owner = "darman"inhosts/terra/secrets.nix. elevenlabs_api_keyis in BOTHsecrets/mars.yamlandsecrets/terra.yaml— same value. terra calls ElevenLabs directly (the client-direct path Hermes documents) rather than relaying audio through mars, so rotating that key means editing two files.hermes_api_server_keyis duplicated for the same reason: mars serves the API, terra is its only client.- terra's mic is QUIET, and every voice threshold is sized for it. Measured on
the AIR 192|4 (
luna-voice level): ~0.001–0.002 RMS ambient, only 0.0045–0.005 for normal speech — about 6dB of separation. Consequences, all of which looked like "the daemon is broken": a VAD multiplier above ~2 can never fire, an absolute trigger floor of 0.006 sits above the user's voice, and whisper's own gate at 0.015 discarded every word the VAD did capture. HencevadMult = 1.8,vadFloor = 0.0025, andstt::normalizescaling the clip up (capped at 30x) before whisper sees it. Raising the gain on the interface is worth more than any of these; tune viahomelab.lunaVoice.vad*rather than editing Rust. try_recv()POPS — anif let Ok(Cancel)destroys every other command. luna-voiced polls for a cancel while she speaks; written as a singleif let, anysay/toggle/mode offarriving mid-reply was taken off the channel and dropped, so SUPER+V during her answer did nothing at all and the next turn never ran. Non-cancel commands are collected and re-sent into the channel after the turn instead (requeueinturn()).- A rising sound must not update the noise floor. Feeding every sample into the floor EMA lets speech drag its own threshold up faster than it rises — measured, a tone peaking at 0.029 was chased by a threshold that reached 0.018 and never triggered. Only samples below the current trigger update the floor.
- whisper invents words out of silence — "you", "Thank you.", "Bye." — and a
held push-to-talk key over a quiet room is long AND empty, so duration alone
does not catch it.
stt::has_speechgates on the loudest 100ms window instead; without it every stray keypress sends a hallucinated sentence to luna. - terra's greeter is a throwaway Hyprland running
dotfiles/quickshell/greeter.qml(services/desktop/quickshell-greeter.nix). It must exit after login or greetd never starts the session, and with a Lua confighyprctl dispatch exitis REJECTED — it needshyprctl dispatch 'hl.dsp.exit()'. Test the flow without touching the real greetd by running greetd'sfakegreet "qs -p …/greeter.qml"inside a nested Hyprland (useruser, passwordpassword, then answer9).