# homelab Flake-based NixOS config. Hosts: `jupiter` (ZimaBlade, NAS + services), `neptun` (netcup VPS: public reverse proxy, Authentik, headscale), `mercury` (Raspberry Pi 3B+, DNS/DHCP), `terra` (desktop), `mars` (on-site, single-purpose: Hermes Agent only). ## Structure ``` flake.nix # inputs + nixosConfigurations (jupiter, neptun, kexec, ...) common.nix # shared base: user, ssh, nix, firewall, timezone services/ # one reusable module per service, by category media/ jellyfin, audiobookshelf, the *arrs, sabnzbd, seerr, ... network/ caddy, samba, avahi, pihole, unbound vpn/ tailscale, headscale (control server), headplane (its web UI) identity/ authentik (OIDC provider, from the authentik-nix flake) dev/ gitea desktop/ hyprland containers.nix # podman backend, shared across categories hosts/ jupiter/ # ZimaBlade NAS configuration.nix # host bits + imports common + the services it runs disk-config.nix # disko: eMMC partitions hardware-configuration.nix secrets.nix # sops-nix wiring vm.nix # VirtualBox test image (jupiter-vbox) neptun/ # netcup public reverse proxy + tailnet node configuration.nix disk-config.nix hardware-configuration.nix secrets.nix mars/ # on-site, single-purpose: Hermes Agent only configuration.nix disk-config.nix hardware-configuration.nix secrets.nix hermes-agent.nix # Hermes Agent (moved here from jupiter) secrets/ # age-encrypted sops files, one per host scripts/ # deploy, edit_secrets ``` Hosts compose by importing `common.nix` + whichever `services/*` modules they run. Each service module opens its own firewall ports. ## Gitea events to Hermes Jupiter's Gitea registers one webhook per Hermes route, straight at Hermes on mars (`http://mars.orbit.sol:8644/webhooks/`), with no relay in between: | route | gitea hook event | wakes luna on | | --- | --- | --- | | `gitea-pr-comments` | `pull_request_comment` | a timeline comment on a PR | | `gitea-pr-reviews` | `pull_request_review` | a review with a body, or changes requested | Approvals cannot be excluded at the hook — `pull_request_review` is one switch for all three review types — so they are delivered and then dropped by the Hermes route, which does not list `pull_request_approved`. Expect them in gitea's delivery log answered 200/ignored; that is the design, not a failure. Gitea's `addDefaultHeaders` signs every webhook type with `X-Hub-Signature-256` in GitHub's exact format and sends `X-GitHub-Event` unconditionally — which is exactly what Hermes validates against the route secret and reads the event name from, so the two speak the same protocol without translation. The URL path is the Hermes route name, so another route is just another hook. Gitea will only deliver to hosts in `[security] ALLOWED_HOST_LIST`, which defaults to `external` and does NOT include tailnet addresses (100.64.0.0/10 is RFC 6598 carrier-grade NAT, neither private nor external as gitea classifies it). `services/dev/gitea.nix` sets it accordingly; without that, deliveries fail with `webhook can only call allowed HTTP servers`. Gitea spells the same event three ways, and two of the spellings collide. The hook's `events` array takes an *api* name (`updateHookEvents` in `routers/api/v1/utils/hook.go`), which is a coarser set than the internal `HookEventType`; `X-GitHub-Event`, which is what each Hermes route matches its `events` against, carries a lossy *wire* name from `HookEventType.Event()`: | HookEventType | wire (mars route) | api (gitea hook) | | --- | --- | --- | | `issue_comment` | `issue_comment` | `issue_comment` | | `pull_request_comment` | `issue_comment` | `pull_request_comment` | | `pull_request_review_comment` | `pull_request_comment` | `pull_request_review` | | `pull_request_review_rejected` | `pull_request_rejected` | `pull_request_review` | | `pull_request_review_approved` | `pull_request_approved` | `pull_request_review` | Watch the api column: `updateHookEvents` **silently ignores strings it does not recognise**, so a plausible-looking name that is a valid `HookEventType` but not a valid api event leaves the hook registered with no events at all — no error, no deliveries. Check a new hook's event list in the UI after adding it. So `services/dev/gitea.nix` and `hosts/mars/hermes-agent.nix` deliberately name the same event differently, and neither is a typo. `X-GitHub-Event-Type` carries the subscription name, but Hermes does not read it. Each route's prompt and filter script live in `hosts/mars/`. The filters are bind-mounted read-only from the nix store so the agent cannot edit her own loop guard out; run `python3 hosts/mars/gitea-pr-comment-filter-test.py` and `python3 hosts/mars/gitea-pr-review-filter-test.py` after editing either. `hermes-agent-webhook-routes` writes the routes into `~/.hermes/webhook_subscriptions.json` directly, host-side, rather than calling `hermes webhook subscribe`. That CLI has no `--toolsets` flag, and without a toolset override a webhook run gets Hermes's constrained default (`web_search`, `web_extract`, `vision_analyze`, `clarify`) — no shell, no file access, so neither prompt can actually be carried out. Upstream's documented answer is to add the `toolsets` key to that file by hand, which does not survive a re-provision, so the whole route definition lives in nix instead. The grant (`terminal`, `file`, `web`) is therefore deliberate and restored on every start — but note it is not *enforced*: that file sits inside `HERMES_WRITE_SAFE_ROOT`, so luna can widen her own toolset until the unit next runs. The real backstop is gitea's branch protection on `master`. Routes the unit does not name are left untouched, so retiring one is a manual `sudo podman exec hermes-agent hermes webhook remove ` on mars — and likewise its hook in the repo's Settings → Webhooks. Before deploying either host, add the same random `gitea_hermes_webhook_secret` value to both `secrets/mars.yaml` and `secrets/jupiter.yaml` using `scripts/edit_secrets`, with no trailing newline — a newline would change the key the HMAC is computed with, and the two ends would disagree. The value is intentionally not included in the repository. ## Test in VirtualBox (no hardware needed) ``` nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA VBoxManage import result/*.ova --vsys 0 --vmname jupiter-vbox VBoxManage startvm jupiter-vbox --type headless ``` Login `darman` / `test`. Forward ports with `VBoxManage modifyvm ... --natpf1`. ## First install on the ZimaBlade — nixos-anywhere + disko Wipes the OS disk and installs the flake over SSH. No USB needed if the box already runs Linux (ZimaOS) reachable by root SSH — nixos-anywhere kexecs into an installer, partitions via disko, installs. > ⚠️ The OS disk in `disk-config.nix` is WIPED. Set `device` to the OS disk > ONLY (by-id). Back up / physically identify the NAS data disk first — it must > NOT appear in disko. `lsblk -o NAME,SERIAL,SIZE,MODEL` to identify. 1. Set the real OS disk id in `hosts/jupiter/disk-config.nix` (`ls -l /dev/disk/by-id`), and the data-disk mount in `configuration.nix`. 2. Add your login SSH pubkey to `users.users.darman.openssh.authorizedKeys.keys`. 3. Set the real samba password: ``` export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt nix shell nixpkgs#sops -c sops secrets/jupiter.yaml # edit, commit ``` 4. Stage the pre-generated host key so sops can decrypt on boot #1 (private key lives off-repo in `~/.config/homelab/jupiter/`): ``` install -Dm600 ~/.config/homelab/jupiter/ssh_host_ed25519_key \ /tmp/extra/etc/ssh/ssh_host_ed25519_key install -Dm644 ~/.config/homelab/jupiter/ssh_host_ed25519_key.pub \ /tmp/extra/etc/ssh/ssh_host_ed25519_key.pub ``` 5. Run from your laptop: ``` nix run github:nix-community/nixos-anywhere -- \ --flake .#jupiter \ --extra-files /tmp/extra \ --generate-hardware-config nixos-generate-config ./hosts/jupiter/hardware-configuration.nix \ --target-host root@ ``` `--extra-files` plants the host key before first boot (its age identity is already a recipient in `.sops.yaml`, so `/run/secrets/samba_password` decrypts on boot #1). `--generate-hardware-config` pulls the target's real kernel modules into the placeholder. Commit the result. Reboot into NixOS. Manual alternative (USB ISO): boot installer, `disko` the disk, then `nixos-install --flake .#jupiter`. ## First install on mars mars is an older x86_64 box (unknown provenance, "got from work"), on-site, running Hermes Agent only (see `hosts/mars/hermes-agent.nix` — moved there from jupiter). Its age recipient, host key (`~/.config/homelab/mars/ssh_host_ed25519_key`), and `secrets/mars.yaml` are already set up, with `darman_password`/`samba_password`/`opencode_go_api_key`/ `telegram_bot_token`/`hermes_dashboard_oidc_client_secret` carried over from jupiter's old instance. Two things are still placeholders and MUST be filled in before installing: 1. **OS disk id** in `hosts/mars/disk-config.nix` (`ls -l /dev/disk/by-id` once you have console/installer access on the box) — same `REPLACE-ME` in `hosts/mars/configuration.nix`'s comment refers to the same disk, but only `disk-config.nix`'s `device` actually needs editing (grub's own device list comes from disko, see that file's comment). 2. **`tailscale_authkey`** in `secrets/mars.yaml` — generate a fresh one (see "Bootstrap the tailnet" under neptun below) rather than reusing an old key; reusable pre-auth keys still expire. Boot mode is assumed **legacy BIOS** (grub, not systemd-boot) — unconfirmed; check `[ -d /sys/firmware/efi ]` once you're at the machine and see `hosts/mars/disk-config.nix`'s header comment if it turns out to be UEFI. Otherwise the flow is identical to the ZimaBlade steps above: ``` nix run github:nix-community/nixos-anywhere -- \ --flake .#mars \ --extra-files /tmp/extra \ --generate-hardware-config nixos-generate-config ./hosts/mars/hardware-configuration.nix \ --target-host root@ ``` (stage the host key into `/tmp/extra/etc/ssh/` first, same as step 4 there). Manual alternative (USB ISO): boot installer, `disko` the disk, then `nixos-install --flake .#mars`. ## First install on terra — no-USB findiso reinstall (replacing CachyOS) terra is a Ryzen 9 5900X / Radeon RX 6800 XT desktop, currently running CachyOS with a writable root and **Limine** as its bootloader (not systemd-boot — see step 4). Everything is already prepped in this repo: real OS-disk id in `disk-config.nix`, real login pubkey in `common.nix`, terra's age recipient in `.sops.yaml`, its host key pre-generated at `~/.config/homelab/terra/`, and `secrets/terra.yaml` already holds real `darman_password` / `tailscale_authkey` values. Nothing to fill in — just run it. > ⚠️ **`./scripts/deploy kexec-local` does NOT work on terra — do not use it.** > Confirmed on real hardware: the jump hangs completely (kexec's own > `device_shutdown()` pass runs — SCSI disks sync fine — then the machine goes > dark and never comes back; `journalctl --list-boots` showed a **~15 minute** > gap before the next boot, i.e. a hard hang needing a manual power cycle, not > a slow jump). Near-certainly amdgpu: discrete AMD GPUs are known to hang > during kexec's device-shutdown pass with no clean way to hand control back > before the jump — same class of issue as jupiter's `reboot=pci` warm-reboot > workaround, just fatal here instead of merely slow. The path below instead > triggers a real ACPI reboot through firmware POST — a materially different > code path that never runs kexec's device-shutdown pass at all. > ⚠️ The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The > dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs > disks are not in disko and are untouched — but double check with > `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have > changed since `disk-config.nix` was written. One command does the whole thing — no need to `sudo` it yourself, it self-elevates: ``` ./scripts/deploy install terra localhost ``` `scripts/deploy` detects it isn't already inside a live installer (checks `uname -n`) and instead: 1. Asks where to stage the iso file — never auto-picks, because the wrong disk here is destroyed mid-install (`HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for scripted use). It **refuses** if that path resolves to a disk `disk-config.nix` is about to wipe, if it can't work out which physical disk the path is on at all (fail-closed — LVM and RAID can span several), or if it's on btrfs (stage-1 mounts a btrfs volume's *top level*, so a path inside a subvolume never resolves and you boot to an emergency shell). On terra, `/mnt/hdd_01` is the right answer; the CachyOS root is btrfs on the OS disk and is rejected on both counts. 2. Prints exactly what it is about to do — OS disk, staging disk, boot entry — and waits for you to type `yes`. `--yes` skips it; that is what the ISO passes when it re-runs the command itself. 3. Builds `installer-iso`'s kernel + initrd + iso image, checks both target partitions have room, then copies the kernel/initrd **and terra's pre-generated ssh host key** to the boot partition (found via `bootctl --print-boot-path`, not assumed to be `/boot`) and the iso to the staging dir. 4. Arms a **one-shot** boot of it with `findiso=` + `homelab.install=terra` + `homelab.keypart=` on the kernel cmdline, and reboots — a real `systemctl reboot`, not kexec. Two mechanisms, picked automatically: - **systemd-boot** (jupiter, neptun, and terra once NixOS is on it): a `bootctl set-oneshot` loader entry. - **anything else** — terra today runs Limine, which reports `One-shot entry control: ✗` and has no equivalent: the firmware's own **`BootNext`**, pointing at a temporary UEFI entry that EFI-stub-boots the kernel straight off the ESP. Created with `--create-only` so it never enters `BootOrder`, which means it is reachable exactly once and nothing else changes. Either way the box falls back to its normal bootloader if the attempt fails — nothing is made permanent before the install succeeds. The temporary UEFI entry is deleted by the installer as soon as it boots. The booted installer clones the repo (`homelab-checkout.service`, needs network — it's public now, no credentials involved) and then `homelab-auto-install.service` reads `homelab.install=terra` back off `/proc/cmdline`, mounts `homelab.keypart=` to pick terra's host key back up into `/root/.config/homelab/terra/`, and re-runs the exact same `./scripts/deploy install terra localhost` itself — now genuinely inside the installer, so it takes the disko + `nixos-install` branch instead of preparing again. That key is what seeds `/etc/ssh` on the new system, which is what lets `/run/secrets/*` decrypt on boot #1; it has to travel this way because the ISO is built from a **public** repo and deliberately carries no credentials. The copy on the boot partition dies with the disko wipe minutes later. The whole thing is unattended after the initial reboot; ssh into `homelab-installer` (same pubkey as the ISO everywhere else) to watch progress — `journalctl -u homelab-checkout -u homelab-auto-install -f`. > The checkout step resolves `git.mgaction.town`, which goes through mercury's > pihole on the LAN. If mercury is down, the installer boots fine but never > gets the repo — fix DNS and `systemctl restart homelab-checkout`. When it's done, reboot again into the freshly installed NixOS. Then, same as any other host: ``` ssh darman@terra sudo -v # DO NOT SKIP — see below ``` `darman` is created with `mutableUsers = true`, so `/etc/shadow` is written **once**. If the sops secret wasn't readable at that moment the account gets `!` (locked) permanently and no `deploy switch` will fix it — verify sudo works while you still have physical console access as a fallback. ## Deploy (the `./deploy` wrapper) All arguments mandatory — no default host, no default config. ``` ./deploy kexec # headless kexec into a RAM installer (RO-root box) ./deploy install [--yes] # first install; wipes OS disk, ships host key ./deploy switch # rebuild + activate on a running host ./deploy boot|test # stage for next boot / activate without boot entry ./deploy image # build an SD-card image (mercury) ./deploy flash # build SD image, write it, drop the sops age key ``` `switch`/`boot`/`test` prompt for darman's password (`wheelNeedsPassword`). `` is a `nixosConfigurations` name (`jupiter`, `neptun`). Its pre-generated SSH host key lives at `~/.config/homelab//ssh_host_ed25519_key`. Examples: ``` ./deploy switch jupiter jupiter.sol ./deploy install neptun 159.195.64.117 ``` Rollback: `nixos-rebuild switch --rollback` on the host, or pick a prior generation at boot. ## Post-deploy steps (per host) Things the flake cannot do for you. Skipping these leaves a host that builds and boots but doesn't work. ### Every host, immediately after a first install ``` ssh darman@ sudo -v # DO NOT SKIP ``` `users.mutableUsers` is `true`, so `/etc/shadow` is written **once**, when the user is created. If the sops secret wasn't readable at that moment the account gets `!` (locked) permanently — `deploy switch` will never fix it, because the activation script only sets a password for users not already in `/etc/shadow`. Combined with `wheelNeedsPassword = true` and `PermitRootLogin = "no"` that means no way to escalate, and recovery is physical: netcup's rescue system for neptun, or pulling the SD card for mercury. Verify sudo while you still have another way in. ### neptun (netcup VPS) 1. **Edge firewall.** In netcup's panel, inbound `ACCEPT` for TCP 22/80/443/2222 **and a rule accepting inbound UDP**. The firewall is stateless: without the UDP rule every DNS and NTP *reply* is dropped, and nothing on the box reports an error — it looks like headscale crash-looping on its DERP fetch and Caddy failing ACME. `grep -A1 '^Udp:' /proc/net/snmp` showing `InDatagrams 0` is the tell. Rules apply on VM restart, not on save. This is safe: `nixos-fw` is stateful and default-deny, so it remains the real policy. Also open UDP 3478 (STUN) and 41641 (tailscale direct). 2. **Authentik** creates `akadmin` on first start; log in at `https://auth.mgaction.town` with `authentik_bootstrap_password` from sops. The username is hardcoded upstream and the bootstrap runs once — later changes to the env vars are ignored. To use your own admin instead: create a user, add it to the **`authentik Admins`** group (superuser is a *group* flag in Authentik, there is no per-user one), verify it works in a private window, then **deactivate** `akadmin` — do not rename or delete it. The bootstrap blueprint keys on `username: akadmin` with `state: created`, so if no user by that name exists it simply makes a new one on the next reconcile. 3. **Bootstrap the tailnet** (headscale starts with an empty database): ``` sudo headscale users create darman sudo headscale preauthkeys create --user darman --reusable --expiration 24h ``` Put that key in **every** host's sops file as `tailscale_authkey` and rebuild. 4. **Headplane API key** — defaults to 90d, after which headplane silently stops listing nodes: ``` sudo headscale apikeys create --expiration 999d # -> headplane_headscale_api_key ``` 5. **Headplane OIDC.** In Authentik create an OAuth2/OpenID provider (confidential, redirect `https://vpn.mgaction.town/admin/oidc/callback`, **a signing key must be selected** or discovery exposes no JWKS) and an application with slug **`headplane`** — the slug is what makes the issuer `.../application/o/headplane/` in `services/vpn/headplane.nix`. Client ID goes in that file, client secret into sops. 6. **Headscale OIDC** (optional — pre-auth keys work without it). A *second* Authentik provider/application, slug **`headscale`**, redirect `https://vpn.mgaction.town/oidc/callback` (headscale's own, not headplane's under `/admin`). Client ID in `services/vpn/headscale.nix`, secret into sops as `headscale_oidc_client_secret`. ⚠️ headscale runs OIDC discovery **at startup and a failure is fatal** — an issuer pointing at an application that doesn't exist yet means the control server won't boot, taking the whole tailnet's control plane with it. Always verify first: ``` curl -s https://auth.mgaction.town/application/o/headscale/.well-known/openid-configuration ``` Users created by OIDC login are distinct from `headscale users create` ones: headplane matches the OIDC `sub` claim against the user's `providerId`, CLI-made users have none, and 0.28 dropped both `map_legacy_users` and node reassignment — so moving an existing node to an OIDC user means re-enrolling it. ### jupiter - `chown -R gitea:gitea /mnt/data/AppData/gitea` after the first deploy (the repos were copied in over CIFS as `darman:users`). - **Re-enrolling after the headscale database was recreated:** `tailscaled` keeps its old node key and reports `Running`, and the autoconnect unit exits early on that state without ever sending the new pre-auth key. Force it: ``` sudo tailscale logout && sudo systemctl restart tailscaled-autoconnect ``` ### mars - **Confirm the OIDC redirect still resolves.** hermes-agent.nix reuses jupiter's old Authentik application (slug `hermes`, redirect `https://hermes.mgaction.town/auth/callback`) unchanged — nothing to reconfigure in Authentik, just verify `neptun`'s `hermes.mgaction.town` vhost (now pointed at `mars.orbit.sol:9119`) actually reaches the dashboard once mars is up and joined the tailnet. - **Carrying forward old chat history/memories:** mars starts with a fresh Hermes state dir (`/var/lib/hermes/.hermes`). jupiter's old instance data is backed up at `/mnt/data/AppData/hermes.bak-2026-08-21` — rsync it over (via the `/mnt/jupiter` samba mount) before the first switch if you want it preserved instead of starting clean. ### Obsidian vaults (jupiter CouchDB + mars bridge) CouchDB itself is fully declarative (`services/dev/obsidian-livesync.nix`), but three things are runtime state it cannot own. **1. Each vault's database is created by the plugin.** Point Self-hosted LiveSync at `https://notes.mgaction.town` (URI field) with the database name in its own field — *not* as a path on the URI. Turn on End-to-End Encryption and Obfuscate Properties **before the first sync**; both are remote-format decisions and changing them later means converting or rebuilding the database. The passphrase lives in the HomeLab Proton Pass vault, never in sops — it is what keeps a publicly reachable database from being a readable one. Database names must start with a lowercase letter (`a-z0-9_$()+-` after that). An illegal name is rejected by neptun's matcher rather than CouchDB, and shows up in Obsidian as a connection failure with **no error message at all**. **2. luna's vault credentials on mars.** `hosts/mars/secrets.nix` needs two values before mars will activate: `couchdb_luna_password` and `obsidian_luna_passphrase`. ``` sops --set '["couchdb_luna_password"] ""' secrets/mars.yaml sops --set '["obsidian_luna_passphrase"] ""' secrets/mars.yaml ``` Keep both alphanumeric. sops substitutes into already-rendered JSON, so a `"` or `\` in either produces an invalid `config.json`; the bridge logs `Could not parse configuration!` and then runs on with **zero peers** instead of exiting, which looks exactly like a bridge that is simply idle. As set up today these are the `obsidian` admin password and the same passphrase as the personal vault, which means mars — the box running an autonomous agent — can decrypt and read every vault database. Optional hardening, either half independently: ``` # password comes straight out of sops; never echo it LUNA_PW=$(sops --decrypt --extract '["couchdb_luna_password"]' secrets/mars.yaml) ADMIN=obsidian # prompts for the admin password curl -u "$ADMIN" -X PUT http://jupiter.orbit.sol:5984/_users/org.couchdb.user:luna \ -H 'Content-Type: application/json' \ -d "{\"name\":\"luna\",\"type\":\"user\",\"roles\":[],\"password\":\"$LUNA_PW\"}" curl -u "$ADMIN" -X PUT http://jupiter.orbit.sol:5984/luna_wiki/_security \ -H 'Content-Type: application/json' \ -d '{"admins":{"names":[],"roles":[]},"members":{"names":["luna"],"roles":[]}}' unset LUNA_PW ``` then set `username` in `hosts/mars/livesync-bridge.nix` to `luna` and put that account's password in `couchdb_luna_password`. Run it against jupiter over the tailnet — `/_users` is blocked on the public vhost on purpose. A vault-specific passphrase is the other half, changed in the plugin and mirrored into sops. **3. The database name must match.** `database` in `hosts/mars/livesync-bridge.nix` has to be exactly the name entered in the plugin. A mismatch does not error — with an admin credential PouchDB simply creates the misnamed database and replicates an empty vault into it. Order matters: set the vault up from Obsidian first so the database exists and carries the plugin's own tweaks, then deploy mars. Afterwards: ``` systemctl status livesync-bridge # on mars cat /var/lib/livesync-bridge/health.json # per-peer ok/backendUp/detail ls /var/lib/livesync-bridge/vault # her notes, as real markdown ``` The vault is mounted into the agent container at `/opt/data/vault`, inside `HERMES_WRITE_SAFE_ROOT`, so luna can write as well as read. A note luna writes reaches CouchDB as soon as the bridge sees it, but whether it then reaches your devices depends on that vault's **Sync Mode** in the plugin. Only "LiveSync (real-time)" pulls continuously; the periodic/on-save presets need their timer or a manual **Replicate**. A file that appears only after clicking Replicate is the client waiting, not the bridge failing — the database already had it. Check the bridge's own side in the journal: ``` journalctl -u livesync-bridge | grep -- '--> luna-remote' ``` ⚠️ **Verify her writes actually land before trusting this.** Upstream has three open issues on the storage→CouchDB direction (#50, #23, #46) and all fail silently — the log reports the upload and the database never updates. Create a note as luna, confirm it appears on a phone, and re-check after any input bump. ### mercury (Raspberry Pi 3B+) - `./deploy flash mercury /dev/sdX` writes the dedicated age key to the root partition. Without `~/.config/homelab/mercury/age.txt` it silently skips that step and **no secret decrypts on the box** — check `ls /run/secrets` after first boot. - It boots from an SD card, so config changes are `./deploy switch mercury ` (an aarch64 build — needs `extra-platforms` + binfmt on the laptop, see the gotchas in `CLAUDE.md`) rather than a reflash. - **Suspect the card first** when binaries crash with `Illegal instruction` or services fail inexplicably. Failing flash returns corrupt data with no I/O errors in `dmesg`: ``` sudo nix-store --verify --check-contents # add --repair to fix ``` A card that has corrupted one path will corrupt more. Replace it and reflash. - **A reflash wipes `/var/lib/pihole`**, taking the gravity database with it. The blocklists themselves are declared in `services/network/pihole.nix`, and the `pihole-adlists` unit re-seeds them on boot and rebuilds gravity when it finds it empty — so this heals itself, but the first boot after a reflash spends several minutes downloading lists. Query history and dynamic DHCP leases are genuinely lost (static leases are declarative). Check with: ``` systemctl status pihole-adlists sudo podman exec pihole pihole-FTL sqlite3 /etc/pihole/gravity.db \ "SELECT address,enabled FROM adlist; SELECT COUNT(*) FROM gravity;" ``` `Blocked DNS queries: 0` in the pihole logs means gravity is empty — DNS resolves fine, nothing is filtered. - mercury's own `resolv.conf` is deliberately public resolvers, not its own pihole (`resolveLocalQueries = false`, see `CLAUDE.md`) — so `.sol` names do not resolve *on mercury itself*. That is expected, not a fault. - **mercury is load-bearing for the whole tailnet's DNS.** headscale sets `override_local_dns = true` with pihole as the only global nameserver, so every node — including a phone on mobile data — resolves through it and gets ad blocking and `.sol` names anywhere. The flip side is that mercury (or the home connection) going down costs name resolution on every device, not just `.sol`. Recovery on a stranded device is turning Tailscale off. There is deliberately no public fallback in `nameservers.global`: tailscale treats that list as a set, so a second entry would let queries slip past the filter whenever mercury is slow. neptun and mercury opt out with `--accept-dns=false` — mercury because it would otherwise resolve through itself, neptun because a public reverse proxy must not depend on a Pi at home to renew its certificates. - Tailnet names are `*.orbit.sol`, LAN names are `*.sol`. Both work everywhere on the tailnet because tailscale matches DNS routes by **longest suffix**, so `orbit.sol` reaches MagicDNS even though everything else goes to pihole. Never name a LAN host `orbit`: pihole's `address=/.sol/` lines match a name *and everything beneath it*, which would swallow the entire tailnet zone. ## Adding a service Copy the `whoami` block in `oci-containers.containers`, swap image/ports/volumes. Native NixOS module exists for many apps (Nextcloud, Jellyfin, Grafana...) — prefer `services.` over a container when available. Add a `caddy` `virtualHosts` block to expose it. ## Notes - Backend is Podman with `dockerCompat` — `docker` CLI works, no daemon. - Samba keeps its own password DB. `services.samba` never sets it; a systemd oneshot (`samba-smbpasswd`) provisions it. Host reads the password from `/run/secrets/samba_password` (**sops-nix**); the VM falls back to plaintext `/etc/samba/smb-password`. - Secrets: `secrets/jupiter.yaml` is age-encrypted (safe to commit) to two recipients in `.sops.yaml` — the **admin** key (edit on laptop, `~/.config/sops/age/keys.txt`) and the **jupiter host** key (derived from its SSH host key via `ssh-to-age`, decrypts at runtime). Private keys live off-repo and are gitignored. Rotate/add recipients with `sops updatekeys`. - Data disk: plain `fileSystems."/mnt/data"` in configuration.nix — kept out of disko so it is never formatted. Reference by `by-id` / `by-uuid`. - `system.stateVersion` = `26.05`, install-time schema. Do NOT bump on upgrades. - Terraform is not used: a single bare-metal box has no provider API. disko + nixos-anywhere cover provisioning natively.