homelab/README.md

# homelab

Flake-based NixOS config. Hosts: `jupiter` (ZimaBlade, NAS + services),
`neptun` (netcup VPS: public reverse proxy, Authentik, headscale),
`mercury` (Raspberry Pi 3B+, DNS/DHCP), `terra` (desktop), `mars` (on-site,
single-purpose: Hermes Agent only).

## Structure

```
flake.nix                  # inputs + nixosConfigurations (jupiter, neptun, kexec, ...)
common.nix                 # shared base: user, ssh, nix, firewall, timezone
services/                  # one reusable module per service, by category
  media/     jellyfin, audiobookshelf, the *arrs, sabnzbd, seerr, ...
  network/   caddy, samba, avahi, pihole, unbound
  vpn/       tailscale, headscale (control server), headplane (its web UI)
  identity/  authentik (OIDC provider, from the authentik-nix flake)
  dev/       gitea
  desktop/   hyprland
  containers.nix           # podman backend, shared across categories
hosts/
  jupiter/                 # ZimaBlade NAS
    configuration.nix      #   host bits + imports common + the services it runs
    disk-config.nix        #   disko: eMMC partitions
    hardware-configuration.nix
    secrets.nix            #   sops-nix wiring
    vm.nix                 #   VirtualBox test image (jupiter-vbox)
  neptun/                     # netcup public reverse proxy + tailnet node
    configuration.nix  disk-config.nix  hardware-configuration.nix  secrets.nix
  mars/                     # on-site, single-purpose: Hermes Agent only
    configuration.nix  disk-config.nix  hardware-configuration.nix  secrets.nix
    hermes-agent.nix          #   Hermes Agent (moved here from jupiter)
secrets/                   # age-encrypted sops files, one per host
scripts/                   # deploy, edit_secrets
```

Hosts compose by importing `common.nix` + whichever `services/*` modules they
run. Each service module opens its own firewall ports.

## Gitea events to Hermes

Jupiter's Gitea registers one webhook per Hermes route, straight at Hermes on
mars (`http://mars.orbit.sol:8644/webhooks/<route>`), with no relay in between:

| route | gitea hook event | wakes luna on |
| --- | --- | --- |
| `gitea-pr-comments` | `pull_request_comment` | a timeline comment on a PR |
| `gitea-pr-reviews` | `pull_request_review` | a review with a body, or changes requested |

Approvals cannot be excluded at the hook — `pull_request_review` is one switch
for all three review types — so they are delivered and then dropped by the
Hermes route, which does not list `pull_request_approved`. Expect them in
gitea's delivery log answered 200/ignored; that is the design, not a failure.
Gitea's `addDefaultHeaders` signs every webhook type with
`X-Hub-Signature-256` in GitHub's exact format and sends `X-GitHub-Event`
unconditionally — which is exactly what Hermes validates against the route
secret and reads the event name from, so the two speak the same protocol
without translation. The URL path is the Hermes route name, so another route
is just another hook.

Gitea will only deliver to hosts in `[security] ALLOWED_HOST_LIST`, which
defaults to `external` and does NOT include tailnet addresses
(100.64.0.0/10 is RFC 6598 carrier-grade NAT, neither private nor external as
gitea classifies it). `services/dev/gitea.nix` sets it accordingly; without
that, deliveries fail with `webhook can only call allowed HTTP servers`.

Gitea spells the same event three ways, and two of the spellings collide. The
hook's `events` array takes an *api* name (`updateHookEvents` in
`routers/api/v1/utils/hook.go`), which is a coarser set than the internal
`HookEventType`; `X-GitHub-Event`, which is what each Hermes route matches its
`events` against, carries a lossy *wire* name from `HookEventType.Event()`:

| HookEventType | wire (mars route) | api (gitea hook) |
| --- | --- | --- |
| `issue_comment` | `issue_comment` | `issue_comment` |
| `pull_request_comment` | `issue_comment` | `pull_request_comment` |
| `pull_request_review_comment` | `pull_request_comment` | `pull_request_review` |
| `pull_request_review_rejected` | `pull_request_rejected` | `pull_request_review` |
| `pull_request_review_approved` | `pull_request_approved` | `pull_request_review` |

Watch the api column: `updateHookEvents` **silently ignores strings it does not
recognise**, so a plausible-looking name that is a valid `HookEventType` but
not a valid api event leaves the hook registered with no events at all — no
error, no deliveries. Check a new hook's event list in the UI after adding it.

So `services/dev/gitea.nix` and `hosts/mars/hermes-agent.nix` deliberately name
the same event differently, and neither is a typo. `X-GitHub-Event-Type`
carries the subscription name, but Hermes does not read it.

Each route's prompt and filter script live in `hosts/mars/`. The filters are
bind-mounted read-only from the nix store so the agent cannot edit her own
loop guard out; run
`python3 hosts/mars/gitea-pr-comment-filter-test.py` and
`python3 hosts/mars/gitea-pr-review-filter-test.py` after editing either.

`hermes-agent-webhook-routes` writes the routes into
`~/.hermes/webhook_subscriptions.json` directly, host-side, rather than
calling `hermes webhook subscribe`. That CLI has no `--toolsets` flag, and
without a toolset override a webhook run gets Hermes's constrained default
(`web_search`, `web_extract`, `vision_analyze`, `clarify`) — no shell, no file
access, so neither prompt can actually be carried out. Upstream's documented
answer is to add the `toolsets` key to that file by hand, which does not
survive a re-provision, so the whole route definition lives in nix instead.
The grant (`terminal`, `file`, `web`) is therefore deliberate and restored on
every start — but note it is not *enforced*: that file sits inside
`HERMES_WRITE_SAFE_ROOT`, so luna can widen her own toolset until the unit
next runs. The real backstop is gitea's branch protection on `master`.

Routes the unit does not name are left untouched, so retiring one is a manual
`sudo podman exec hermes-agent hermes webhook remove <name>` on mars — and
likewise its hook in the repo's Settings → Webhooks.

Before deploying either host, add the same random
`gitea_hermes_webhook_secret` value to both `secrets/mars.yaml` and
`secrets/jupiter.yaml` using `scripts/edit_secrets`, with no trailing newline
— a newline would change the key the HMAC is computed with, and the two ends
would disagree. The value is intentionally not included in the repository.

## Test in VirtualBox (no hardware needed)

```
nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
VBoxManage import result/*.ova --vsys 0 --vmname jupiter-vbox
VBoxManage startvm jupiter-vbox --type headless
```
Login `darman` / `test`. Forward ports with `VBoxManage modifyvm ... --natpf1`.

## First install on the ZimaBlade — nixos-anywhere + disko

Wipes the OS disk and installs the flake over SSH. No USB needed if the box
already runs Linux (ZimaOS) reachable by root SSH — nixos-anywhere kexecs into
an installer, partitions via disko, installs.

> ⚠️  The OS disk in `disk-config.nix` is WIPED. Set `device` to the OS disk
> ONLY (by-id). Back up / physically identify the NAS data disk first — it must
> NOT appear in disko. `lsblk -o NAME,SERIAL,SIZE,MODEL` to identify.

1. Set the real OS disk id in `hosts/jupiter/disk-config.nix`
   (`ls -l /dev/disk/by-id`), and the data-disk mount in `configuration.nix`.
2. Add your login SSH pubkey to `users.users.darman.openssh.authorizedKeys.keys`.
3. Set the real samba password:
   ```
   export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt
   nix shell nixpkgs#sops -c sops secrets/jupiter.yaml   # edit, commit
   ```
4. Stage the pre-generated host key so sops can decrypt on boot #1
   (private key lives off-repo in `~/.config/homelab/jupiter/`):
   ```
   install -Dm600 ~/.config/homelab/jupiter/ssh_host_ed25519_key \
     /tmp/extra/etc/ssh/ssh_host_ed25519_key
   install -Dm644 ~/.config/homelab/jupiter/ssh_host_ed25519_key.pub \
     /tmp/extra/etc/ssh/ssh_host_ed25519_key.pub
   ```
5. Run from your laptop:
   ```
   nix run github:nix-community/nixos-anywhere -- \
     --flake .#jupiter \
     --extra-files /tmp/extra \
     --generate-hardware-config nixos-generate-config ./hosts/jupiter/hardware-configuration.nix \
     --target-host root@<zimablade-ip>
   ```
   `--extra-files` plants the host key before first boot (its age identity is
   already a recipient in `.sops.yaml`, so `/run/secrets/samba_password`
   decrypts on boot #1). `--generate-hardware-config` pulls the target's real
   kernel modules into the placeholder. Commit the result. Reboot into NixOS.

Manual alternative (USB ISO): boot installer, `disko` the disk, then
`nixos-install --flake .#jupiter`.

## First install on mars

mars is an older x86_64 box (unknown provenance, "got from work"), on-site,
running Hermes Agent only (see `hosts/mars/hermes-agent.nix` — moved there
from jupiter). Its age recipient, host key
(`~/.config/homelab/mars/ssh_host_ed25519_key`), and `secrets/mars.yaml` are
already set up, with `darman_password`/`samba_password`/`opencode_go_api_key`/
`telegram_bot_token`/`hermes_dashboard_oidc_client_secret` carried over from
jupiter's old instance. Two things are still placeholders and MUST be filled
in before installing:

1. **OS disk id** in `hosts/mars/disk-config.nix` (`ls -l /dev/disk/by-id`
   once you have console/installer access on the box) — same `REPLACE-ME` in
   `hosts/mars/configuration.nix`'s comment refers to the same disk, but only
   `disk-config.nix`'s `device` actually needs editing (grub's own device list
   comes from disko, see that file's comment).
2. **`tailscale_authkey`** in `secrets/mars.yaml` — generate a fresh one
   (see "Bootstrap the tailnet" under neptun below) rather than reusing an
   old key; reusable pre-auth keys still expire.

Boot mode is assumed **legacy BIOS** (grub, not systemd-boot) — unconfirmed;
check `[ -d /sys/firmware/efi ]` once you're at the machine and see
`hosts/mars/disk-config.nix`'s header comment if it turns out to be UEFI.

Otherwise the flow is identical to the ZimaBlade steps above:
```
nix run github:nix-community/nixos-anywhere -- \
  --flake .#mars \
  --extra-files /tmp/extra \
  --generate-hardware-config nixos-generate-config ./hosts/mars/hardware-configuration.nix \
  --target-host root@<mars-ip>
```
(stage the host key into `/tmp/extra/etc/ssh/` first, same as step 4 there).
Manual alternative (USB ISO): boot installer, `disko` the disk, then
`nixos-install --flake .#mars`.

## First install on terra — no-USB findiso reinstall (replacing CachyOS)

terra is a Ryzen 9 5900X / Radeon RX 6800 XT desktop, currently running
CachyOS with a writable root and **Limine** as its bootloader (not
systemd-boot — see step 4). Everything is already prepped
in this repo: real OS-disk id in `disk-config.nix`, real login pubkey in
`common.nix`, terra's age recipient in `.sops.yaml`, its host key
pre-generated at `~/.config/homelab/terra/`, and `secrets/terra.yaml` already
holds real `darman_password` / `tailscale_authkey` values. Nothing to fill
in — just run it.

> ⚠️  **`./scripts/deploy kexec-local` does NOT work on terra — do not use it.**
> Confirmed on real hardware: the jump hangs completely (kexec's own
> `device_shutdown()` pass runs — SCSI disks sync fine — then the machine goes
> dark and never comes back; `journalctl --list-boots` showed a **~15 minute**
> gap before the next boot, i.e. a hard hang needing a manual power cycle, not
> a slow jump). Near-certainly amdgpu: discrete AMD GPUs are known to hang
> during kexec's device-shutdown pass with no clean way to hand control back
> before the jump — same class of issue as jupiter's `reboot=pci` warm-reboot
> workaround, just fatal here instead of merely slow. The path below instead
> triggers a real ACPI reboot through firmware POST — a materially different
> code path that never runs kexec's device-shutdown pass at all.

> ⚠️  The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The
> dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs
> disks are not in disko and are untouched — but double check with
> `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have
> changed since `disk-config.nix` was written.

One command does the whole thing — no need to `sudo` it yourself, it
self-elevates:
```
./scripts/deploy install terra localhost
```
`scripts/deploy` detects it isn't already inside a live installer (checks
`uname -n`) and instead:

1. Asks where to stage the iso file — never auto-picks, because the wrong disk
   here is destroyed mid-install (`HOMELAB_INSTALLER_STAGE_DIR` skips the
   prompt for scripted use). It **refuses** if that path resolves to a disk
   `disk-config.nix` is about to wipe, if it can't work out which physical disk
   the path is on at all (fail-closed — LVM and RAID can span several), or if
   it's on btrfs (stage-1 mounts a btrfs volume's *top level*, so a path inside
   a subvolume never resolves and you boot to an emergency shell). On terra,
   `/mnt/hdd_01` is the right answer; the CachyOS root is btrfs on the OS disk
   and is rejected on both counts.
2. Prints exactly what it is about to do — OS disk, staging disk, boot entry —
   and waits for you to type `yes`. `--yes` skips it; that is what the ISO
   passes when it re-runs the command itself.
3. Builds `installer-iso`'s kernel + initrd + iso image, checks both target
   partitions have room, then copies the kernel/initrd **and terra's
   pre-generated ssh host key** to the boot partition (found via
   `bootctl --print-boot-path`, not assumed to be `/boot`) and the iso to the
   staging dir.
4. Arms a **one-shot** boot of it with `findiso=` + `homelab.install=terra` +
   `homelab.keypart=<PARTUUID>` on the kernel cmdline, and reboots — a real
   `systemctl reboot`, not kexec. Two mechanisms, picked automatically:
   - **systemd-boot** (jupiter, neptun, and terra once NixOS is on it): a
     `bootctl set-oneshot` loader entry.
   - **anything else** — terra today runs Limine, which reports `One-shot entry
     control: ✗` and has no equivalent: the firmware's own **`BootNext`**,
     pointing at a temporary UEFI entry that EFI-stub-boots the kernel straight
     off the ESP. Created with `--create-only` so it never enters `BootOrder`,
     which means it is reachable exactly once and nothing else changes.

   Either way the box falls back to its normal bootloader if the attempt
   fails — nothing is made permanent before the install succeeds. The
   temporary UEFI entry is deleted by the installer as soon as it boots.

The booted installer clones the repo (`homelab-checkout.service`, needs
network — it's public now, no credentials involved) and then
`homelab-auto-install.service` reads `homelab.install=terra` back off
`/proc/cmdline`, mounts `homelab.keypart=` to pick terra's host key back up
into `/root/.config/homelab/terra/`, and re-runs the exact same
`./scripts/deploy install terra localhost` itself — now genuinely inside the
installer, so it takes the disko + `nixos-install` branch instead of preparing
again. That key is what seeds `/etc/ssh` on the new system, which is what lets
`/run/secrets/*` decrypt on boot #1; it has to travel this way because the ISO
is built from a **public** repo and deliberately carries no credentials. The
copy on the boot partition dies with the disko wipe minutes later.

The whole thing is unattended after the initial reboot; ssh into
`homelab-installer` (same pubkey as the ISO everywhere else) to watch
progress — `journalctl -u homelab-checkout -u homelab-auto-install -f`.

> The checkout step resolves `git.mgaction.town`, which goes through mercury's
> pihole on the LAN. If mercury is down, the installer boots fine but never
> gets the repo — fix DNS and `systemctl restart homelab-checkout`.

When it's done, reboot again into the freshly installed NixOS. Then, same as
any other host:
```
ssh darman@terra sudo -v          # DO NOT SKIP — see below
```

`darman` is created with `mutableUsers = true`, so `/etc/shadow` is written
**once**. If the sops secret wasn't readable at that moment the account gets
`!` (locked) permanently and no `deploy switch` will fix it — verify sudo
works while you still have physical console access as a fallback.

## Deploy (the `./deploy` wrapper)

All arguments mandatory — no default host, no default config.

```
./deploy kexec   <config> <host>   # headless kexec into a RAM installer (RO-root box)
./deploy install <config> <host> [--yes]   # first install; wipes OS disk, ships host key
./deploy switch  <config> <host>   # rebuild + activate on a running host
./deploy boot|test <config> <host> # stage for next boot / activate without boot entry
./deploy image   <config>          # build an SD-card image (mercury)
./deploy flash   <config> <dev>    # build SD image, write it, drop the sops age key
```
`switch`/`boot`/`test` prompt for darman's password (`wheelNeedsPassword`).
`<config>` is a `nixosConfigurations` name (`jupiter`, `neptun`). Its pre-generated
SSH host key lives at `~/.config/homelab/<config>/ssh_host_ed25519_key`.

Examples:
```
./deploy switch jupiter jupiter.sol
./deploy install neptun 159.195.64.117
```
Rollback: `nixos-rebuild switch --rollback` on the host, or pick a prior
generation at boot.

## Post-deploy steps (per host)

Things the flake cannot do for you. Skipping these leaves a host that builds
and boots but doesn't work.

### Every host, immediately after a first install

```
ssh darman@<host> sudo -v          # DO NOT SKIP
```

`users.mutableUsers` is `true`, so `/etc/shadow` is written **once**, when the
user is created. If the sops secret wasn't readable at that moment the account
gets `!` (locked) permanently — `deploy switch` will never fix it, because the
activation script only sets a password for users not already in `/etc/shadow`.
Combined with `wheelNeedsPassword = true` and `PermitRootLogin = "no"` that
means no way to escalate, and recovery is physical: netcup's rescue system for
neptun, or pulling the SD card for mercury. Verify sudo while you still have
another way in.

### neptun (netcup VPS)

1. **Edge firewall.** In netcup's panel, inbound `ACCEPT` for TCP 22/80/443/2222
   **and a rule accepting inbound UDP**. The firewall is stateless: without the
   UDP rule every DNS and NTP *reply* is dropped, and nothing on the box reports
   an error — it looks like headscale crash-looping on its DERP fetch and Caddy
   failing ACME. `grep -A1 '^Udp:' /proc/net/snmp` showing `InDatagrams 0` is the
   tell. Rules apply on VM restart, not on save. This is safe: `nixos-fw` is
   stateful and default-deny, so it remains the real policy.
   Also open UDP 3478 (STUN) and 41641 (tailscale direct).
2. **Authentik** creates `akadmin` on first start; log in at
   `https://auth.mgaction.town` with `authentik_bootstrap_password` from sops.
   The username is hardcoded upstream and the bootstrap runs once — later
   changes to the env vars are ignored.
   To use your own admin instead: create a user, add it to the **`authentik
   Admins`** group (superuser is a *group* flag in Authentik, there is no
   per-user one), verify it works in a private window, then **deactivate**
   `akadmin` — do not rename or delete it. The bootstrap blueprint keys on
   `username: akadmin` with `state: created`, so if no user by that name
   exists it simply makes a new one on the next reconcile.
3. **Bootstrap the tailnet** (headscale starts with an empty database):
   ```
   sudo headscale users create darman
   sudo headscale preauthkeys create --user darman --reusable --expiration 24h
   ```
   Put that key in **every** host's sops file as `tailscale_authkey` and rebuild.
4. **Headplane API key** — defaults to 90d, after which headplane silently stops
   listing nodes:
   ```
   sudo headscale apikeys create --expiration 999d      # -> headplane_headscale_api_key
   ```
5. **Headplane OIDC.** In Authentik create an OAuth2/OpenID provider
   (confidential, redirect `https://vpn.mgaction.town/admin/oidc/callback`,
   **a signing key must be selected** or discovery exposes no JWKS) and an
   application with slug **`headplane`** — the slug is what makes the issuer
   `.../application/o/headplane/` in `services/vpn/headplane.nix`. Client ID goes
   in that file, client secret into sops.
6. **Headscale OIDC** (optional — pre-auth keys work without it). A *second*
   Authentik provider/application, slug **`headscale`**, redirect
   `https://vpn.mgaction.town/oidc/callback` (headscale's own, not headplane's
   under `/admin`). Client ID in `services/vpn/headscale.nix`, secret into sops
   as `headscale_oidc_client_secret`.
   ⚠️  headscale runs OIDC discovery **at startup and a failure is fatal** —
   an issuer pointing at an application that doesn't exist yet means the
   control server won't boot, taking the whole tailnet's control plane with
   it. Always verify first:
   ```
   curl -s https://auth.mgaction.town/application/o/headscale/.well-known/openid-configuration
   ```
   Users created by OIDC login are distinct from `headscale users create`
   ones: headplane matches the OIDC `sub` claim against the user's
   `providerId`, CLI-made users have none, and 0.28 dropped both
   `map_legacy_users` and node reassignment — so moving an existing node to
   an OIDC user means re-enrolling it.

### jupiter

- `chown -R gitea:gitea /mnt/data/AppData/gitea` after the first deploy (the
  repos were copied in over CIFS as `darman:users`).
- **Re-enrolling after the headscale database was recreated:** `tailscaled`
  keeps its old node key and reports `Running`, and the autoconnect unit exits
  early on that state without ever sending the new pre-auth key. Force it:
  ```
  sudo tailscale logout && sudo systemctl restart tailscaled-autoconnect
  ```

### mars

- **Confirm the OIDC redirect still resolves.** hermes-agent.nix reuses
  jupiter's old Authentik application (slug `hermes`, redirect
  `https://hermes.mgaction.town/auth/callback`) unchanged — nothing to
  reconfigure in Authentik, just verify `neptun`'s `hermes.mgaction.town`
  vhost (now pointed at `mars.orbit.sol:9119`) actually reaches the
  dashboard once mars is up and joined the tailnet.
- **Carrying forward old chat history/memories:** mars starts with a fresh
  Hermes state dir (`/var/lib/hermes/.hermes`). jupiter's old instance data
  is backed up at `/mnt/data/AppData/hermes.bak-2026-08-21` — rsync it over
  (via the `/mnt/jupiter` samba mount) before the first switch if you want
  it preserved instead of starting clean.

### mercury (Raspberry Pi 3B+)

- `./deploy flash mercury /dev/sdX` writes the dedicated age key to the root
  partition. Without `~/.config/homelab/mercury/age.txt` it silently skips that
  step and **no secret decrypts on the box** — check `ls /run/secrets` after
  first boot.
- It boots from an SD card, so config changes are `./deploy switch mercury <ip>`
  (an aarch64 build — needs `extra-platforms` + binfmt on the laptop, see the
  gotchas in `CLAUDE.md`) rather than a reflash.
- **Suspect the card first** when binaries crash with `Illegal instruction` or
  services fail inexplicably. Failing flash returns corrupt data with no I/O
  errors in `dmesg`:
  ```
  sudo nix-store --verify --check-contents      # add --repair to fix
  ```
  A card that has corrupted one path will corrupt more. Replace it and reflash.
- **A reflash wipes `/var/lib/pihole`**, taking the gravity database with it.
  The blocklists themselves are declared in `services/network/pihole.nix`, and
  the `pihole-adlists` unit re-seeds them on boot and rebuilds gravity when it
  finds it empty — so this heals itself, but the first boot after a reflash
  spends several minutes downloading lists. Query history and dynamic DHCP
  leases are genuinely lost (static leases are declarative). Check with:
  ```
  systemctl status pihole-adlists
  sudo podman exec pihole pihole-FTL sqlite3 /etc/pihole/gravity.db \
    "SELECT address,enabled FROM adlist; SELECT COUNT(*) FROM gravity;"
  ```
  `Blocked DNS queries: 0` in the pihole logs means gravity is empty — DNS
  resolves fine, nothing is filtered.
- mercury's own `resolv.conf` is deliberately public resolvers, not its own
  pihole (`resolveLocalQueries = false`, see `CLAUDE.md`) — so `.sol` names do
  not resolve *on mercury itself*. That is expected, not a fault.
- **mercury is load-bearing for the whole tailnet's DNS.** headscale sets
  `override_local_dns = true` with pihole as the only global nameserver, so
  every node — including a phone on mobile data — resolves through it and gets
  ad blocking and `.sol` names anywhere. The flip side is that mercury (or the
  home connection) going down costs name resolution on every device, not just
  `.sol`. Recovery on a stranded device is turning Tailscale off.
  There is deliberately no public fallback in `nameservers.global`: tailscale
  treats that list as a set, so a second entry would let queries slip past the
  filter whenever mercury is slow.
  neptun and mercury opt out with `--accept-dns=false` — mercury because it
  would otherwise resolve through itself, neptun because a public reverse
  proxy must not depend on a Pi at home to renew its certificates.
- Tailnet names are `*.orbit.sol`, LAN names are `*.sol`. Both work everywhere
  on the tailnet because tailscale matches DNS routes by **longest suffix**, so
  `orbit.sol` reaches MagicDNS even though everything else goes to pihole.
  Never name a LAN host `orbit`: pihole's `address=/<host>.sol/<ip>` lines match
  a name *and everything beneath it*, which would swallow the entire tailnet
  zone.

## Adding a service

Copy the `whoami` block in `oci-containers.containers`, swap image/ports/volumes.
Native NixOS module exists for many apps (Nextcloud, Jellyfin, Grafana...) —
prefer `services.<app>` over a container when available. Add a `caddy`
`virtualHosts` block to expose it.

## Notes

- Backend is Podman with `dockerCompat` — `docker` CLI works, no daemon.
- Samba keeps its own password DB. `services.samba` never sets it; a systemd
  oneshot (`samba-smbpasswd`) provisions it. Host reads the password from
  `/run/secrets/samba_password` (**sops-nix**); the VM falls back to plaintext
  `/etc/samba/smb-password`.
- Secrets: `secrets/jupiter.yaml` is age-encrypted (safe to commit) to two
  recipients in `.sops.yaml` — the **admin** key (edit on laptop,
  `~/.config/sops/age/keys.txt`) and the **jupiter host** key (derived from its
  SSH host key via `ssh-to-age`, decrypts at runtime). Private keys live
  off-repo and are gitignored. Rotate/add recipients with `sops updatekeys`.
- Data disk: plain `fileSystems."/mnt/data"` in configuration.nix — kept out of
  disko so it is never formatted. Reference by `by-id` / `by-uuid`.
- `system.stateVersion` = `26.05`, install-time schema. Do NOT bump on upgrades.
- Terraform is not used: a single bare-metal box has no provider API. disko +
  nixos-anywhere cover provisioning natively.