diff --git a/README.md b/README.md index 55310f5..ff9b71c 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ hosts/ vm.nix # VirtualBox test image (jupiter-vbox) neptun/ # netcup public reverse proxy + tailnet node configuration.nix disk-config.nix hardware-configuration.nix secrets.nix -secrets/ # age-encrypted sops files (jupiter.yaml, neptun.yaml) +secrets/ # age-encrypted sops files, one per host scripts/ # deploy, edit_secrets ``` @@ -93,7 +93,10 @@ All arguments mandatory — no default host, no default config. ./deploy install # first install; wipes OS disk, ships host key ./deploy switch # rebuild + activate on a running host ./deploy boot|test # stage for next boot / activate without boot entry +./deploy image # build an SD-card image (mercury) +./deploy flash # build SD image, write it, drop the sops age key ``` +`switch`/`boot`/`test` prompt for darman's password (`wheelNeedsPassword`). `` is a `nixosConfigurations` name (`jupiter`, `neptun`). Its pre-generated SSH host key lives at `~/.config/homelab//ssh_host_ed25519_key`. @@ -105,6 +108,92 @@ Examples: Rollback: `nixos-rebuild switch --rollback` on the host, or pick a prior generation at boot. +## Post-deploy steps (per host) + +Things the flake cannot do for you. Skipping these leaves a host that builds +and boots but doesn't work. + +### Every host, immediately after a first install + +``` +ssh darman@ sudo -v # DO NOT SKIP +``` + +`users.mutableUsers` is `true`, so `/etc/shadow` is written **once**, when the +user is created. If the sops secret wasn't readable at that moment the account +gets `!` (locked) permanently — `deploy switch` will never fix it, because the +activation script only sets a password for users not already in `/etc/shadow`. +Combined with `wheelNeedsPassword = true` and `PermitRootLogin = "no"` that +means no way to escalate, and recovery is physical: netcup's rescue system for +neptun, or pulling the SD card for mercury. Verify sudo while you still have +another way in. + +### neptun (netcup VPS) + +1. **Edge firewall.** In netcup's panel, inbound `ACCEPT` for TCP 22/80/443/2222 + **and a rule accepting inbound UDP**. The firewall is stateless: without the + UDP rule every DNS and NTP *reply* is dropped, and nothing on the box reports + an error — it looks like headscale crash-looping on its DERP fetch and Caddy + failing ACME. `grep -A1 '^Udp:' /proc/net/snmp` showing `InDatagrams 0` is the + tell. Rules apply on VM restart, not on save. This is safe: `nixos-fw` is + stateful and default-deny, so it remains the real policy. + Also open UDP 3478 (STUN) and 41641 (tailscale direct). +2. **Authentik** creates `akadmin` on first start; log in at + `https://auth.mgaction.town` with `authentik_bootstrap_password` from sops. + The username is hardcoded upstream and the bootstrap runs once — later + changes to the env vars are ignored. +3. **Bootstrap the tailnet** (headscale starts with an empty database): + ``` + sudo headscale users create darman + sudo headscale preauthkeys create --user darman --reusable --expiration 24h + ``` + Put that key in **every** host's sops file as `tailscale_authkey` and rebuild. +4. **Headplane API key** — defaults to 90d, after which headplane silently stops + listing nodes: + ``` + sudo headscale apikeys create --expiration 999d # -> headplane_headscale_api_key + ``` +5. **Headplane OIDC.** In Authentik create an OAuth2/OpenID provider + (confidential, redirect `https://vpn.mgaction.town/admin/oidc/callback`, + **a signing key must be selected** or discovery exposes no JWKS) and an + application with slug **`headplane`** — the slug is what makes the issuer + `.../application/o/headplane/` in `services/vpn/headplane.nix`. Client ID goes + in that file, client secret into sops. +6. **Pin jupiter's tailnet address.** `networking.hosts` in + `hosts/neptun/configuration.nix` carries jupiter's IP, because neptun runs + `--accept-dns=false` and cannot use MagicDNS. Until it matches + `headscale nodes list`, the `abs.` and `git.` vhosts fail to proxy. + +### jupiter + +- `chown -R gitea:gitea /mnt/data/AppData/gitea` after the first deploy (the + repos were copied in over CIFS as `darman:users`). +- **Re-enrolling after the headscale database was recreated:** `tailscaled` + keeps its old node key and reports `Running`, and the autoconnect unit exits + early on that state without ever sending the new pre-auth key. Force it: + ``` + sudo tailscale logout && sudo systemctl restart tailscaled-autoconnect + ``` + +### mercury (Raspberry Pi 3B+) + +- `./deploy flash mercury /dev/sdX` writes the dedicated age key to the root + partition. Without `~/.config/homelab/mercury/age.txt` it silently skips that + step and **no secret decrypts on the box** — check `ls /run/secrets` after + first boot. +- It boots from an SD card, so config changes are `./deploy switch mercury ` + (an aarch64 build — needs `extra-platforms` + binfmt on the laptop, see the + gotchas in `CLAUDE.md`) rather than a reflash. +- **Suspect the card first** when binaries crash with `Illegal instruction` or + services fail inexplicably. Failing flash returns corrupt data with no I/O + errors in `dmesg`: + ``` + sudo nix-store --verify --check-contents # add --repair to fix + ``` + A card that has corrupted one path will corrupt more. Replace it and reflash; + only pihole's runtime state (query history, dynamic leases) is lost — the + static leases are declarative. + ## Adding a service Copy the `whoami` block in `oci-containers.containers`, swap image/ports/volumes. @@ -129,4 +218,3 @@ prefer `services.` over a container when available. Add a `caddy` - `system.stateVersion` = `26.05`, install-time schema. Do NOT bump on upgrades. - Terraform is not used: a single bare-metal box has no provider API. disko + nixos-anywhere cover provisioning natively. -```