README: document the per-host post-deploy steps
Everything here is something the flake cannot do for you, and all of it was learned by hitting it: a host that builds and boots cleanly is not necessarily a host that works. The sudo check applies to every host and is the one that cost the most. mutableUsers is true, so /etc/shadow is written once at user creation -- if the sops secret wasn't readable at that moment the account is locked forever and no rebuild will fix it. That happened twice, and recovery was netcup's rescue system for neptun and pulling the SD card for mercury. neptun's netcup firewall is stateless and denies inbound UDP by default, which drops every DNS and NTP reply while reporting nothing anywhere. Also covers the Authentik/headscale/headplane bootstrap, which is a chain of manual steps producing values the config needs. jupiter gets the tailscaled stale-state trap: after the headscale database is recreated the daemon still reports Running, and the autoconnect unit exits early without sending the new pre-auth key. mercury gets the SD-card failure mode, since silent flash corruption surfaces as SIGILL from random binaries with a clean dmesg. Also drops a stray code fence that had been dangling at EOF. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
@@ -26,7 +26,7 @@ hosts/
|
||||
vm.nix # VirtualBox test image (jupiter-vbox)
|
||||
neptun/ # netcup public reverse proxy + tailnet node
|
||||
configuration.nix disk-config.nix hardware-configuration.nix secrets.nix
|
||||
secrets/ # age-encrypted sops files (jupiter.yaml, neptun.yaml)
|
||||
secrets/ # age-encrypted sops files, one per host
|
||||
scripts/ # deploy, edit_secrets
|
||||
```
|
||||
|
||||
@@ -93,7 +93,10 @@ All arguments mandatory — no default host, no default config.
|
||||
./deploy install <config> <host> # first install; wipes OS disk, ships host key
|
||||
./deploy switch <config> <host> # rebuild + activate on a running host
|
||||
./deploy boot|test <config> <host> # stage for next boot / activate without boot entry
|
||||
./deploy image <config> # build an SD-card image (mercury)
|
||||
./deploy flash <config> <dev> # build SD image, write it, drop the sops age key
|
||||
```
|
||||
`switch`/`boot`/`test` prompt for darman's password (`wheelNeedsPassword`).
|
||||
`<config>` is a `nixosConfigurations` name (`jupiter`, `neptun`). Its pre-generated
|
||||
SSH host key lives at `~/.config/homelab/<config>/ssh_host_ed25519_key`.
|
||||
|
||||
@@ -105,6 +108,92 @@ Examples:
|
||||
Rollback: `nixos-rebuild switch --rollback` on the host, or pick a prior
|
||||
generation at boot.
|
||||
|
||||
## Post-deploy steps (per host)
|
||||
|
||||
Things the flake cannot do for you. Skipping these leaves a host that builds
|
||||
and boots but doesn't work.
|
||||
|
||||
### Every host, immediately after a first install
|
||||
|
||||
```
|
||||
ssh darman@<host> sudo -v # DO NOT SKIP
|
||||
```
|
||||
|
||||
`users.mutableUsers` is `true`, so `/etc/shadow` is written **once**, when the
|
||||
user is created. If the sops secret wasn't readable at that moment the account
|
||||
gets `!` (locked) permanently — `deploy switch` will never fix it, because the
|
||||
activation script only sets a password for users not already in `/etc/shadow`.
|
||||
Combined with `wheelNeedsPassword = true` and `PermitRootLogin = "no"` that
|
||||
means no way to escalate, and recovery is physical: netcup's rescue system for
|
||||
neptun, or pulling the SD card for mercury. Verify sudo while you still have
|
||||
another way in.
|
||||
|
||||
### neptun (netcup VPS)
|
||||
|
||||
1. **Edge firewall.** In netcup's panel, inbound `ACCEPT` for TCP 22/80/443/2222
|
||||
**and a rule accepting inbound UDP**. The firewall is stateless: without the
|
||||
UDP rule every DNS and NTP *reply* is dropped, and nothing on the box reports
|
||||
an error — it looks like headscale crash-looping on its DERP fetch and Caddy
|
||||
failing ACME. `grep -A1 '^Udp:' /proc/net/snmp` showing `InDatagrams 0` is the
|
||||
tell. Rules apply on VM restart, not on save. This is safe: `nixos-fw` is
|
||||
stateful and default-deny, so it remains the real policy.
|
||||
Also open UDP 3478 (STUN) and 41641 (tailscale direct).
|
||||
2. **Authentik** creates `akadmin` on first start; log in at
|
||||
`https://auth.mgaction.town` with `authentik_bootstrap_password` from sops.
|
||||
The username is hardcoded upstream and the bootstrap runs once — later
|
||||
changes to the env vars are ignored.
|
||||
3. **Bootstrap the tailnet** (headscale starts with an empty database):
|
||||
```
|
||||
sudo headscale users create darman
|
||||
sudo headscale preauthkeys create --user darman --reusable --expiration 24h
|
||||
```
|
||||
Put that key in **every** host's sops file as `tailscale_authkey` and rebuild.
|
||||
4. **Headplane API key** — defaults to 90d, after which headplane silently stops
|
||||
listing nodes:
|
||||
```
|
||||
sudo headscale apikeys create --expiration 999d # -> headplane_headscale_api_key
|
||||
```
|
||||
5. **Headplane OIDC.** In Authentik create an OAuth2/OpenID provider
|
||||
(confidential, redirect `https://vpn.mgaction.town/admin/oidc/callback`,
|
||||
**a signing key must be selected** or discovery exposes no JWKS) and an
|
||||
application with slug **`headplane`** — the slug is what makes the issuer
|
||||
`.../application/o/headplane/` in `services/vpn/headplane.nix`. Client ID goes
|
||||
in that file, client secret into sops.
|
||||
6. **Pin jupiter's tailnet address.** `networking.hosts` in
|
||||
`hosts/neptun/configuration.nix` carries jupiter's IP, because neptun runs
|
||||
`--accept-dns=false` and cannot use MagicDNS. Until it matches
|
||||
`headscale nodes list`, the `abs.` and `git.` vhosts fail to proxy.
|
||||
|
||||
### jupiter
|
||||
|
||||
- `chown -R gitea:gitea /mnt/data/AppData/gitea` after the first deploy (the
|
||||
repos were copied in over CIFS as `darman:users`).
|
||||
- **Re-enrolling after the headscale database was recreated:** `tailscaled`
|
||||
keeps its old node key and reports `Running`, and the autoconnect unit exits
|
||||
early on that state without ever sending the new pre-auth key. Force it:
|
||||
```
|
||||
sudo tailscale logout && sudo systemctl restart tailscaled-autoconnect
|
||||
```
|
||||
|
||||
### mercury (Raspberry Pi 3B+)
|
||||
|
||||
- `./deploy flash mercury /dev/sdX` writes the dedicated age key to the root
|
||||
partition. Without `~/.config/homelab/mercury/age.txt` it silently skips that
|
||||
step and **no secret decrypts on the box** — check `ls /run/secrets` after
|
||||
first boot.
|
||||
- It boots from an SD card, so config changes are `./deploy switch mercury <ip>`
|
||||
(an aarch64 build — needs `extra-platforms` + binfmt on the laptop, see the
|
||||
gotchas in `CLAUDE.md`) rather than a reflash.
|
||||
- **Suspect the card first** when binaries crash with `Illegal instruction` or
|
||||
services fail inexplicably. Failing flash returns corrupt data with no I/O
|
||||
errors in `dmesg`:
|
||||
```
|
||||
sudo nix-store --verify --check-contents # add --repair to fix
|
||||
```
|
||||
A card that has corrupted one path will corrupt more. Replace it and reflash;
|
||||
only pihole's runtime state (query history, dynamic leases) is lost — the
|
||||
static leases are declarative.
|
||||
|
||||
## Adding a service
|
||||
|
||||
Copy the `whoami` block in `oci-containers.containers`, swap image/ports/volumes.
|
||||
@@ -129,4 +218,3 @@ prefer `services.<app>` over a container when available. Add a `caddy`
|
||||
- `system.stateVersion` = `26.05`, install-time schema. Do NOT bump on upgrades.
|
||||
- Terraform is not used: a single bare-metal box has no provider API. disko +
|
||||
nixos-anywhere cover provisioning natively.
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user