Gives the Hermes agent a real directory of markdown for the luna_wiki
vault, at /var/lib/livesync-bridge/vault and mounted into her container at
/opt/data/vault (inside HERMES_WRITE_SAFE_ROOT, so she can write, not only
read). Obsidian itself is an Electron GUI with no headless mode, and an
agent wants files rather than an app.
livesync-bridge is Deno, not packaged, and publishes no image — upstream
ships only a `build: .` compose file. So it comes in as a pinned non-flake
input and runs under systemd. Two things that are not obvious:
- The source is COPIED to a fixed path rather than run from /nix/store.
Deno keys localStorage — where the bridge records per-file sync state —
by the main module's origin. Verified by running one source tree from
two paths against a single DENO_DIR: two origin directories appear. Run
from the store, every input bump would silently reset both peers to a
full rescan.
- It runs as uid 986/gid 983, the same identity the hermes container
uses. Two uids in a shared group only works while every file stays
group-writable, and one 0644 file dropped by the agent would stall sync
on that path.
Talks to CouchDB over the tailnet (jupiter.orbit.sol:5984), so neptun's
vhost, its TLS and its path allowlist are all out of the picture.
Verified before deploying: `deno check` passes on nixpkgs' 2.8.3 (upstream
pins 2.6.9), and the bridge starts, reads LSB_CONFIG, detects a file and
writes its health heartbeat. Both directions confirmed working on mars
afterwards.
Credentials are currently the `obsidian` admin account and the personal
vault's passphrase, which means mars can decrypt every vault database and
not just luna's. Deliberate reuse of what existed; hosts/mars/secrets.nix
records the two independent ways to narrow it.
⚠️ Upstream has three open, unanswered issues on the storage->couchdb
direction (#50, #23, #46) and all fail silently — the log reports the
upload and the database is never updated. Do not treat this directory as
durable storage for anything luna cannot regenerate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLN5nkLBtCciD3ZnUwtw2b
The hook registered with no events at all and delivered nothing.
"pull_request_review_comment" and "pull_request_review_rejected" are real
HookEventTypes and real X-GitHub-Event-Type values, but they are not
things gitea's hook API accepts. updateHookEvents
(routers/api/v1/utils/hook.go) matches a fixed list of api names and
silently ignores anything else, so every event flag stayed false, the POST
succeeded, and the hook sat there inert.
There is no narrower api name: HasEvent (models/webhook/webhook.go)
collapses approved, rejected and review-comment onto
HookEventPullRequestReview, so `pull_request_review` is a single switch for
all three. Approvals consequently cannot be excluded at the hook any more.
They now cross the wire as "pull_request_approved", which is not in the
route's event list, so Hermes ignores them on the event match -- before the
filter script and before any LLM call. Gitea's delivery log will show them
answered 200/ignored, which is intended.
That makes three namespaces for the same event rather than two, so the
tables in both nix files and the README now carry the api column, and the
README warns about the silent-ignore behaviour that hid this.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Fills in the subscription/wire name table for all five mappings rather
than the two prose examples, and records why the routes are written as
config instead of subscribed -- including that the toolset grant is
deliberate but not enforced, since the file it lives in is inside
HERMES_WRITE_SAFE_ROOT.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
A timeline comment on a PR never reached the route. Gitea reuses the same
strings in two namespaces and they collide:
subscription name wire name (X-GitHub-Event) what it is
pull_request_comment issue_comment comment on a PR
issue_comment issue_comment comment on an issue
pull_request_review_comment pull_request_comment review on a PR
The hook's `events` array takes the subscription name; Hermes matches
--events against X-GitHub-Event, the wire name, produced by
HookEventType.Event() in modules/webhook/type.go. So --events
pull_request_comment was selecting review submissions and could never match a
comment -- the exact inversion of what it reads like.
That also explains both observed failures. The review submission matched
(wire name pull_request_comment) and reached the filter, which correctly
dropped it on action=reviewed since a PullRequestPayload carries no comment
object. The timeline comment arrived as issue_comment, matched nothing, and
was dropped by the events filter before the script ever ran.
gitea.nix and hermes-agent.nix now deliberately name the same event
differently, so both carry the table and say the other is not a typo.
issue_comment on the wire also covers comments on plain issues. The hook does
not subscribe those, and the filter's is_pull check drops them regardless, so
widening the hook later cannot leak issue comments into the agent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
The relay existed on the premise that Gitea sends no header Hermes can read
an event name from, so something had to copy X-Gitea-Event into
X-GitHub-Event. That premise was wrong. Gitea's addDefaultHeaders sets
req.Header["X-GitHub-Delivery"] = []string{t.UUID}
req.Header["X-GitHub-Event"] = []string{event}
req.Header["X-GitHub-Event-Type"] = []string{eventType}
unconditionally, for every webhook type, alongside X-Hub-Signature-256 in
GitHub's exact format. (Direct map assignment rather than .Add() specifically
to keep the "GitHub" casing that canonicalisation would destroy.) Hermes
validates that signature on any route without provider gating and reads the
event name from that header, so gitea and hermes already speak the same
protocol and the translation layer was translating nothing.
Gitea now posts straight at http://mars.orbit.sol:8644/webhooks/gitea-pr-comments.
The URL path is the Hermes route name, so a second subscription is a second
hook and nothing else -- the route-in-path indirection the relay grew was a
reimplementation of something Hermes already had.
Removes the module, the 200-line relay, its test, the mars import, the 8645
listener, and the stale gitea-hermes-webhook-relay.service entry left in the
secret's restartUnits. hermes-agent-webhook-route moves to
hosts/mars/hermes-agent.nix, next to the container and the read-only prompt
and filter mounts it depends on.
Also makes that unit refuse to subscribe when GITEA_HERMES_WEBHOOK_SECRET is
unset in the container, matching the existing empty-prompt check. An empty
secret silently fails every delivery signature check afterwards while the
unit still reports success -- the worst possible failure shape, and one this
setup can actually produce on a first deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
jq was only ever on the `path` of the units that call it, so it was absent
from an interactive shell — which made the hook-migration commands in the
README unrunnable on the host they target. It is a general-purpose tool and
every host already carries curl, so it belongs alongside it rather than being
pulled in per-unit.
Also simplifies those README commands now that jq is present, and uses mars's
existing `hermes` alias instead of spelling out the podman exec.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Retiring gitea-events is a one-off migration, not something worth re-running
on every boot. Both units now only touch what they own: jupiter's creates or
updates its own hook and deletes nothing, and mars's removes only the route
it is about to re-subscribe, as the idempotency step for `subscribe`.
Keeping the deletes would have meant a redeploy could silently remove a hook
or route someone added deliberately -- a real risk now that sibling hooks
for other Hermes routes are the intended pattern.
README carries the manual commands, and the note that both hooks fire until
the old one is removed by hand, so events arrive twice in the meantime.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Renames the subscription to gitea-pr-comments (it handles one event; the old
gitea-events name promised more than it delivered) and drops --deliver.
Rather than move the hardcoded route from one constant to another, the relay
now reads it from the request path: POST /gitea/<route> forwards to
<base>/webhooks/<route>. The route name was the last thing tying this service
to a specific subscription, so a second Hermes route is now a `hermes webhook
subscribe <name>` plus a Gitea hook at /gitea/<name>, with no relay change --
previously it would also have needed a second relay URL baked in here.
The path segment is interpolated into an outbound URL, so it is validated
against ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$ and refused rather than sanitised
when it does not match. The path is matched raw and never URL-decoded, so
percent-encoded separators fail the charset check instead of surviving it;
requiring an alphanumeric first character also rejects "." and "..". Without
this, POST /gitea/..%2fadmin would let anything that can reach the relay
steer it at other Hermes endpoints. Tests cover traversal, encoded traversal,
embedded slashes, leading dot/dash, and the length bound, and assert nothing
reaches the stub Hermes in any of those cases.
Dropping --deliver leaves it at its default of `log`. The prompt tells her to
answer in the pull request, so the PR comment is the delivery and a Telegram
copy would only duplicate it; this also removes the hardcoded chat id that
was a third copy of TELEGRAM_HOME_CHANNEL.
Provisioning retires the pre-rename hook by its EXACT old URL rather than by
"points at the relay". Now that sibling hooks for other routes are the
intended pattern, a prefix match would delete them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
The relay was forwarding X-Gitea-Event and re-signing the body into the
deprecated generic-V1 X-Webhook-Signature header. Neither is something
Hermes acts on, which left the PR's core premise — "Hermes owns event
selection" — impossible to reach:
- Hermes reads the event name only from X-GitHub-Event/X-GitLab-Event,
then payload event_type/type, then falls back to the literal string
"unknown" (gateway/platforms/webhook.py). Gitea sends X-Gitea-Event and
no such payload key, so every delivery arrived as "unknown" and
`hermes webhook subscribe --events ...` could never select anything.
- Gitea's addDefaultHeaders() already signs every webhook type with
X-Hub-Signature-256 in GitHub's exact format, and Hermes accepts that
header on any route with no per-route provider gating. Re-signing into
V1 was both redundant and on a deprecated path.
So the relay now verifies the signature (accepting either X-Hub-Signature-256
or X-Gitea-Signature), forwards body and signature byte-for-byte, and copies
the one header Hermes actually needs. Authentication alone never justified
this service; that header copy does, and the module comment now says so.
Also fixed:
- gitea-hermes-webhook-provision had no API readiness wait, unlike both
sibling units in the same file. After=gitea.service does not mean gitea
is serving HTTP, so under `set -e` a Type=oneshot with no Restart= would
fail on first boot and stay failed, leaving the webhook unregistered.
- podman-hermes-agent added to the secret's restartUnits. The secret
reaches the container only via sops.templates, whose rendered path never
changes, so systemd would not restart the container when the secret was
first added — hermes-agent-webhook-route then read an empty value back
out of it and subscribed with an empty secret.
- Webhook provisioning passes the request body to curl on stdin rather
than in argv, keeping the shared secret out of /proc/<pid>/cmdline.
- Missing Content-Length now returns 411 rather than 413; dropped the
unreachable non-2xx branch (urlopen raises on non-2xx); env-var secret
fallback is stripped to match the credential-file path.
Adds gitea-hermes-webhook-relay-test.py, which drives the real relay over
real HTTP against a stub Hermes and covers the header copy as a regression
test. Both nixosConfigurations still evaluate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
New on-site host mars runs Hermes Agent as its sole service: joins the
tailnet, mounts jupiter's samba share at /mnt/jupiter (doubling as
Hermes's shared dropbox), and hosts state locally under /var/lib/hermes.
Same Authentik OIDC app/Telegram bot as before, just relocated — neptun's
hermes.mgaction.town vhost now points at mars.orbit.sol instead of jupiter.
hosts/jupiter/hermes-agent.nix and its three sops secrets are removed;
jupiter's Caddy vhost for it is gone too. Also refreshes tailscale_authkey
across all hosts and fixes two stale "erik@laptop" keys in flake.nix's
kexec/installer-iso images (leftover from a previous laptop, already
swapped out of common.nix back in 2fd5752) to darman@terra.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011FHr5ug9pu8q4XPrRkFnzJ
`install <config> localhost` assumed systemd-boot. terra's CachyOS boots
Limine, so it stopped at "/boot/loader/entries doesn't exist" — the check
added in e538788 doing its job, but with nowhere to go.
Limine cannot help here at all: `bootctl status` lists it as
`✗ One-shot entry control`, and CachyOS's pacman hooks regenerate
limine.conf regardless. So drop below the bootloader entirely and use the
firmware's own BootNext, pointing at a temporary UEFI entry that
EFI-stub-boots the installer kernel straight off the ESP. That keeps the
property which makes this safe to attempt: BootNext is spent by that one
boot, so a failed try still comes back on the normal bootloader.
- picked at runtime: systemd-boot loader entry when $BOOT/loader/entries
exists, else arm_efi_bootnext(). jupiter/neptun and terra-after-install
keep the systemd-boot path.
- `efibootmgr --create-only`, NOT `--create`: the latter pushes the entry
to the front of BootOrder, which would make a wiped installer the
permanent default if the install died halfway.
- the EFI stub loads initrd= from the volume it was loaded from, so this
mode stages on --print-esp-path rather than --print-boot-path.
- stale entries from an earlier attempt are removed before adding one, and
homelab-auto-install.service deletes the entry as soon as it boots, so
nothing lingers in NVRAM pointing at a reformatted partition.
- label matching is EXACT ("Homelab Installer"); a prefix match would have
deleted this box's Windows or Limine entry.
Verified against terra's real NVRAM (read-only): the label parser picks out
Limine/UEFI OS/Windows by exact name and rejects prefixes, and both branches
run end-to-end under stubs — BootNext mode emits the right --disk/--part,
loader path and initrd= cmdline, systemd-boot mode still writes its entry and
never calls efibootmgr.
README/CLAUDE.md corrected: terra runs Limine, not systemd-boot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The `install <config> localhost` auto-path added in 0ea9020 could not have
completed. Fixed three blockers plus the guard that was silently not guarding.
Inside installer-iso the run died before doing anything:
- systemd sets no $HOME for a service without User= (SetLoginEnvironment=
defaults to false), and this script runs under `set -u`, so it aborted on
the bare $HOME with "unbound variable". Added $KEYDIR + Environment=HOME.
- the host key it needs to seed /etc/ssh isn't on the ISO at all — that is
built from a public repo and carries no credentials on purpose. It now
travels on the boot partition, located via homelab.keypart=<PARTUUID> on
the kernel cmdline, and dies with the disko wipe minutes later. Without
it sops can't decrypt on boot #1 and mutableUsers locks darman for good.
- installation-cd-minimal leaves experimental-features unset, so both
`nix run` and `nixos-install --flake` failed. (The nixos-images kexec
installer sets them itself, which is why the same branch worked after
kexec-local but not from the ISO.)
The staging-dir guard passed everything on btrfs: findmnt prints the
subvolume as /dev/sdb2[/@], lsblk can't open that, and an empty parent was
treated as "different disk" — so it allowed staging the iso on the very disk
disko then wiped. terra's current CachyOS root is exactly that layout. Now
uses --nofsroot, resolves EVERY whole-disk ancestor (LVM/RAID span several:
/mnt/ssd_01 -> sdd+sde), and treats "can't tell" as a hard error. btrfs
staging is refused outright — stage-1 mounts a btrfs volume's top level, so
an iso inside a subvolume is unreachable.
findiso= lost its leading slash whenever the staging mountpoint was /,
giving /findisovar/tmp/x.iso and an emergency shell after the reboot.
Also:
- confirm before rebooting, like flash/kexec-local already do; --yes skips
it and is what the ISO passes itself
- $BOOT from `bootctl --print-boot-path`, not a hardcoded /boot
- free-space checks on both target partitions before the ~1GB copy
- `nix run .#disko` / `.#nixos-anywhere` from locked inputs instead of
github:... master-of-the-day, resolved while a disk is being wiped
- one_match warns instead of silently taking [0]; require_tracked covers
every hosts/<config>/*.nix; flash traps its mount
- drop nixos-images' `inputs.nixpkgs.follows` — that input doesn't exist,
it only printed a warning on every nix command
Verified: the prepare path exercised under stubs against this box's real
disks (btrfs-on-OS-disk, tmpfs, LVM, subdirectory), shellcheck clean, all
six configs evaluate, checks.kexec-local still passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
./scripts/deploy install <config> localhost now branches on is_live_installer()
(checks uname -n): outside a live installer it builds installer-iso, stages
its kernel/initrd on the ESP and the iso file on a disk the caller picks
(never auto-picked — the wrong disk here is destroyed mid-install), writes a
systemd-boot one-shot findiso= entry with homelab.install=<config> on the
kernel cmdline, and does a real systemctl reboot (not kexec — terra's
kexec-local hang is specifically in kexec's device-shutdown pass, a real ACPI
reboot never runs that code at all).
installer-iso gains homelab-auto-install.service: once homelab-checkout.service
clones the repo, it reads homelab.install= back off /proc/cmdline and re-runs
the identical deploy command itself, now genuinely inside the installer, so
it takes the disko+nixos-install branch instead of preparing again. The whole
reinstall is one command and unattended after the first reboot.
Also: every root-requiring path (kexec-local, the new prepare-and-reboot
branch, the disko+nixos-install branch) self-elevates via a require_root()
helper that re-execs the original invocation under sudo -E, instead of dying
and asking the caller to prefix sudo themselves. Uses an absolute script path
captured before the script's own cd, so the re-exec is correct regardless of
how it was invoked.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
require_tracked() in scripts/deploy now skips its git-tracked-file check
when there's no .git at all (nothing can be untracked in that case) — needed
for an earlier baked-in-`self` approach and kept as a generic fallback.
Since the repo is public now, installer-iso instead clones current master
via a homelab-checkout.service (after network-online.target) on every boot,
to /root/homelab. One ISO build stays useful indefinitely instead of going
stale, and there's still no rsync-the-repo-over step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed on real hardware: kexec's device_shutdown() pass runs (SCSI disks
sync fine in the log) then the machine goes dark for good — journalctl
--list-boots showed a ~15min gap before the next boot, a genuine hang needing
a manual power cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT):
discrete AMD GPUs are known to hang during kexec's device-shutdown pass with
no clean handoff before the jump, same class of issue as jupiter's
reboot=pci warm-reboot workaround, just fatal here instead of slow.
README's terra install section now leads with the USB installer path instead
(build ISO, dd to USB, rsync the repo over, disko + nixos-install locally).
CLAUDE.md's gotchas list gets the same warning. installer-iso is renamed from
jupiter-installer to homelab-installer since it's genuinely host-agnostic,
and now ships git.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kexec-local could never actually jump. nixos-images' kexec-run.sh ends with
`nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &` and returns immediately, so
the EXIT trap's `rm -rf "$stage"` deleted the kexec binary out from under the
sleeping shell. The box stayed on the old kernel and it looked like a slow boot.
Clear the trap before jumping, verify /sys/kernel/kexec_loaded, then sleep past
the timer.
Preflight everything before the point of no return, since this jumps the machine
you are typing at: CONFIG_KEXEC, kernel lockdown, exec-capable staging dir, free
space, RAM vs image size, and that the tarball holds all five expected files.
Stage on /var/tmp rather than /tmp because kexec-run.sh appends to initrd in
place and execs from that directory. sync before jumping (kexec -e skips
unmount). Confirmation prompt naming the host, since run in the wrong terminal
this kexecs the laptop; --yes skips it.
Drop the ssh-keygen -R added to the remote kexec path: kexec-run.sh copies
/etc/ssh/ssh_host_* into the appended initrd and restore-remote-access.nix
installs them back, so the host key survives the jump.
Proton Pass items are now keyed by <config> instead of <host>, since the address
is incidental and the config name is stable. kexec therefore takes <config>
<host>. Resolve titles among --filter-state active items first: a trashed item
with the same title shadowed the active one and returned an empty password,
which is indistinguishable from "no entry" and silently fell back to prompting
(hit on darman@neptun).
Other fixes: replace `ls glob | head -1` (returns empty with exit 0 on no match)
with a helper that dies; guard against untracked hosts/<config> since flakes
ignore untracked files; feed the sudo password more than once under setsid;
handle empty arrays under set -u; tolerate empty FSTYPE in the SD-card root
partition lookup; preflight zstdcat/dd/lsblk before the destructive dd; list
image and flash in the usage strings.
Add checks.x86_64-linux.kexec-local, a VM test driving the real script. It is
the only way to exercise kexec-local, which cannot be rehearsed on hardware. It
asserts the box left the old kernel, returned as nixos-installer, lost its old
/run, and kept its ssh host key. HOMELAB_KEXEC_TARBALL lets it reuse a prebuilt
installer instead of building ~500MB inside the guest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three connected changes, all triggered by the same outage.
base_domain leaves mgaction.town. That zone has a wildcard A+AAAA pointing
at neptun, and DNS wildcards match multi-label names, so
jupiter.hosts.mgaction.town resolved publicly to NEPTUN and Caddy proxied
to itself -- a silent loop rather than a lookup failure. Nesting the
tailnet inside the LAN domain as orbit.sol keeps the theme and resolves
unambiguously, since tailscale matches routes by longest suffix.
override_local_dns = true with pihole as the only global nameserver, so
roaming devices get ad blocking and .sol names off-LAN. With it false,
globalResolvers land in the netmap's FallbackResolvers, which a phone
with carrier DNS never consults. No public fallback is listed on purpose:
tailscale treats the list as a set, so a second entry would let queries
slip past the filter whenever mercury is slow. The cost is that mercury
is now a single point of failure for tailnet DNS.
neptun and mercury opt out individually. mercury would otherwise resolve
through itself. neptun must not depend on a Pi behind a domestic line to
renew the certificates for the control server every other node needs --
and it is circular besides, since tailscaled has to resolve
vpn.mgaction.town to connect at all. Instead neptun runs a dnsmasq stub
forwarding just orbit.sol to MagicDNS on 100.100.100.100, which tailscaled
answers whenever it is running regardless of --accept-dns. That resolves
jupiter live, so the hardcoded /etc/hosts pin is gone.
Also sets dns.nameservers.split explicitly: nixpkgs renders its own
dns.split option one level too high, but headscale reads
dns.nameservers.split (hscontrol/types/config.go:722) and so does
headplane, whose DNS page dies on the missing key with "Cannot convert
undefined or null to object". The module's option is dead as written.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FTL could not write gravity.db, reporting "attempt to write a readonly
database". The database file was writable; the directory was not. sqlite
creates a sibling gravity.db-journal for every write transaction, so FTL
needs to CREATE files in /var/lib/pihole, and the tmpfiles rule left it
root-owned. The error names the database rather than the directory, which
sends you looking at the file and the filesystem, neither of which is at
fault.
Own the directory as 1000 instead -- the pihole user FTL drops to after
the entrypoint's root phase. Podman is rootful here with no userns
remapping, so the number is the same inside and out; on the host it
collides with darman, harmlessly.
The blocklists are now declared in this module and seeded by a oneshot,
because /var/lib/pihole is not declarative and a reflash took gravity
with it. INSERT OR IGNORE keyed on the URL is idempotent so it can run on
every boot, while the expensive rebuild only runs when gravity is empty.
Adding a list to the Nix attribute needs a manual `pihole -g` -- that is
deliberate, since the rebuild downloads every list and is slow on a Pi.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The jupiter /etc/hosts pin step is gone: headscale no longer overrides
clients' local DNS, so MagicDNS resolves that name properly and the
hardcoded tailnet address is no longer needed.
Replaces it with the headscale OIDC application, including the warning
that discovery failure at startup is fatal, and that OIDC users cannot be
reconciled with CLI-created ones -- 0.28 dropped both map_legacy_users
and node reassignment, so switching a node's owner means re-enrolling it.
Documents the Authentik admin swap: superuser is a group flag, and
akadmin must be deactivated rather than renamed or deleted, since the
bootstrap blueprint keys on the username and recreates it otherwise.
For mercury, records that a reflash wipes the gravity database along with
the adlists -- resolution keeps working with nothing blocked, which is
easy to miss -- and that .sol not resolving on mercury itself is by
design, while it resolving on tailnet members depends on
override_local_dns staying false.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Everything here is something the flake cannot do for you, and all of it
was learned by hitting it: a host that builds and boots cleanly is not
necessarily a host that works.
The sudo check applies to every host and is the one that cost the most.
mutableUsers is true, so /etc/shadow is written once at user creation --
if the sops secret wasn't readable at that moment the account is locked
forever and no rebuild will fix it. That happened twice, and recovery was
netcup's rescue system for neptun and pulling the SD card for mercury.
neptun's netcup firewall is stateless and denies inbound UDP by default,
which drops every DNS and NTP reply while reporting nothing anywhere.
Also covers the Authentik/headscale/headplane bootstrap, which is a
chain of manual steps producing values the config needs.
jupiter gets the tailscaled stale-state trap: after the headscale
database is recreated the daemon still reports Running, and the
autoconnect unit exits early without sending the new pre-auth key.
mercury gets the SD-card failure mode, since silent flash corruption
surfaces as SIGILL from random binaries with a clean dmesg.
Also drops a stray code fence that had been dangling at EOF.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nixpkgs only carries Zitadel 2.71, which predates the login-v2 split and
cannot take a v3/v4 database (its migrations are forward-only), so the
instance running on the old Debian VPS could never have moved onto it.
authentik-nix ships 2026.5.4 and tracks upstream closely.
The authentik-nix input deliberately does not follow our nixpkgs, per
upstream's warning that overriding it breaks their pinned python
dependency set. That costs a second nixpkgs in the lock, so add
nix-community's Cachix to common.nix -- without it the closure is ~400
local derivations (npm, rust, python). The laptop that runs
scripts/deploy needs the same two lines in /etc/nix/nix.custom.conf.
Authentik's own module creates the database and orders its units against
postgresql.target, and recent versions need no redis, so the wiring is
just the module plus a secret. Pin postgresql explicitly so that editing
system.stateVersion can never silently demand a pg_upgrade of the
identity store.
Secret ownership is not uniform and the difference matters: authentik
and caddy take a systemd EnvironmentFile, which PID 1 reads as root
before dropping privileges, so root:root 0400 is correct. Headplane
opens its secret paths itself while already running as the headscale
user, so those three need an explicit owner or they fail to start.
Also on neptun:
- Pass Caddy's ACME account email through the same EnvironmentFile
mechanism and reference it with the Caddyfile {$VAR} placeholder.
services.caddy.email would render the address into the world-readable
store.
- Stop accepting MagicDNS from our own control server. headscale pushes
override_local_dns, so joining the tailnet would point neptun's
resolv.conf at a MagicDNS served by the tailscaled neptun itself hosts
-- a tailscaled failure would then also take out DNS, ACME renewal and
finally the certs for the control server every other node needs in
order to recover.
- Give headplane a writable DNS extra-records file. Its view of
headscale's config stays read-only, which is the right outcome for a
declarative box; records are data rather than config.
- Require a password for sudo. Deploys become interactive, but darman's
key is otherwise the only thing between the public internet and root.
- Enable zram (8 GB, and disko leaves no room for a swap device), and let
tailscaled-autoconnect retry instead of failing permanently when the
control server isn't up yet on a first boot.
networking.hosts still carries a PLACEHOLDER address for jupiter --
replace it from `headscale nodes list` once jupiter first enrols.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- add disko input; jupiter partitions/formats OS disk declaratively
- hardware-configuration.nix carries kernel modules only (disko owns fileSystems)
- data disk stays a plain unformatted mount, out of disko
- vbox unchanged (virtualbox-image supplies its own disk)
- README: nixos-anywhere remote install + daily rebuild loop
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
darmanandClaude Opus 5
luna