The shell was running two unrelated palettes: an amber one (#FFD063 accent,
#EEEEEE text, #0F1012 panels) hardcoded as ~200 raw hex literals across the
launchers, sidebar, systray, vitals and notifications, and an orange one
(#e8722a) that only the dense bar had, tokenized as per-file properties.
This unifies on the ORANGE values under the AMBER naming scheme, and moves the
lot into widgets/theme/Theme.qml. `surface` takes the dense bar's void
(#0a0a0a) rather than the old panel background. Zero color and font literals
remain anywhere under widgets/ outside Theme.qml.
Collisions resolved, all near-duplicates that wanted to be one token:
- #0F1012 + #0A0A0C + #0a0a0a -> surface
- #EEEEEE + #dedede -> text
- #7A7B7D + #858585 -> muted
- #292C30 + #22262C -> raised
- #FFD063 + #e8722a -> accent
Two derived things rather than literals. accentSoft (the pale flash the
top/bottom bars show while a launcher is open) was a hand-picked #FFF3C0
against amber, which is simply wrong against orange; it is now
Qt.tint(accent, white 55%), a ratio checked against the original (amber tinted
55% gives #FFE9B8 vs the hand-picked #FFF3C0). And the dense bar had been
hand-encoding Qt.rgba(0.87,0.87,0.87,a) and Qt.rgba(0.91,0.45,0.16,a), which
are just text and accent at alpha -- now textAlpha(a)/accentAlpha(a), so they
track a palette change instead of silently drifting.
Fonts came along too. Digital-7 Mono is dropped for DepartureMono: it was
never packaged, relying on a manual ~/.dots/fonts/digital_7 install that does
not exist on terra, so `fc-match "Digital-7 Mono"` resolved to DejaVu Sans and
all 38 of those sites -- the launcher lists, sidebar clock, systray labels,
every vitals readout -- were silently rendering in a PROPORTIONAL fallback.
Numeric columns should visibly improve. readoutFont is an alias of displayFont
rather than a second literal so the two roles cannot drift apart.
quickshell/CLAUDE.md updated: it said "No shared theme/tokens file yet" and
told contributors to grep for the existing hex color, which would now
reintroduce exactly what this removes.
Verified: no file references Theme. without the import, none imports it
unused, and the whole shell -- launchers, sidebar, vitals, systray,
notifications, not just the harness -- hot-reloaded clean on terra.
tests/HeadlessSmoke.qml deliberately keeps its own copies; its value is having
no dependencies.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SgcWkm3t6BDQktYHQvb8Hx
The dense bar's panel chrome gets a squarer outline (top-right and bottom-left
chamfers only, square on the other two corners) and a second accent line in
the lower right to balance the existing upper-left one.
The chrome then moves out of DenseBarContent's inline `component
TelemetryPanel` into its own file. The call sites are unchanged apart from the
name -- children still come from the default property -- and the widgets they
pass in (RadarGauge, NetworkTrace, MetricBlock) stay declared in
DenseBarContent, so their scope is unaffected by moving only the definition.
Two things could not come along and had to be reproduced locally, because a
component in its own file has no access to the enclosing scope:
- the palette and the two font families, previously read off `root`. They
are properties with defaults matching DenseBarContent's, which is this
repo's per-component convention. Duplicated on purpose for now; a shared
theme singleton is the place to collapse it.
- MicroText, which is an *inline* component of DenseBarContent and so
invisible from another file. Expanded to the Text it desugars to.
Also qualified the bare offsetY/chamfer/accentLineThickness references as
panel.*; they resolved through the component scope before, but being explicit
avoids ComponentBehavior: Bound warnings in the new file.
Verified the move was verbatim by normalising the old inline block and the new
file body and diffing them -- the only differences are the relocated property
block, the panel.* qualification and the MicroText expansion. Loads clean both
headlessly and in the real layer-shell shell.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SgcWkm3t6BDQktYHQvb8Hx
hosts/terra/home.nix ships dotfiles/quickshell via xdg.configFile, which
copies the tree into the store: ~/.config/quickshell is a read-only symlink
into /nix/store and every QML tweak costs a nixos-rebuild. quickshell does
hot-reload on file save -- but only for the files it watches, which are those
frozen store copies. `nix develop` now swaps the running shell to the working
tree (`qs -p`) and swaps it back on exit, so QML edits need no rebuild at all.
The swap starts the dev instance FIRST and kills the packaged one only once
dev is confirmed up. A QML error in the working tree then leaves you on your
normal bar instead of no bar, which matters because a broken save is exactly
when you would be running this. Liveness is "did `qs list -j` return json" --
it exits 0 whether or not it found anything, so the exit code says nothing.
Every kill is scoped to one config (`qs kill` = default, `qs kill -p` = that
path). A blanket kill would also take out unrelated instances; pkgs/rishot.nix
is one.
Three guards on the auto-swap, all learned by testing it:
- interactive only. `nix develop --command X` EXECs X, replacing the shell
that set the `trap ... EXIT`, so the restore never runs and you are left
on the dev instance. Non-interactive use gets `nix develop -c qs-dev`.
- WAYLAND_DISPLAY, so entering the shell over ssh cannot kill the desktop's
bar and leave nothing in its place.
- a sentinel, so a nested `nix develop` does not swap and restore twice.
Deliberately not wired to direnv (no .envrc): programs.direnv is enabled for
this user, so a `use flake` would swap the running desktop shell on every `cd`
into the checkout.
Verified end to end on terra: swap, hot-reload of a working-tree edit, and
restore, plus both the interactive and non-interactive paths.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SgcWkm3t6BDQktYHQvb8Hx
The bind onto /var/lib/private/victoriametrics needs its source to exist or
the mount fails -- and because it is `nofail`, quietly: RequiresMountsFor is
satisfied by /mnt/data itself, so the service would start anyway and write
the TSDB to the eMMC, which is the one thing the bind exists to prevent.
prowlarr.nix has no tmpfiles rule only because its directory predates the
module (migrated from ZimaOS); this is a fresh service, so it creates its
own, same as seerr.nix. Verified on jupiter: the mount is live on md127 and
nothing lands on the OS disk.
scrape_timeout was left implicit at the Prometheus default of 10s, which is
longer than the 5s interval -- VictoriaMetrics clamps it down rather than
erroring, so the config claimed 10s while the scraper used 5s. Say what
actually happens. Checked with `victoria-metrics -promscrape.config.dryRun`,
not just nix eval, which never builds the checked-config derivation.
Also comments: why the bind exists and why `nofail` is load-bearing (the
fileSystems block had none, unlike prowlarr.nix and seerr.nix), and why
mercury needs its own job -- scrape_interval is per-job and job_name must be
unique, so its `job` label will always differ from the other hosts'. Select
on `host` in dashboards or mercury drops out of them silently.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
The hook registered with no events at all and delivered nothing.
"pull_request_review_comment" and "pull_request_review_rejected" are real
HookEventTypes and real X-GitHub-Event-Type values, but they are not
things gitea's hook API accepts. updateHookEvents
(routers/api/v1/utils/hook.go) matches a fixed list of api names and
silently ignores anything else, so every event flag stayed false, the POST
succeeded, and the hook sat there inert.
There is no narrower api name: HasEvent (models/webhook/webhook.go)
collapses approved, rejected and review-comment onto
HookEventPullRequestReview, so `pull_request_review` is a single switch for
all three. Approvals consequently cannot be excluded at the hook any more.
They now cross the wire as "pull_request_approved", which is not in the
route's event list, so Hermes ignores them on the event match -- before the
filter script and before any LLM call. Gitea's delivery log will show them
answered 200/ignored, which is intended.
That makes three namespaces for the same event rather than two, so the
tables in both nix files and the README now carry the api column, and the
README warns about the silent-ignore behaviour that hid this.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Fills in the subscription/wire name table for all five mappings rather
than the two prose examples, and records why the routes are written as
config instead of subscribed -- including that the toolset grant is
deliberate but not enforced, since the file it lives in is inside
HERMES_WRITE_SAFE_ROOT.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
One webhook per route, from a list, so adding a route is an entry rather
than a copy of the unit. The PR-review hook subscribes
pull_request_review_comment and pull_request_review_rejected.
The unit runs as the gitea user on a multi-user box, where
/proc/<pid>/cmdline is world-readable for the lifetime of the process, so
`-H "Authorization: token $t"` published the admin token and
`jq --arg secret "$s"` the webhook secret -- which is exactly what the
existing comment claimed to be avoiding by putting the body on stdin. The
token now goes through a 0600 `curl -K` config written with printf (a
shell builtin, so the substitution never reaches an argv) and the secret
through jq --rawfile.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
`hermes webhook subscribe` has no --toolsets flag, so a webhook run got
Hermes's constrained default (web_search, web_extract, vision_analyze,
clarify) -- no shell, no file access, which meant neither prompt could
actually be carried out: luna was woken, read the comment, and had no way
to act on it. Upstream's documented answer is to add the `toolsets` key to
webhook_subscriptions.json by hand, and a hand edit does not survive this
unit's re-provision. So the whole route definition moves here and the CLI
is not used at all.
The file is written host-side with jq. hermesHome is the bind-mount source
for /opt/data, so the container sees the same inode and hot-reloads it on
the next delivery -- no podman exec, no readiness loop, and no quoting
chain between nix and the prompt text. The merge is per-route: routes this
unit does not name survive, created_at is carried over, and every other
key is replaced outright so a hand-added `deliver_only` or `filters`
cannot linger.
The secret now comes from the sops file directly instead of being read
back out of the container's environment, which drops podman-hermes-agent
from restartUnits (the ordering constraint it existed for is gone) and
takes GITEA_HERMES_WEBHOOK_SECRET out of an env var luna can read.
The new gitea-pr-reviews route covers reviews with a body and
changes-requested. Those are not IssueCommentPayloads: gitea sends a
PullRequestPayload with action "reviewed" and a `review` object of exactly
{type, content} -- no review id, no line comments. So the prompt fetches
them with `tea pulls review-comments` and acts only on ones whose
`resolver` is empty, resolving each as it goes; with no stable id in the
payload, resolved state is the only workable duplicate-delivery guard.
An empty review body is deliberately NOT a drop, unlike in the comment
filter: a review whose substance is entirely in line comments has none.
Approvals are left unsubscribed -- an approval is darman signing off, not
asking for work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Not executable under the toolset a webhook run actually got: Hermes
defaults those to web_search/web_extract/vision_analyze/clarify, with no
shell. Worth revisiting now that the routes grant `terminal` explicitly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Every drop so far has been silent. The script printed its reason to stderr
and exited 0 with "[SILENT]", but Hermes only logs stderr on the nonzero
path, as
script ignored webhook path=... code=... stderr=...
so from outside, a deliberate drop, a crash, a timeout and a missing file all
looked identical: {"status":"ignored","reason":"script"} and nothing else.
Finding out which one it was meant re-running the payload through the script
by hand.
Drops now exit 3 with an empty stdout. Both still mean "ignored" to Hermes,
but the reason lands in the log. Exit 3 rather than 1 keeps a deliberate drop
distinguishable from an unhandled exception, which exits 1, so the code alone
says which happened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
A timeline comment on a PR never reached the route. Gitea reuses the same
strings in two namespaces and they collide:
subscription name wire name (X-GitHub-Event) what it is
pull_request_comment issue_comment comment on a PR
issue_comment issue_comment comment on an issue
pull_request_review_comment pull_request_comment review on a PR
The hook's `events` array takes the subscription name; Hermes matches
--events against X-GitHub-Event, the wire name, produced by
HookEventType.Event() in modules/webhook/type.go. So --events
pull_request_comment was selecting review submissions and could never match a
comment -- the exact inversion of what it reads like.
That also explains both observed failures. The review submission matched
(wire name pull_request_comment) and reached the filter, which correctly
dropped it on action=reviewed since a PullRequestPayload carries no comment
object. The timeline comment arrived as issue_comment, matched nothing, and
was dropped by the events filter before the script ever ran.
gitea.nix and hermes-agent.nix now deliberately name the same event
differently, so both carry the table and say the other is not a typo.
issue_comment on the wire also covers comments on plain issues. The hook does
not subscribe those, and the filter's is_pull check drops them regardless, so
widening the hook later cannot leak issue comments into the agent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Names the webhook "PR comments Hermes" (gitea's CreateHookOption/EditHookOption
both carry an optional `name`, so it survives the create and the update path)
and narrows it from all 26 event types to pull_request_comment alone.
Gitea sends pull_request_comment distinctly from issue_comment, so the hook
now covers comments on pull requests and nothing else. Hermes would have
dropped the rest anyway -- its route filters on X-GitHub-Event before any LLM
call -- so this is defence in depth rather than the only gate, but it keeps
traffic that can never be acted on from crossing the wire and reaching the
agent's process at all.
The tradeoff is that event selection now lives on both sides: a second Hermes
route needs its event adding here as well as being subscribed. That is the
right way round for a single-purpose hook, and the comment says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
The relay existed on the premise that Gitea sends no header Hermes can read
an event name from, so something had to copy X-Gitea-Event into
X-GitHub-Event. That premise was wrong. Gitea's addDefaultHeaders sets
req.Header["X-GitHub-Delivery"] = []string{t.UUID}
req.Header["X-GitHub-Event"] = []string{event}
req.Header["X-GitHub-Event-Type"] = []string{eventType}
unconditionally, for every webhook type, alongside X-Hub-Signature-256 in
GitHub's exact format. (Direct map assignment rather than .Add() specifically
to keep the "GitHub" casing that canonicalisation would destroy.) Hermes
validates that signature on any route without provider gating and reads the
event name from that header, so gitea and hermes already speak the same
protocol and the translation layer was translating nothing.
Gitea now posts straight at http://mars.orbit.sol:8644/webhooks/gitea-pr-comments.
The URL path is the Hermes route name, so a second subscription is a second
hook and nothing else -- the route-in-path indirection the relay grew was a
reimplementation of something Hermes already had.
Removes the module, the 200-line relay, its test, the mars import, the 8645
listener, and the stale gitea-hermes-webhook-relay.service entry left in the
secret's restartUnits. hermes-agent-webhook-route moves to
hosts/mars/hermes-agent.nix, next to the container and the read-only prompt
and filter mounts it depends on.
Also makes that unit refuse to subscribe when GITEA_HERMES_WEBHOOK_SECRET is
unset in the container, matching the existing empty-prompt check. An empty
secret silently fails every delivery signature check afterwards while the
unit still reports success -- the worst possible failure shape, and one this
setup can actually produce on a first deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
The previous token was write:repository only, which clones, fetches and
pushes branches perfectly well and then fails at `tea pr create` — a pull
request is an issue in gitea's data model, so every /pulls endpoint gates on
the issue scope category rather than the repository one.
Regenerated with write:repository,write:issue,read:user. Confirmed against
the running instance: gitea reports the granted set as
read:activitypub, read:misc, read:notification, read:organization,
read:package, write:issue, write:repository, read:user
so write:issue is present rather than only read:issue, which would satisfy
the GET half and still fail the POST that opens the PR. The extra read:*
categories are gitea expanding the request, not something asked for.
No manual step on mars: gitea_luna_token already restarts
hermes-agent-prepare-dirs, which does delete-then-add for the tea login on
every start and so picks up the rotation by itself.
The old token is NOT revoked — gitea's CLI cannot delete tokens and the API
route needs basic auth as luna, which nothing here sets. It stays valid until
removed by hand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Mirrors the `hermes` alias on mars. The admin CLI is effectively
undiscoverable without it: the package is not in systemPackages so `gitea` is
not on PATH at all, every admin subcommand needs GITEA_WORK_DIR pointed at a
stateDir that is not the module default, and it has to run as the gitea user
or it drops root-owned files into that directory. Getting any of the three
wrong fails in a different and unhelpful way.
Both the package path and the stateDir come from the config rather than being
written out, so a gitea bump or a stateDir move cannot leave the alias
pointing at something stale — which is exactly what a hardcoded /nix/store
path would do.
Lives in services/dev/gitea.nix, which only jupiter imports, so it does not
leak onto hosts with no gitea to administer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Webhook delivery to the hermes relay was refused outright:
Post "http://mars.orbit.sol:8645/gitea/gitea-pr-comments":
dial tcp 100.64.0.6:8645: webhook can only call allowed HTTP servers
(check your security.ALLOWED_HOST_LIST setting),
deny 'mars.orbit.sol(100.64.0.6:8645)'
ALLOWED_HOST_LIST defaults to `external`, documented as "a valid non-private
unicast IP". Tailscale addresses come from 100.64.0.0/10 — RFC 6598
carrier-grade NAT space — which is not RFC1918 private but does not satisfy
gitea's notion of external either, so every tailnet target is denied by
default. Nothing about the relay or the URL was wrong; the request never left
jupiter.
Sets the tailnet CIDR explicitly and keeps `external`, so a future webhook to
a public service still works without another edit here.
Goes in [security], not [webhook]: the webhook-section key is deprecated in
favour of this one and now merely falls back to it, and [security] is the
name the delivery error itself reports.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
jq was only ever on the `path` of the units that call it, so it was absent
from an interactive shell — which made the hook-migration commands in the
README unrunnable on the host they target. It is a general-purpose tool and
every host already carries curl, so it belongs alongside it rather than being
pulled in per-unit.
Also simplifies those README commands now that jq is present, and uses mars's
existing `hermes` alias instead of spelling out the podman exec.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Retiring gitea-events is a one-off migration, not something worth re-running
on every boot. Both units now only touch what they own: jupiter's creates or
updates its own hook and deletes nothing, and mars's removes only the route
it is about to re-subscribe, as the idempotency step for `subscribe`.
Keeping the deletes would have meant a redeploy could silently remove a hook
or route someone added deliberately -- a real risk now that sibling hooks
for other Hermes routes are the intended pattern.
README carries the manual commands, and the note that both hooks fire until
the old one is removed by hand, so events arrive twice in the meantime.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Renames the subscription to gitea-pr-comments (it handles one event; the old
gitea-events name promised more than it delivered) and drops --deliver.
Rather than move the hardcoded route from one constant to another, the relay
now reads it from the request path: POST /gitea/<route> forwards to
<base>/webhooks/<route>. The route name was the last thing tying this service
to a specific subscription, so a second Hermes route is now a `hermes webhook
subscribe <name>` plus a Gitea hook at /gitea/<name>, with no relay change --
previously it would also have needed a second relay URL baked in here.
The path segment is interpolated into an outbound URL, so it is validated
against ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$ and refused rather than sanitised
when it does not match. The path is matched raw and never URL-decoded, so
percent-encoded separators fail the charset check instead of surviving it;
requiring an alphanumeric first character also rejects "." and "..". Without
this, POST /gitea/..%2fadmin would let anything that can reach the relay
steer it at other Hermes endpoints. Tests cover traversal, encoded traversal,
embedded slashes, leading dot/dash, and the length bound, and assert nothing
reaches the stub Hermes in any of those cases.
Dropping --deliver leaves it at its default of `log`. The prompt tells her to
answer in the pull request, so the PR comment is the delivery and a Telegram
copy would only duplicate it; this also removes the hardcoded chat id that
was a third copy of TELEGRAM_HOME_CHANNEL.
Provisioning retires the pre-rename hook by its EXACT old URL rather than by
"points at the relay". Now that sibling hooks for other routes are the
intended pattern, a prefix match would delete them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Completes the subscription: it had a secret, a delivery target and a script,
but no prompt and no event list, so it woke the agent on every forwarded
event with nothing to tell her what to do.
--events pull_request_comment narrows the route to the one event the prompt
handles. This only works because the relay copies X-Gitea-Event into
X-GitHub-Event; without that every delivery arrives as "unknown" and matches
nothing. Gitea sends pull_request_comment distinctly from issue_comment, so
plain issue comments no longer reach the agent at all. The Gitea-side hook
still posts the full event set to the relay and Hermes drops the rest before
any LLM call.
The prompt lives in hosts/mars/gitea-pr-comment-prompt.md, mounted read-only
next to the filter, and is read with $(cat) at subscribe time rather than
passed inline. That is not only about escaping: the text has to survive nix
`` string escaping, the systemd unit file, and `podman exec sh -c '...'`
single-quoting. It contains an apostrophe ("the PR's head branch") that
would terminate that single-quoted string early. Read from a file at runtime
the content never passes through shell source, so it can contain anything.
Verified end to end against the rendered unit with stubbed podman/hermes:
the value reaching --prompt is byte-identical to the repo file apart from
the trailing newline that command substitution strips.
`set -eu` inside the container shell is load-bearing. Without it a missing
prompt file makes cat fail, the substitution yields "", and the subscription
is created with an empty prompt -- a silent failure that still looks like a
healthy unit.
On what read-only does not buy: it protects the sources, and this unit
re-subscribes from them on every start, so a restart restores the intended
prompt, filter and events. The live subscription itself lives in
webhook_subscriptions.json under /opt/data and is hot-reloaded, which is
inside the agent's own write-safe root -- a self-modification would stick
until this unit next runs.
The prompt keeps its own stop conditions even though the filter already drops
those deliveries, and says explicitly that reaching them means the filter
failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
The gitea-events subscription woke the agent on every delivery. That is an
unbounded loop as soon as she is given a prompt that tells her to answer on
the PR: her answer is itself a pull_request_comment, which wakes her again.
Adds a Hermes route script that drops the deliveries that must never reach an
LLM call: luna's own comments (the loop guard), "deleted" actions (the body
is still in the payload, so acting on one means acting on a request that was
explicitly withdrawn), non-pull-request comments, empty bodies, and edits
that did not actually change the body — a label or attachment change fires
"edited" too. Everything else passes through unchanged.
Mounted READ-ONLY from the nix store rather than written into hermesHome.
Hermes resolves route scripts under ~/.hermes/scripts, which here is inside
/opt/data — HERMES_WRITE_SAFE_ROOT — so a filter written there would be a
loop guard sitting in the writable root of the agent it constrains. Deleting
it fails closed (Hermes treats a missing script as "ignore"), but rewriting
it to always-allow would silently restore the loop. Read-only from the store
makes that impossible and keeps the guard in git.
The script also normalises changes.body.from to always exist. Gitea omits
`changes` entirely on created events, and Hermes replaces the prompt payload
with whatever JSON the script emits, so guaranteeing the key here means a
prompt referencing {changes.body.from} renders empty instead of leaving an
unfilled placeholder.
Note the stdout contract (gateway/platforms/webhook.py): only exactly
"[SILENT]", empty output, or a nonzero exit drop a delivery. Any OTHER text
on stdout lets it through and is attached as script_output — so a stray
debug print would silently defeat the filter. All diagnostics go to stderr,
and gitea-pr-comment-filter-test.py asserts that discipline along with each
drop rule (25 cases). Run it after any edit: the fail-closed behaviour means
a syntax error produces silence, not an error.
--events is still unset; event selection remains runtime-tunable policy.
The filter covers only what must not be.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Three fixes to how luna's git/tea credentials are set up on mars, all found
against the running instance on 2026-08-23.
Drop the host-side clone. hermes-agent-prepare-dirs used to clone this repo
into ${hermesHome}/workspace/homelab, but nothing ever told luna at runtime
that it was there — she self-manages config/profiles/memories, so a path
baked into this file never reached her. She searched /opt/data/homelab and
/workspace, found neither, and concluded she had no repo at all. The
credentials are what actually grant access; any checkout is hers to make
anywhere inside HERMES_WRITE_SAFE_ROOT. The stale directory left by the old
version is deliberately not cleaned up, just unmanaged from here on.
Point credential.helper at the CONTAINER's path. It was written as the host
path (${hermesHome}/.git-credentials), which does not exist inside the
container where git actually reads the config — broken this way from 3c1f3e5
until now. Nothing host-side consumes those credentials any more, so the
container's view is the only one that has to be right; added `containerHome`
to make the distinction explicit at the point of use.
Chown what the oneshot writes. The image's cont-init only chowns the top
level of hermesHome and its own state — it does not recurse into the
root-owned 0600 files this unit drops there (.git-credentials, and tea's
config.yml, which tea also writes 0600), even though it runs afterwards. The
symptom was not an error but an absence: git reported no credential helper
and tea no login. Uses `if` rather than `[ -d x ] && chown` because under
`set -e` a false test on the left of an && list aborts the unit.
gitea.nix carries the matching comment updates: the luna provisioning unit is
server-side only, and her token needs write:repository,write:issue,read:user.
write:issue is the one that is easy to miss — a pull request IS an issue in
gitea's data model, so /pulls endpoints gate on the issue scope category and
`tea pr create` fails with write:repository alone even though clone, fetch
and push all work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Shared HMAC secret for the Gitea -> Hermes webhook relay merged in #2. Both
hosts need the same value: jupiter signs deliveries with it (the webhook
config registered by gitea-hermes-webhook-provision), and mars verifies them
in the relay and hands it to Hermes through the hermes-agent.env template.
32 random bytes, hex-encoded, stored with no trailing newline — the value
reaches Hermes via an env-file template where a newline would both corrupt
the file and change the key the HMAC is computed with.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
The relay was forwarding X-Gitea-Event and re-signing the body into the
deprecated generic-V1 X-Webhook-Signature header. Neither is something
Hermes acts on, which left the PR's core premise — "Hermes owns event
selection" — impossible to reach:
- Hermes reads the event name only from X-GitHub-Event/X-GitLab-Event,
then payload event_type/type, then falls back to the literal string
"unknown" (gateway/platforms/webhook.py). Gitea sends X-Gitea-Event and
no such payload key, so every delivery arrived as "unknown" and
`hermes webhook subscribe --events ...` could never select anything.
- Gitea's addDefaultHeaders() already signs every webhook type with
X-Hub-Signature-256 in GitHub's exact format, and Hermes accepts that
header on any route with no per-route provider gating. Re-signing into
V1 was both redundant and on a deprecated path.
So the relay now verifies the signature (accepting either X-Hub-Signature-256
or X-Gitea-Signature), forwards body and signature byte-for-byte, and copies
the one header Hermes actually needs. Authentication alone never justified
this service; that header copy does, and the module comment now says so.
Also fixed:
- gitea-hermes-webhook-provision had no API readiness wait, unlike both
sibling units in the same file. After=gitea.service does not mean gitea
is serving HTTP, so under `set -e` a Type=oneshot with no Restart= would
fail on first boot and stay failed, leaving the webhook unregistered.
- podman-hermes-agent added to the secret's restartUnits. The secret
reaches the container only via sops.templates, whose rendered path never
changes, so systemd would not restart the container when the secret was
first added — hermes-agent-webhook-route then read an empty value back
out of it and subscribed with an empty secret.
- Webhook provisioning passes the request body to curl on stdin rather
than in argv, keeping the shared secret out of /proc/<pid>/cmdline.
- Missing Content-Length now returns 411 rather than 413; dropped the
unreachable non-2xx branch (urlopen raises on non-2xx); env-var secret
fallback is stripped to match the credential-file path.
Adds gitea-hermes-webhook-relay-test.py, which drives the real relay over
real HTTP against a stub Hermes and covers the header copy as a regression
test. Both nixosConfigurations still evaluate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S94o42aQ8VkBmEWvDem5xa
Provisions a dedicated PR-tier gitea account (luna) with branch protection
restricting master push/merge/approve to darman only, then wires git and
tea directly into the hermes-agent container (mounted from the host's Nix
store, credential-store + tea login set up by a host-side prepare oneshot,
repo cloned inside Hermes's own writable sandbox root at
/opt/data/workspace/homelab). Replaces an earlier standalone MCP-server
approach, scrapped in favor of direct CLI access for simplicity.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011FHr5ug9pu8q4XPrRkFnzJ
terra was the only host with real ~/.zshrc/.zshenv (via home-manager),
so it never hit the plain-zsh zsh-newuser-install wizard that shows up
on first login everywhere else. Wire home-manager.nixosModules.home-manager
into jupiter/neptun/mars/mercury (+ mercury-vm/jupiter-vbox test targets)
and give darman terra's shared zsh baseline via home/common.nix, imported
from common.nix. terra's own home.nix now only carries its
desktop/dev-specific profile (Hyprland, alacritty, git identity, direnv,
dev packages) layered on top via home-manager.users.darman.imports.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Du1WQRk1F8DenrPhf8TofF
darman's own podman is rootless while the container runs under root's
(system) podman, so plain `podman exec` couldn't see it. Alias runs it
with sudo against the right socket.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Du1WQRk1F8DenrPhf8TofF
podman-hermes-agent.service RequiresMountsFor /mnt/jupiter, but the mount
option copied from terra's (read-only, nothing depends on it) browsing
mount included x-systemd.idle-timeout=60 — confirmed on real hardware,
this killed the container ~60-70s after every start with no crash or
error, just an idle auto-unmount taking the dependent service down with
it. Keep the lazy x-systemd.automount (so boot doesn't stall if jupiter's
down) but drop the timeout now that something needs the mount to persist.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011FHr5ug9pu8q4XPrRkFnzJ
New on-site host mars runs Hermes Agent as its sole service: joins the
tailnet, mounts jupiter's samba share at /mnt/jupiter (doubling as
Hermes's shared dropbox), and hosts state locally under /var/lib/hermes.
Same Authentik OIDC app/Telegram bot as before, just relocated — neptun's
hermes.mgaction.town vhost now points at mars.orbit.sol instead of jupiter.
hosts/jupiter/hermes-agent.nix and its three sops secrets are removed;
jupiter's Caddy vhost for it is gone too. Also refreshes tailscale_authkey
across all hosts and fixes two stale "erik@laptop" keys in flake.nix's
kexec/installer-iso images (leftover from a previous laptop, already
swapped out of common.nix back in 2fd5752) to darman@terra.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011FHr5ug9pu8q4XPrRkFnzJ
The container has no host /etc/localtime bind-mount, so hermes_time.py's
timezone resolution fell through to UTC. HERMES_TIMEZONE is its
highest-priority source (checked before config.yaml's timezone key).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Follow-up to e5ba1bf — that commit only staged the deleted module file.
Drops the flake module import, opencode_go_api_key/telegram_bot_token
secrets, and the stale hermes-agent.nix cross-reference in ollama's
context_length comment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Consolidating on jupiter's isolated instance (hosts/jupiter/hermes-agent.nix)
instead of running a second one here. Drops the module import, its
opencode_go_api_key/telegram_bot_token secrets, and the now-stale
cross-reference in ollama's context_length comment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A separate instance from terra's, deliberately locked down harder given
jupiter's much bigger blast radius (irreplaceable immich photos on an
unredundant RAID0, gitea/CI tokens, the whole media stack): its own
dedicated "hermes" system user rather than darman (who is in jupiter's
root-equivalent docker group), container.enable = true for whole-process
containment rather than native/bare-metal, its own Telegram bot + explicit
allowlist, and no volume access to /mnt/data or this repo. stateDir/
workingDirectory live on the array (off the 29G eMMC) for future coding-task
state, guarded by RequiresMountsFor like the rest of jupiter's array-backed
services.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
nix.gc (common.nix) is weekly, too slow to catch a switch pinning the old
generation's closure on a 29G eMMC — add a full nix-collect-garbage on every
boot instead. Also set boot.swraid.mdadmConf so eval stops warning that
mdmon will crash (dormant here: the RAID0 array uses native superblocks, so
mdmon never actually runs).
sabnzbd.configFile is deprecated by the module; move to services.sabnzbd.settings
with credentials (web login, api/nzb keys, eweka.nl server) sourced from sops via
secretValues instead of living in a plaintext ini. admin_dir/log_dir are pinned
absolute at their original /mnt/data location so the existing download
queue/history isn't reset by the ini moving to /var/lib/sabnzbd.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Points Nous Research's Hermes Agent at terra's own ROCm ollama server
(gemma4:12b) as a custom OpenAI-compatible provider instead of a cloud
key. Native systemd mode via the hermes-agent flake's own NixOS module
— simpler than container mode, avoids the podman-rootful-sudo dance
its docs call out.
Also bumps OLLAMA_CONTEXT_LENGTH (and Hermes' mirrored model.context_length)
from ollama's ~4k default to 131072, load-tested with real multi-ten-
thousand-token prompts rather than just idle `ollama ps` checks — chosen
as the practical ceiling where VRAM headroom and prefill throughput both
start visibly degrading, not just the largest number that technically fit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Enables hardware.graphics + intel-media-driver for the Apollo Lake's
Gen9 iGPU (VAAPI only — QSV needs an insecure/EOL runtime on this
chip) and adds jellyfin's service user to video/render for the DRI
card node. 4K HDR still can't be tone-mapped on this hardware; those
files need to direct-play or be kept as 1080p SDR.
Also relocates podman's container storage and immich's postgres
cluster to /mnt/data/AppData, after a deploy holding two ~9G closures
at once filled the 29G eMMC and postgres died mid-write.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Running sops --decrypt/edit_secrets --show and displaying the result
puts every plaintext secret in the file wherever that output lands,
not just the one value being checked. Point at `sops --set` instead
for adding/changing a single value non-interactively.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Local LLM server on the 6800 XT (ollama-rocm, gfx1030 needs no
HSA_OVERRIDE_GFX_VERSION) fronted by a LibreChat web UI, talking to it
over the OpenAI-compatible /v1 route. Also wires up LibreChat's
persistent-memory feature, which needed its own agent+model plus a
custom extraction prompt: the default 3b model couldn't reliably tell
the user's stated facts apart from its own boilerplate, and even a
tuned prompt didn't fix that — so memory extraction now reuses
gemma4:12b, the same model as the daily-driver chat endpoint.
flake.lock bump pulls in the ollama and librechat NixOS modules.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
After the rack move one of the RAID0 disks failed to enumerate, and jupiter
boot-looped into an emergency shell nobody could use — root is locked, so
sulogin offers a prompt with no answer, and there is no ssh from there:
Timed out waiting for device /dev/disk/by-uuid/dadbff6f-...
Dependency failed for /mnt/data.
Dependency failed for /var/lib/private/prowlarr.
Dependency failed for Local File Systems.
local-fs.target: Job local-fs.target/start failed with result 'dependency'
Reached target Emergency Mode.
`nofail` on /mnt/data did not help, because the prowlarr and seerr bind
mounts layered on top of it had none: without it a mount is RequiredBy
local-fs.target, so those two failed the target on the array's behalf. Give
them `nofail` too and let them fail alone. `systemd.enableEmergencyMode =
false` then keeps a bad array from costing a reachable box at all — far more
useful on a headless host than a console prompt.
Booting further is only safe if nothing quietly relocates onto the 29G eMMC,
so pin the array-backed services to the mount. systemd derives
RequiresMountsFor from a unit's own paths, which for these is somewhere under
/var/lib (eMMC) — nothing pointed immich at mediaLocation or sabnzbd at its
configFile, so with the array gone they would have started and written to the
OS disk, into directories that go invisible the moment /mnt/data mounts over
them. jellyfin, sonarr, radarr and gitea already had a real dependency and
are untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Workflows push as a dedicated ci-bot account rather than a human one, so its
PAT can be scoped, rotated and revoked on its own. Adding a repo to
`ciBotRepos` and redeploying is all it takes to grant access.
Collaborator access and branch-protection push-whitelisting exist only on
gitea's HTTP API — no CLI, no config-file surface — so this one part stays
imperative: a oneshot that PUT/PATCHes the API into the desired state. It
runs on deploys where the script changed, which means it won't self-heal a
revert done through the web UI unless the unit is restarted too.
Two secrets, deliberately distinct:
- gitea_provisioning_token is darman's own token (write:repository +
write:user). Only an owner-scoped token clears reqOwnerCheck on the
collaborator and branch-protection endpoints, and write:user is what lets
it write the Actions secret below. ci-bot cannot grant itself access.
- gitea_ci_bot_token is ci-bot's push token, generated once by hand (the
command is in the comment) and pushed into gitea as a user-level Actions
secret CI_BOT_TOKEN. Gitea has no instance-wide secret scope, and every
repo here is owned by darman directly rather than an org, so a user-level
secret is the closest thing — repo-level lookups fall back to it.
Branch protection is applied to the default branch plus `develop`, since
version-bump.yml pushes there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Now that the plugin draws its own outline and glow, hyprland's own border is
redundant — border_size 0 and let hyprchrome own the window edge (outline_size
2 in fg_color). Fill in the rest of its knobs: glow 12/0.85, shadow 24 offset
{4,8} in bg_color.
Accents go warm: bg_accent to a muted red (963c38) and a new fg_accent_alt
(ff9d42) so the active-border gradient runs amber->orange instead of
amber->background. Colour literals lowercased for consistency.
Also re-pick the placeholder wallpaper, and drop the comment explaining
hyprland's lua gradient table format — the surviving call site is now the
only one and reads plainly enough.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- install tome from pkgs/tome.nix, built against the re-added flake input
- import services/containers.nix and put darman in `render`/`video`:
/dev/dri/renderD128 is root:render 0660, so a rootless container can only
reach the GPU if the host user is in the group. Needed by the Vulkan
whisper.cpp/llama.cpp containers in content-trigger-scanner.
- point DOCKER_HOST at the podman *user* socket and add docker-compose.
dockerCompat gives a `docker` CLI shim, but compose v2 is its own binary
talking to a socket, and rootless podman's socket is the user one under
/run/user/1000 — not root's /var/run/docker.sock.
- direnv + nix-direnv, so per-repo devShells load in the shell and in Rider
via its direnv plugin, instead of hand-wiring a toolbox SDK per repo
- kicad as a flatpak, alongside the other flatpak desktop apps
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three separate reasons the packaged build didn't behave like the one Rider
launches:
- dotnet-runtime must be aspnetcore_10_0, not runtime_10_0. Tome.App's
runtimeconfig.json requires Microsoft.AspNetCore.App as well as
Microsoft.NETCore.App, because Photino hosts a local Kestrel server, and
only the aspnetcore bundle ships it.
- wrapGAppsHook3, so gappsWrapperArgs get spliced into buildDotnetModule's
own wrap step (it sets dontWrapGApps itself; same pattern as nixpkgs'
libation). Without it nothing sets XDG_DATA_DIRS/GSETTINGS_SCHEMA_DIR, so
GTK/WebKitGTK found neither the icon theme nor GTK settings — missing icons
and a denser default font than in an already-initialized session.
- glib in runtimeDeps. It doesn't arrive via gtk3/webkitgtk's RPATH because
the consumer is Photino.Native.so, a prebuilt binary out of the nuget
package rather than something Nix built and patched.
tome-deps.json is the regenerated nuget lock for the aspnetcore switch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
c5a231b pinned the hash by hand because nixpkgs bumped 1.98.8->1.98.9 without
updating vendorHash (NixOS/nixpkgs#545860). The previous commit's lock moves
nixos-26.05 past the point where that fix was promoted from release-26.05, so
the override is now dead weight — and a stale vendorHash override is worse
than none, since it silently wins over a correct upstream value on the next
version bump.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tome (formerly AudibleLibrary) comes back as a git+ssh input against our own
gitea, fetched with darman's ambient key. `flake = false` — it's a plain
source tree consumed by pkgs/tome.nix, not a flake of its own.
This re-breaks `./scripts/deploy install terra localhost` exactly the way
4f79ec7 removed it for: the installer-iso has no credentials, so the git+ssh
fetch fails at nixos-install, post-disko. Taking that tradeoff knowingly
rather than losing the app from the desktop config again — the note in
flake.nix spells it out for whoever hits it next.
The lock also picks up the routine input refresh, including the nixos-26.05
rev that finally carries the tailscale vendorHash fix (next commit) and a new
client-ts-generator-src node pulled in by authentik-nix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- swap gtk portal/apps for cosmic (xdg-desktop-portal-cosmic, cosmic-files,
cosmic-settings) and drop dolphin/protonplus/bambu-studio (bambu-studio
moved to flatpak alongside the other comms/gaming flatpaks)
- mount jupiter's samba share at /mnt/jupiter (automount, credentials from
the same samba_password secret jupiter itself uses)
- add tea (gitea's remote API CLI) for talking to git.mgaction.town from
terra without SSHing into jupiter
- new dark icon themes (Amy, Azure Glassy, Slot Beauty) vendored from
gnome-look.org tarballs, packaged since pling download links expire
- rishot: fix Qt5Compat.GraphicalEffects QML import (was missing qt6.qt5compat
on QML_IMPORT_PATH, so quickshell failed at config-load)
- launcher widgets: stop LauncherConsole/LauncherDock from reserving
compositor space (ExclusionMode.Ignore, they're overlays not real docks);
bump LauncherCorner app icon size 28->34
- comms script: launch telegram/discord via flatpak, not native binaries
- nix-ld + boot.binfmt aarch64 emulation (for building/flashing mercury
from terra)
configurationLimit prunes generations beyond the cap as part of the switch,
but pruning only drops a generation as a GC root — the store paths
themselves still need an actual collect to free the disk. Do that right
after every jupiter switch rather than waiting up to a week for
gc.dates=weekly to matter again.
Own Hyprland plugin (border + title bar), public repo on our own gitea,
fetched over https (no credentials needed). nixpkgs.follows keeps its build
ABI-correct — Hyprland plugins are ABI-locked to the exact Hyprland build
they load into, so it has to build against this flake's own nixpkgs rather
than whatever hypr-chrome's own flake.lock pins standalone.
TEMPORARY: nixpkgs bumped tailscale 1.98.8->1.98.9 without updating
vendorHash (NixOS/nixpkgs#545860, fixed on release-26.05 but not yet
promoted to the nixos-26.05 channel branch this flake tracks). Remove once
`nix flake lock --update-input nixpkgs` picks up a fixed rev.
Addressing is fully static, but netcup's router still sends periodic RAs on
this segment; the kernel then tries (and fails, since the static route
already exists) to install its own default route from them, spamming
"ndisc_router_discovery failed to add default route" on the console.
Runner registers against the same gitea instance and runs jobs in podman
containers (services/containers.nix), one image per runs-on label using the
catthehacker act-compatible images. Registration token comes from gitea
itself (gitea actions generate-runner-token) and is stored in
secrets/jupiter.yaml, rendered into a TOKEN=... env file via sops.templates
since gitea-actions-runner takes an EnvironmentFile, not a raw secret path.
The old key was a leftover from a previous laptop. Also cap every host at
5 boot generations and journald at 200M so none of them can quietly repeat
jupiter's 34-generations-on-a-29G-eMMC incident.
The staged installer booted, the auto-install service picked up terra's host
key and removed its temporary UEFI entry — then died before running anything:
env: 'bash': No such file or directory (status 127)
The service ran with the restricted PATH a `path = [...]` list produces, which
has no bash — so `./scripts/deploy`'s `#!/usr/bin/env bash` shebang could not
resolve, let alone the nix / nixos-install / git / sudo it then calls.
Point the unit's PATH at /run/current-system/sw/bin (+ /run/wrappers/bin for
sudo), which carries the whole installer toolset. mkForce because NixOS
otherwise derives environment.PATH from `path` and that line would win. HOME
moves into the same environment attr.
Verified: environment renders {HOME=/root,
PATH=/run/current-system/sw/bin:/run/wrappers/bin}, and sw/bin contains bash,
nix, nixos-install, git, sudo, efibootmgr, mount, grep, sed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
findiso now works — the installer loop-mounted the iso, mounted the store
squashfs and the overlay — then died with:
stage 2 init script (/mnt-root//init) not found
The live ISO's root is a tmpfs; stage 1 locates the real system's init via
init=<toplevel>/init, which the grub/isolinux menu supplies on a normal boot
(iso-image.nix:47,159). EFI-stub-booting our own cmdline off the ESP, we
never passed it, so stage 1 fell back to /mnt-root/init on the empty tmpfs.
Build the installer-iso toplevel and prepend init=$toplevel/init to the
cmdline (both boot modes). That path resolves once the store squashfs mounts,
because the iso carries the full closure of its own toplevel. Also switch
root=fstab -> root=LABEL=<volumeID> to match what the ISO menu passes (findiso
overwrites /dev/root regardless), and add boot.shell_on_fail for a shell
instead of the reboot/ignore prompt if stage 1 ever fails again.
Verified: the generated cmdline carries init=/nix/store/<toplevel>/init and
that store path contains /init.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
terra booted the staged installer this time but dropped to an emergency
shell: stage 1 mounted /sysroot, then timed out on /sysroot/nix/.ro-store
waiting for /dev/disk/by-label/nixos-minimal-26.05-x86_64.
findiso= is handled only in the scripted stage-1-init.sh, which loop-mounts
the file the param points at and symlinks it to /dev/root. The systemd initrd
— the default since 26.05 — has no findiso handling at all: iso-image.nix
mounts /iso directly from /dev/disk/by-label/<volumeID>, a label that only
exists when the ISO is the physical boot medium. Booted as kernel + initrd
off the ESP with the iso as a plain file on another partition, that label
never appears, so the store squashfs never mounts.
The entire `install <config> localhost` path is built on findiso, so pin the
installer to script stage 1. Verified: /iso device flips to /dev/root,
root=LABEL=... is added to the params, the rebuilt initrd's /init is
stage-1-init.sh and carries the findiso logic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`install <config> localhost` assumed systemd-boot. terra's CachyOS boots
Limine, so it stopped at "/boot/loader/entries doesn't exist" — the check
added in e538788 doing its job, but with nowhere to go.
Limine cannot help here at all: `bootctl status` lists it as
`✗ One-shot entry control`, and CachyOS's pacman hooks regenerate
limine.conf regardless. So drop below the bootloader entirely and use the
firmware's own BootNext, pointing at a temporary UEFI entry that
EFI-stub-boots the installer kernel straight off the ESP. That keeps the
property which makes this safe to attempt: BootNext is spent by that one
boot, so a failed try still comes back on the normal bootloader.
- picked at runtime: systemd-boot loader entry when $BOOT/loader/entries
exists, else arm_efi_bootnext(). jupiter/neptun and terra-after-install
keep the systemd-boot path.
- `efibootmgr --create-only`, NOT `--create`: the latter pushes the entry
to the front of BootOrder, which would make a wiped installer the
permanent default if the install died halfway.
- the EFI stub loads initrd= from the volume it was loaded from, so this
mode stages on --print-esp-path rather than --print-boot-path.
- stale entries from an earlier attempt are removed before adding one, and
homelab-auto-install.service deletes the entry as soon as it boots, so
nothing lingers in NVRAM pointing at a reformatted partition.
- label matching is EXACT ("Homelab Installer"); a prefix match would have
deleted this box's Windows or Limine entry.
Verified against terra's real NVRAM (read-only): the label parser picks out
Limine/UEFI OS/Windows by exact name and rejects prefixes, and both branches
run end-to-end under stubs — BootNext mode emits the right --disk/--part,
loader path and initrd= cmdline, systemd-boot mode still writes its entry and
never calls efibootmgr.
README/CLAUDE.md corrected: terra runs Limine, not systemd-boot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The `install <config> localhost` auto-path added in 0ea9020 could not have
completed. Fixed three blockers plus the guard that was silently not guarding.
Inside installer-iso the run died before doing anything:
- systemd sets no $HOME for a service without User= (SetLoginEnvironment=
defaults to false), and this script runs under `set -u`, so it aborted on
the bare $HOME with "unbound variable". Added $KEYDIR + Environment=HOME.
- the host key it needs to seed /etc/ssh isn't on the ISO at all — that is
built from a public repo and carries no credentials on purpose. It now
travels on the boot partition, located via homelab.keypart=<PARTUUID> on
the kernel cmdline, and dies with the disko wipe minutes later. Without
it sops can't decrypt on boot #1 and mutableUsers locks darman for good.
- installation-cd-minimal leaves experimental-features unset, so both
`nix run` and `nixos-install --flake` failed. (The nixos-images kexec
installer sets them itself, which is why the same branch worked after
kexec-local but not from the ISO.)
The staging-dir guard passed everything on btrfs: findmnt prints the
subvolume as /dev/sdb2[/@], lsblk can't open that, and an empty parent was
treated as "different disk" — so it allowed staging the iso on the very disk
disko then wiped. terra's current CachyOS root is exactly that layout. Now
uses --nofsroot, resolves EVERY whole-disk ancestor (LVM/RAID span several:
/mnt/ssd_01 -> sdd+sde), and treats "can't tell" as a hard error. btrfs
staging is refused outright — stage-1 mounts a btrfs volume's top level, so
an iso inside a subvolume is unreachable.
findiso= lost its leading slash whenever the staging mountpoint was /,
giving /findisovar/tmp/x.iso and an emergency shell after the reboot.
Also:
- confirm before rebooting, like flash/kexec-local already do; --yes skips
it and is what the ISO passes itself
- $BOOT from `bootctl --print-boot-path`, not a hardcoded /boot
- free-space checks on both target partitions before the ~1GB copy
- `nix run .#disko` / `.#nixos-anywhere` from locked inputs instead of
github:... master-of-the-day, resolved while a disk is being wiped
- one_match warns instead of silently taking [0]; require_tracked covers
every hosts/<config>/*.nix; flash traps its mount
- drop nixos-images' `inputs.nixpkgs.follows` — that input doesn't exist,
it only printed a warning on every nix command
Verified: the prepare path exercised under stubs against this box's real
disks (btrfs-on-OS-disk, tmpfs, LVM, subdirectory), shellcheck clean, all
six configs evaluate, checks.kexec-local still passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
./scripts/deploy install <config> localhost now branches on is_live_installer()
(checks uname -n): outside a live installer it builds installer-iso, stages
its kernel/initrd on the ESP and the iso file on a disk the caller picks
(never auto-picked — the wrong disk here is destroyed mid-install), writes a
systemd-boot one-shot findiso= entry with homelab.install=<config> on the
kernel cmdline, and does a real systemctl reboot (not kexec — terra's
kexec-local hang is specifically in kexec's device-shutdown pass, a real ACPI
reboot never runs that code at all).
installer-iso gains homelab-auto-install.service: once homelab-checkout.service
clones the repo, it reads homelab.install= back off /proc/cmdline and re-runs
the identical deploy command itself, now genuinely inside the installer, so
it takes the disko+nixos-install branch instead of preparing again. The whole
reinstall is one command and unattended after the first reboot.
Also: every root-requiring path (kexec-local, the new prepare-and-reboot
branch, the disko+nixos-install branch) self-elevates via a require_root()
helper that re-execs the original invocation under sudo -E, instead of dying
and asking the caller to prefix sudo themselves. Uses an absolute script path
captured before the script's own cd, so the re-exec is correct regardless of
how it was invoked.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
require_tracked() in scripts/deploy now skips its git-tracked-file check
when there's no .git at all (nothing can be untracked in that case) — needed
for an earlier baked-in-`self` approach and kept as a generic fallback.
Since the repo is public now, installer-iso instead clones current master
via a homelab-checkout.service (after network-online.target) on every boot,
to /root/homelab. One ISO build stays useful indefinitely instead of going
stale, and there's still no rsync-the-repo-over step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed on real hardware: kexec's device_shutdown() pass runs (SCSI disks
sync fine in the log) then the machine goes dark for good — journalctl
--list-boots showed a ~15min gap before the next boot, a genuine hang needing
a manual power cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT):
discrete AMD GPUs are known to hang during kexec's device-shutdown pass with
no clean handoff before the jump, same class of issue as jupiter's
reboot=pci warm-reboot workaround, just fatal here instead of slow.
README's terra install section now leads with the USB installer path instead
(build ISO, dd to USB, rsync the repo over, disko + nixos-install locally).
CLAUDE.md's gotchas list gets the same warning. installer-iso is renamed from
jupiter-installer to homelab-installer since it's genuinely host-agnostic,
and now ships git.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Tome (née AudibleLibrary) is darman's own Photino/.NET desktop app, private
repo on our own gitea. Fetched as a flake input over ssh with darman's
ambient key — same mechanism as any other git input, private or not.
buildDotnetModule package: the Preact/Vite frontend (Tome.App/ClientApp)
builds as its own buildNpmPackage derivation and gets copied into the
published app's wwwroot, since upstream's in-project MSBuild npm target has
no network access in the Nix sandbox. Photino.Native's runtime deps (gtk3,
webkitgtk_4_1, libnotify) are wrapped in — confirmed via readelf/ldd that
this Photino build already targets webkit2gtk-4.1, not the now-removed 4.0.
Also added dotnet-sdk + nodejs to terra for developing Tome locally, and
vivaldi (unfree, extends the existing allowUnfreePredicate).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Not in nixpkgs — upstream ships a shell launcher + QML tree with no build
step, driven entirely by `qs -p <dir>`. Packaged as a stdenvNoCC derivation
that wraps the launcher with RISHOT_CONFIG_DIR (sidesteps its argv0-relative
self-lookup, which wrapProgram breaks) and its runtime deps on PATH.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Not in nixpkgs; packaged by github:tomsch/proton-pass-cli-nix. Used by
./scripts/deploy to autofill sudo/ssh passwords from the "HomeLab" vault.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Config was symlinked from ~/.dots/quickshell (separate dotfiles repo); now
tracked here and applied via home-manager xdg.configFile.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces CachyOS on the OS SSD (Kingston SA400, disko-managed). Dev-data
disks (sdc ext4 /mnt/hdd_01, LVM vg_ssd /mnt/ssd_01) stay out of disko and
are mounted as plain filesystems so they're never wiped. Desktop split into
services/desktop/desktop-hyprland.nix (session: compositor, greeter, audio,
portals) and desktop-apps.nix (things darman actually launches, including
claude-code — allowlisted alongside the other unfree desktop apps).
clientId was still the placeholder "immich" instead of Authentik's actual
generated id, and the sops secret it points at (immich_oauth_client_secret)
was never declared on jupiter. Wire both up, and set endSessionEndpoint so
logout lands back on immich instead of Authentik's "logged out" page.
kexec-local could never actually jump. nixos-images' kexec-run.sh ends with
`nohup sh -c "sleep 6 && $SCRIPT_DIR/kexec -e" &` and returns immediately, so
the EXIT trap's `rm -rf "$stage"` deleted the kexec binary out from under the
sleeping shell. The box stayed on the old kernel and it looked like a slow boot.
Clear the trap before jumping, verify /sys/kernel/kexec_loaded, then sleep past
the timer.
Preflight everything before the point of no return, since this jumps the machine
you are typing at: CONFIG_KEXEC, kernel lockdown, exec-capable staging dir, free
space, RAM vs image size, and that the tarball holds all five expected files.
Stage on /var/tmp rather than /tmp because kexec-run.sh appends to initrd in
place and execs from that directory. sync before jumping (kexec -e skips
unmount). Confirmation prompt naming the host, since run in the wrong terminal
this kexecs the laptop; --yes skips it.
Drop the ssh-keygen -R added to the remote kexec path: kexec-run.sh copies
/etc/ssh/ssh_host_* into the appended initrd and restore-remote-access.nix
installs them back, so the host key survives the jump.
Proton Pass items are now keyed by <config> instead of <host>, since the address
is incidental and the config name is stable. kexec therefore takes <config>
<host>. Resolve titles among --filter-state active items first: a trashed item
with the same title shadowed the active one and returned an empty password,
which is indistinguishable from "no entry" and silently fell back to prompting
(hit on darman@neptun).
Other fixes: replace `ls glob | head -1` (returns empty with exit 0 on no match)
with a helper that dies; guard against untracked hosts/<config> since flakes
ignore untracked files; feed the sudo password more than once under setsid;
handle empty arrays under set -u; tolerate empty FSTYPE in the SD-card root
partition lookup; preflight zstdcat/dd/lsblk before the destructive dd; list
image and flash in the usage strings.
Add checks.x86_64-linux.kexec-local, a VM test driving the real script. It is
the only way to exercise kexec-local, which cannot be rehearsed on hardware. It
asserts the box left the old kernel, returned as nixos-installer, lost its old
/run, and kept its ssh host key. HOMELAB_KEXEC_TARBALL lets it reuse a prebuilt
installer instead of building ~500MB inside the guest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jupiter had a leftover docker-compose Immich on the RAID (/mnt/data/Immich,
9.9G) that survived the NixOS install. Native module now, media at
/mnt/data/AppData/immich, caddy vhost on 2283 with a 50GB body limit
(caddy's default rejects video uploads).
The package comes from nixpkgs-unstable, the module from the 26.05 pin:
26.05 ships immich 2.7.5, but that database was last written by 3.0.0 and
migrations only run forward --
corrupted migrations: previously executed migration
1776217577402-DropAuditTable is missing
Safe because the two module files are byte-identical at these revisions;
services/media/immich.nix carries the diff command to re-check on a bump.
Drop the input once the stable pin ships >= 3.0.0.
immich needs group "users" only to traverse /mnt/data/AppData (drwx--x---);
its own dir stays 0700 immich:immich. mediaLocation is outside /var/lib, so
the module's tmpfiles entry only ADJUSTS it -- add a rule that creates it.
scripts/immich-import-legacy-db does the database half: boots a copy of the
legacy PGDATA under the matching image (PG14 + vchord 0.3.0 + pgvector
0.8.1), dumps it with the local pg_dump 17, restores into a scratch DB,
fixes ownership, and only swaps after confirmation. Never touches the
original. The old cluster ran VectorChord, not pgvecto.rs, so the smart
search and face embeddings survive -- no ML re-run.
Imported: 666 assets, 25 people, 647 clip + 359 face embeddings, 2 users.
Neither is imported by any host -- both are parked while their upstreams
settle (cinephage ships only a container image, mediamanager comes from a
community flake). Grouping them apart from services/media keeps that
category to what jupiter actually runs.
Pure rename, no content change; nothing imported them, so no host config
moves with them.
Every deploy stopped at a password prompt. pass-cli is installed, so read
the passwords from the HomeLab vault instead, per command:
switch/boot/test darman@<config> -> nixos-rebuild's sudo prompt
kexec/install root@<host> -> the target's ssh password
nixos-rebuild prompts via getpass(), which reads /dev/tty and ignores a
piped stdin, so that one runs under setsid: no controlling terminal means
getpass falls back to stdin. kexec wraps the master ssh in `sshpass -e`
(scp rides the control socket) and pins password auth so a key can't fall
through into a second prompt; install uses nixos-anywhere's own
--env-password.
Missing pass-cli, a logged-out session, or an absent item all yield an
empty string and the original interactive prompt -- nothing becomes
mandatory. Passwords never reach a command line.
Three connected changes, all triggered by the same outage.
base_domain leaves mgaction.town. That zone has a wildcard A+AAAA pointing
at neptun, and DNS wildcards match multi-label names, so
jupiter.hosts.mgaction.town resolved publicly to NEPTUN and Caddy proxied
to itself -- a silent loop rather than a lookup failure. Nesting the
tailnet inside the LAN domain as orbit.sol keeps the theme and resolves
unambiguously, since tailscale matches routes by longest suffix.
override_local_dns = true with pihole as the only global nameserver, so
roaming devices get ad blocking and .sol names off-LAN. With it false,
globalResolvers land in the netmap's FallbackResolvers, which a phone
with carrier DNS never consults. No public fallback is listed on purpose:
tailscale treats the list as a set, so a second entry would let queries
slip past the filter whenever mercury is slow. The cost is that mercury
is now a single point of failure for tailnet DNS.
neptun and mercury opt out individually. mercury would otherwise resolve
through itself. neptun must not depend on a Pi behind a domestic line to
renew the certificates for the control server every other node needs --
and it is circular besides, since tailscaled has to resolve
vpn.mgaction.town to connect at all. Instead neptun runs a dnsmasq stub
forwarding just orbit.sol to MagicDNS on 100.100.100.100, which tailscaled
answers whenever it is running regardless of --accept-dns. That resolves
jupiter live, so the hardcoded /etc/hosts pin is gone.
Also sets dns.nameservers.split explicitly: nixpkgs renders its own
dns.split option one level too high, but headscale reads
dns.nameservers.split (hscontrol/types/config.go:722) and so does
headplane, whose DNS page dies on the missing key with "Cannot convert
undefined or null to object". The module's option is dead as written.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FTL could not write gravity.db, reporting "attempt to write a readonly
database". The database file was writable; the directory was not. sqlite
creates a sibling gravity.db-journal for every write transaction, so FTL
needs to CREATE files in /var/lib/pihole, and the tmpfiles rule left it
root-owned. The error names the database rather than the directory, which
sends you looking at the file and the filesystem, neither of which is at
fault.
Own the directory as 1000 instead -- the pihole user FTL drops to after
the entrypoint's root phase. Podman is rootful here with no userns
remapping, so the number is the same inside and out; on the host it
collides with darman, harmlessly.
The blocklists are now declared in this module and seeded by a oneshot,
because /var/lib/pihole is not declarative and a reflash took gravity
with it. INSERT OR IGNORE keyed on the URL is idempotent so it can run on
every boot, while the expensive rebuild only runs when gravity is empty.
Adding a list to the Nix attribute needs a manual `pihole -g` -- that is
deliberate, since the rebuild downloads every list and is slow on a Pi.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The jupiter /etc/hosts pin step is gone: headscale no longer overrides
clients' local DNS, so MagicDNS resolves that name properly and the
hardcoded tailnet address is no longer needed.
Replaces it with the headscale OIDC application, including the warning
that discovery failure at startup is fatal, and that OIDC users cannot be
reconciled with CLI-created ones -- 0.28 dropped both map_legacy_users
and node reassignment, so switching a node's owner means re-enrolling it.
Documents the Authentik admin swap: superuser is a group flag, and
akadmin must be deactivated rather than renamed or deleted, since the
bootstrap blueprint keys on the username and recreates it otherwise.
For mercury, records that a reflash wipes the gravity database along with
the adlists -- resolution keeps working with nothing blocked, which is
easy to miss -- and that .sol not resolving on mercury itself is by
design, while it resolving on tailnet members depends on
override_local_dns staying false.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two changes to the control server, plus the fallout on the hosts.
OIDC via Authentik, so `tailscale up --login-server ...` opens a browser
instead of needing a pre-auth key. This is a second Authentik application,
separate from headplane's, with headscale's own /oidc/callback redirect.
Headless hosts keep using pre-auth keys. Note that headscale runs OIDC
discovery at startup and a failure is FATAL -- pointing `issuer` at an
application that does not exist yet means the control server will not
boot, so verify the discovery document before deploying.
override_local_dns = false, because the upstream default of true replaces
resolv.conf with 100.100.100.100 on every node. That silently broke the
LAN's `.sol` names -- pihole serves those and the global nameservers
return NXDOMAIN for them -- and took ad blocking down with them. It also
made each node's entire DNS depend on tailscaled, which is what had
forced --accept-dns=false onto neptun and mercury individually; both of
those workarounds are now removed, and with MagicDNS resolving properly
again neptun no longer needs its hardcoded /etc/hosts pin for jupiter.
Also serves jellyfin and seerr from jupiter, matching the ports they
already use on its LAN vhosts, and rotates the tailnet pre-auth keys.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Everything here is something the flake cannot do for you, and all of it
was learned by hitting it: a host that builds and boots cleanly is not
necessarily a host that works.
The sudo check applies to every host and is the one that cost the most.
mutableUsers is true, so /etc/shadow is written once at user creation --
if the sops secret wasn't readable at that moment the account is locked
forever and no rebuild will fix it. That happened twice, and recovery was
netcup's rescue system for neptun and pulling the SD card for mercury.
neptun's netcup firewall is stateless and denies inbound UDP by default,
which drops every DNS and NTP reply while reporting nothing anywhere.
Also covers the Authentik/headscale/headplane bootstrap, which is a
chain of manual steps producing values the config needs.
jupiter gets the tailscaled stale-state trap: after the headscale
database is recreated the daemon still reports Running, and the
autoconnect unit exits early without sending the new pre-auth key.
mercury gets the SD-card failure mode, since silent flash corruption
surfaces as SIGILL from random binaries with a clean dmesg.
Also drops a stray code fence that had been dangling at EOF.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Authentik provider and application now exist (slug "headplane", which
is what makes the issuer .../application/o/headplane/), so the client ID
is a real value rather than a placeholder, and the client secret and
headscale API key are in sops.
The tailscale pre-auth keys for neptun and jupiter are rotated because
the tailnet was recreated from scratch: the old headscale database went
with the VPS's OS disk, so every key issued against it is meaningless to
the new control server.
Note the headscale API key defaults to a 90d expiry. When it lapses
headplane stops listing nodes with no obvious cause -- `headscale apikeys
list` shows the date.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mercury was the only host with no tailscale at all -- no module import,
no secret, no key in its sops file. It had been enrolled before the NixOS
migration and silently dropped off the tailnet when it was reflashed with
a config that omitted it.
--accept-dns=false, as on neptun and for a sharper reason: headscale
pushes override_local_dns, so accepting MagicDNS would repoint the LAN's
own DNS server at 100.100.100.100 and make house-wide name resolution
depend on tailscaled being up. This host has already deadlocked once on
boot-time DNS (see CLAUDE.md).
darman_password is also rotated: the account had "!" in /etc/shadow,
because on mercury's first boot the secret wasn't readable yet and
update-users-groups.pl falls back to a locked account. mutableUsers is
true, so no later rebuild ever revisited it and the lock was permanent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
common.nix now sets security.sudo.wheelNeedsPassword = true, but
--use-remote-sudo is deprecated and only prefixes commands with sudo --
it never prompts, so every remote rebuild failed. --ask-sudo-password is
the alias for --elevate=sudo --ask-elevate-password, which asks once and
feeds it via sudo --stdin.
This should have gone in with the wheelNeedsPassword change itself.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hardware-configuration.nix as regenerated by nixos-anywhere during the
install, replacing the placeholder. The detected initrd modules differ
from what the placeholder guessed (ata_piix, uhci_hcd), but the virtio
modules pinned in configuration.nix merge in regardless, so root mounts
either way.
darman_password is rotated because the previous hash's plaintext was not
recorded anywhere. Combined with wheelNeedsPassword = true and
PermitRootLogin = "no" that left no way to escalate on the box, and
recovery needed netcup's rescue system to edit /etc/shadow directly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
By default headscale fetches https://controlplane.tailscale.com/derpmap/default
at startup and treats failure as fatal, so it cannot boot when that URL is
unreachable. A self-hosted control plane that will not start without
Tailscale's infrastructure rather misses the point of self-hosting -- and
it crash-looped for exactly that reason while neptun had no DNS.
Enable the embedded DERP server on region 999 and drop the upstream map.
The relay rides Caddy on :443, which is why that vhost already sets
flush_interval -1; only STUN needs a port of its own.
Verified against headscale 0.28.0 before committing: it starts clean with
urls = [], registers "DERP region: {RegionID:999 ...}" pointing at
vpn.mgaction.town with DERPPort 443, and brings up STUN.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Probing the live Debian VPS turned up three mismatches between what it
serves and what this config declares:
- git.mgaction.town had no vhost at all. Gitea's web UI and HTTPS clones
are public today; only its SSH side (the :2222 socat forward) had been
ported, so a deploy would have taken the web side offline.
- Audiobookshelf is served as abs.mgaction.town, not the longer
audiobookshelf.mgaction.town this config used. The mobile app is
configured with the short name.
- The apex returns 200 from Caddy. Left unserved deliberately, so it now
gets Caddy's default 404; noted in a comment so it doesn't look like an
oversight next time.
Gitea's ROOT_URL was http:// while Caddy terminates TLS for that name.
Gitea builds absolute URLs from it, so clone buttons, redirects and
webhooks were handing out downgraded links.
Also record that defaultGateway6 is confirmed rather than assumed --
`ip -6 route show default` on the VPS gives "default via fe80::1 dev
eth0 metric 1024 onlink".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nixpkgs only carries Zitadel 2.71, which predates the login-v2 split and
cannot take a v3/v4 database (its migrations are forward-only), so the
instance running on the old Debian VPS could never have moved onto it.
authentik-nix ships 2026.5.4 and tracks upstream closely.
The authentik-nix input deliberately does not follow our nixpkgs, per
upstream's warning that overriding it breaks their pinned python
dependency set. That costs a second nixpkgs in the lock, so add
nix-community's Cachix to common.nix -- without it the closure is ~400
local derivations (npm, rust, python). The laptop that runs
scripts/deploy needs the same two lines in /etc/nix/nix.custom.conf.
Authentik's own module creates the database and orders its units against
postgresql.target, and recent versions need no redis, so the wiring is
just the module plus a secret. Pin postgresql explicitly so that editing
system.stateVersion can never silently demand a pg_upgrade of the
identity store.
Secret ownership is not uniform and the difference matters: authentik
and caddy take a systemd EnvironmentFile, which PID 1 reads as root
before dropping privileges, so root:root 0400 is correct. Headplane
opens its secret paths itself while already running as the headscale
user, so those three need an explicit owner or they fail to start.
Also on neptun:
- Pass Caddy's ACME account email through the same EnvironmentFile
mechanism and reference it with the Caddyfile {$VAR} placeholder.
services.caddy.email would render the address into the world-readable
store.
- Stop accepting MagicDNS from our own control server. headscale pushes
override_local_dns, so joining the tailnet would point neptun's
resolv.conf at a MagicDNS served by the tailscaled neptun itself hosts
-- a tailscaled failure would then also take out DNS, ACME renewal and
finally the certs for the control server every other node needs in
order to recover.
- Give headplane a writable DNS extra-records file. Its view of
headscale's config stays read-only, which is the right outcome for a
declarative box; records are data rather than config.
- Require a password for sudo. Deploys become interactive, but darman's
key is otherwise the only thing between the public internet and root.
- Enable zram (8 GB, and disko leaves no room for a swap device), and let
tailscaled-autoconnect retry instead of failing permanently when the
control server isn't up yet on a first boot.
networking.hosts still carries a PLACEHOLDER address for jupiter --
replace it from `headscale nodes list` once jupiter first enrols.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
darman
