Follow-up to #6 (Mnemosyne) — voice for Hermes on mars. Reuses the provisioning pattern that review shaped, so this is its own PR rather than a rider on the merged branch (orig 76ebc8c, cherry-picked clean onto master).
Model: `KittenML/kitten-tts-mini-0.8` (80M params, ~74 MB onnx) — the bigger one. Voice: Luna (expr-voice-3-f via the model's config.json aliases).
CPU-only onnxruntime; no GPU on mars. Validated live on a weaker dev box (i5-4460, 4 cores): load 1.9 s, generation ~1× realtime (8.9 s speech in 8.9 s), HF_HUB_OFFLINE=1.
Upstream's misaki[en]+spacy declaration deliberately not honored — pulls torch→CUDA (~5.6 GB verified). Runtime phonemizes via phonemizer-fork + espeakng-loader exclusively; dead `from misaki import en, espeak` import satisfied by a .pth-registered stub that fails loudly if misaki is ever touched.
HF model cache pre-seeded from hash-pinned fetchurl store paths → zero boot-time network, no drift.
Review-shaped invariants carried over: `uv venv --clear` (no re-provision wedge), root never executes python from the venv (chowned to container uid before anything imports), stamp checked against venv and model files, trust note documented (venv inside HERMES_WRITE_SAFE_ROOT is a deliberate call — it's luna's voice; webhook filters remain the only write-protected-load-bearing items).
Config wiring (`tts.provider: kittentts`, `voice: Luna`) happens container-side post-merge — not in this commit, so the module stays purely provisioning.
Caveat unchanged: no Nix eval on my side — brace balance is my only static check beyond live CPU validation.
Follow-up to #6 (Mnemosyne) — voice for Hermes on mars. Reuses the provisioning pattern that review shaped, so this is its own PR rather than a rider on the merged branch (orig 76ebc8c, cherry-picked clean onto master).
- Model: \`KittenML/kitten-tts-mini-0.8\` (80M params, ~74 MB onnx) — the bigger one. Voice: **Luna** (expr-voice-3-f via the model's config.json aliases).
- CPU-only onnxruntime; no GPU on mars. Validated live on a weaker dev box (i5-4460, 4 cores): load 1.9 s, generation ~1× realtime (8.9 s speech in 8.9 s), HF_HUB_OFFLINE=1.
- Upstream's misaki[en]+spacy declaration deliberately not honored — pulls torch→CUDA (~5.6 GB verified). Runtime phonemizes via phonemizer-fork + espeakng-loader exclusively; dead \`from misaki import en, espeak\` import satisfied by a .pth-registered stub that fails loudly if misaki is ever touched.
- HF model cache pre-seeded from hash-pinned fetchurl store paths → zero boot-time network, no drift.
- Review-shaped invariants carried over: \`uv venv --clear\` (no re-provision wedge), root never executes python from the venv (chowned to container uid before anything imports), stamp checked against venv **and** model files, trust note documented (venv inside HERMES_WRITE_SAFE_ROOT is a deliberate call — it's luna's voice; webhook filters remain the only write-protected-load-bearing items).
Config wiring (\`tts.provider: kittentts\`, \`voice: Luna\`) happens container-side post-merge — not in this commit, so the module stays purely provisioning.
Caveat unchanged: no Nix eval on my side — brace balance is my only static check beyond live CPU validation.
tts.provider: kittentts, voice Luna, mini-0.8 model (80M params) per
darman. CPU-only onnxruntime inference — no GPU on mars, validated ~1x
realtime on a weaker dev box. Upstream's misaki[en]+spacy declaration is
deliberately not honored (pulls torch→CUDA, 5.6 GB); a .pth-registered
stub satisfies the dead misaki import, failing loudly if it is ever
actually used. HF model cache pre-seeded from hash-pinned store paths so
HF_HUB_OFFLINE=1 runs with zero boot-time network. Provisioner shaped by
the Mnemosyne review: uv venv --clear, root never executes python from
the venv, ownership to the container uid before anything imports, stamp
checked against venv+model files.
Requesting changes: two ways for the container to get root on mars, and the model pinning doesn't take effect.
Security
1. Symlink takeover through the provision stamp (hosts/mars/hermes-agent.nix:562) kittentts-provision.stamp lives in hermesHome, whose top level is owned by uid 986. The container can:
replace the stamp with a symlink to, say, /etc/shadow
delete $venv/bin/python to force a re-provision
On the next boot, root writes the hash through the link, which truncates the target. Then chown (no -h) on L563 hands the target to uid 986. The same swap on $snap (L551) redirects root's install to any path.
2. Root runs Python from a cache the container can write (:534) UV_CACHE_DIR=${hermesHome}/kittentts-uv-cache is writable by uid 986. A planted wheel in that cache gets installed as-is. Root then starts $venv/bin/python twice: uv probes the interpreter, and L545 calls sysconfig. Any .pth file shipped in that wheel runs as root. That breaks the "root runs no python from this venv" rule stated in the PR.
Suggested fix for both: keep the stamp, the HF snapshot staging and the uv cache in a root-owned directory outside hermesHome (e.g. under /var/lib/hermes-kittentts). Build the venv there, and only copy or bind the finished result into the container's tree. Never follow a path the container can write while running as root.
Correctness
3. Seeded HF cache isn't found (:519)
Files go into snapshots/kitten, but refs/main is never written; only an empty refs/ directory is created. hf_hub_download with the default revision="main" resolves the snapshot through refs/main. With HF_HUB_OFFLINE=1 this raises LocalEntryNotFoundError. Without it, the model is downloaded again.
Fix: use the real commit hash as the snapshot directory name, and write that hash to refs/main.
4. Runtime never uses the seeded cache (:280)
The container env doesn't set HF_HUB_CACHE (or HF_HOME) to /opt/data/kittentts-hf, and doesn't set HF_HUB_OFFLINE=1. HF falls back to ~/.cache/huggingface, so the no-network-at-boot guarantee doesn't hold.
5. Stamp only tracks requirements.txt (:524)
Changing the wheel URL, kitten-misaki-stub.py, or a model URL or hash creates a new store path. The early exit still passes, though, so mars keeps the old install. Key the stamp on the provisioner's own store path, or on a hash of every input.
6. Missing ordering on hermes-agent-prepare-dirs (:503) (minor)
The Mnemosyne unit orders itself after hermes-agent-prepare-dirs; this one doesn't. On a fresh state dir it can create hermesHome owned by root before the ownership layout is set up. Add after/requires like Mnemosyne has.
Requesting changes: two ways for the container to get root on mars, and the model pinning doesn't take effect.
### Security
**1. Symlink takeover through the provision stamp (`hosts/mars/hermes-agent.nix:562`)**
`kittentts-provision.stamp` lives in `hermesHome`, whose top level is owned by uid 986. The container can:
- replace the stamp with a symlink to, say, `/etc/shadow`
- delete `$venv/bin/python` to force a re-provision
On the next boot, root writes the hash through the link, which truncates the target. Then `chown` (no `-h`) on L563 hands the target to uid 986. The same swap on `$snap` (L551) redirects root's `install` to any path.
**2. Root runs Python from a cache the container can write (`:534`)**
`UV_CACHE_DIR=${hermesHome}/kittentts-uv-cache` is writable by uid 986. A planted wheel in that cache gets installed as-is. Root then starts `$venv/bin/python` twice: uv probes the interpreter, and L545 calls `sysconfig`. Any `.pth` file shipped in that wheel runs as root. That breaks the "root runs no python from this venv" rule stated in the PR.
Suggested fix for both: keep the stamp, the HF snapshot staging and the uv cache in a root-owned directory outside `hermesHome` (e.g. under `/var/lib/hermes-kittentts`). Build the venv there, and only copy or bind the finished result into the container's tree. Never follow a path the container can write while running as root.
### Correctness
**3. Seeded HF cache isn't found (`:519`)**
Files go into `snapshots/kitten`, but `refs/main` is never written; only an empty `refs/` directory is created. `hf_hub_download` with the default `revision="main"` resolves the snapshot through `refs/main`. With `HF_HUB_OFFLINE=1` this raises `LocalEntryNotFoundError`. Without it, the model is downloaded again.
Fix: use the real commit hash as the snapshot directory name, and write that hash to `refs/main`.
**4. Runtime never uses the seeded cache (`:280`)**
The container env doesn't set `HF_HUB_CACHE` (or `HF_HOME`) to `/opt/data/kittentts-hf`, and doesn't set `HF_HUB_OFFLINE=1`. HF falls back to `~/.cache/huggingface`, so the no-network-at-boot guarantee doesn't hold.
**5. Stamp only tracks `requirements.txt` (`:524`)**
Changing the wheel URL, `kitten-misaki-stub.py`, or a model URL or hash creates a new store path. The early exit still passes, though, so mars keeps the old install. Key the stamp on the provisioner's own store path, or on a hash of every input.
**6. Missing ordering on `hermes-agent-prepare-dirs` (`:503`)** (minor)
The Mnemosyne unit orders itself after `hermes-agent-prepare-dirs`; this one doesn't. On a fresh state dir it can create `hermesHome` owned by root before the ownership layout is set up. Add `after`/`requires` like Mnemosyne has.
Review round 1 fixes:
1+2 (security): build+stamp+uv-cache moved out of hermesHome into a
0700 root-owned staging dir (/var/lib/hermes-kittentts). Root executes
python only from the staging venv after an offline import check;
finished artifacts are copied into hermesHome and the stamp compares
delivered copy vs staging byte-for-byte. No root code path follows
anything the container can write, so no stamp symlink-takeover and no
planted-wheel-in-cache execution.
3: HF snapshot dir is the real commit sha with refs/main written, so
hf_hub_download resolves the seeded cache offline (was 'kitten', never
found).
4: container env pins HF_HOME=/opt/data/kittentts-hf + HF_HUB_OFFLINE=1.
5: stamp keyed on the full input set (requirements+wheel+stub+models),
not requirements alone; deletion of a delivered file drops out of the
fast path and re-provisions cheaply.
6: orders after/requires hermes-agent-prepare-dirs.
All six addressed in b2982c4. (Gut gemacht mit dem Takeover-Aufdeckungen — der Stamp-Symlink war ein echter Blöder.)
#1+#2 (security) — staging redesign. Venv build, stamp, uv cache and model staging now live in a root-owned /var/lib/hermes-kittentts (0700), completely outside hermesHome. Root runs python ONLY from the staging venv — and only after an offline from kittentts import KittenTTS sanity check fails the whole provision (fail-closed). Finished artifacts are cp'd into hermesHome (.tmp + atomic mv -Tf swap of the delivered venv) and the stamp compares delivered copy vs staging byte-for-byte. No root code path follows anything the container can write: symlink takeover of the stamp impossible (stamp is root-owned outside the container's tree), planted uv-cache wheel impossible (cache is under staging, root-only).
#3: snapshot dir is now the real commit SHA c0272566... with refs/main written to it, so hf_hub_download(revision="main") resolves offline.
#4: container environment now sets HF_HOME=/opt/data/kittentts-hf and HF_HUB_OFFLINE=1 — the sealed cache is actually on the runtime path, and network is refused rather than merely absent.
#5: stamp keyed on the full input set — cat requirements wheel stub onnx voices config | sha256sum. A pin change on ANY of those store paths yields a new hash, invalidating the stamp.
One consequence worth flagging: the delivered venv inside hermesHome is now self-healing rather than hardened — luna can delete or modify her copy, and the next boot simply re-delivers from staging (byte-compare on the three model files + venv exec check misses → re-copy). She cannot get root via any of it, and the root-owned staging dir never runs from her tree. That's the tradeoff documented in the unit header comment.
Same standing caveat: no nix eval on my side; static balance + live CPU equivalent was re-verified in the validation container earlier today.
All six addressed in b2982c4. (Gut gemacht mit dem Takeover-Aufdeckungen — der Stamp-Symlink war ein echter Blöder.)
**#1+#2 (security) — staging redesign.** Venv build, stamp, uv cache and model staging now live in a root-owned `/var/lib/hermes-kittentts` (0700), completely outside `hermesHome`. Root runs python ONLY from the staging venv — and only after an offline `from kittentts import KittenTTS` sanity check fails the whole provision (fail-closed). Finished artifacts are cp'd into hermesHome (`.tmp` + atomic `mv -Tf` swap of the delivered venv) and the stamp compares delivered copy vs staging **byte-for-byte**. No root code path follows anything the container can write: symlink takeover of the stamp impossible (stamp is root-owned outside the container's tree), planted uv-cache wheel impossible (cache is under staging, root-only).
**#3:** snapshot dir is now the real commit SHA `c0272566...` with `refs/main` written to it, so `hf_hub_download(revision="main")` resolves offline.
**#4:** container environment now sets `HF_HOME=/opt/data/kittentts-hf` and `HF_HUB_OFFLINE=1` — the sealed cache is actually on the runtime path, and network is refused rather than merely absent.
**#5:** stamp keyed on the full input set — `cat requirements wheel stub onnx voices config | sha256sum`. A pin change on ANY of those store paths yields a new hash, invalidating the stamp.
**#6:** `after`/`requires hermes-agent-prepare-dirs` added, mirroring mnemosyne.
One consequence worth flagging: the delivered venv inside hermesHome is now **self-healing rather than hardened** — luna can delete or modify her copy, and the next boot simply re-delivers from staging (byte-compare on the three model files + venv exec check misses → re-copy). She cannot get root via any of it, and the root-owned staging dir never runs from her tree. That's the tradeoff documented in the unit header comment.
Same standing caveat: no nix eval on my side; static balance + live CPU equivalent was re-verified in the validation container earlier today.
Thanks for the fixes. #2–#6 are addressed: the uv cache and root's Python now live under a root-owned 0700 dir, refs/main → snapshots/<sha> is correct (sha matches the HF API), HF_HOME + HF_HUB_OFFLINE are set, the stamp covers all inputs, and the unit is ordered after prepare-dirs. Still requesting changes, though: the new code has three bugs that break the unit after its first boot, and #1 isn't fully closed.
Blocking
1. $stageVenv is used before it's assigned (hosts/mars/hermes-agent.nix:566)
The early-exit check reads $stageVenv/bin/python, but stageVenv= is only set on L584, and the script runs under set -u. On the first boot there's no stamp, so && short-circuits and provisioning works. On every later boot the stamp matches, bash aborts with stageVenv: unbound variable, and the unit fails. The self-heal path is unreachable.
Fix: move stageVenv=$stageDir/venv up next to the other path variables.
2. mv -Tf "$venv.tmp" "$venv" fails once a venv exists (:624)
rename(2) can't replace a directory that isn't empty (mv: cannot overwrite '…/venv': Directory not empty, exit 1). The first install works. Every re-provision (changed wheel, requirements or model, or a damaged delivery) aborts here. The Mnemosyne unit gets away with the same mv -Tf because $pluginDir is a symlink, not a directory.
3. network-online.target was dropped from after (:528) wants still pulls it in, but without ordering, uv pip install can run before the network is up on any boot that has to provision. Mnemosyne doesn't need this because it builds from a store path, but this unit downloads at boot. Use:
4. Root still writes through paths uid 986 controls (#1, not fully closed)
The stamp is safe now, but the delivery step moved the same problem to other paths:
printf '%s' "$modelSha" > "$hubDir/refs/main" (L627) follows symlinks. Link refs/main → /etc/shadow, trigger a re-provision, and root overwrites shadow with the sha.
mkdir -p / install into $hubDir / $snap resolve through intermediate directories under kittentts-hf/ that uid 986 owns and can replace with symlinks.
rm -rf "$venv.tmp" → cp -a (L622–623) is a race: recreate kittentts-venv.tmp as a symlink in between, and root copies into its target.
Suggested fix: root should never write into hermesHome at all.
Stage the full delivered tree, including kittentts-hf/hub/… with refs/main, under a root-owned 0755 dir. uid 986 can read it but not write it.
Deliver as the container user:
setpriv --reuid=${hermesUid} --regid=${hermesGid} --clear-groups sh -c '
rm -rf "$1" "$2" && cp -a "$3" "$1" && cp -a "$4" "$2"
' _ "$venv""${hermesHome}/kittentts-hf""$stageVenv""$stageDir/kittentts-hf"
This also fixes#2 above and makes the chown -R unnecessary.
Minor
The sanity check doesn't test the offline cache (:616). It only runs from kittentts import KittenTTS and never goes through hf_hub_download. Point HF_HOME at the staged hub tree and actually construct KittenTTS("KittenML/kitten-tts-mini-0.8"), so a broken cache layout fails here instead of in the gateway.
The model fetchurls still use resolve/main/ (L95–103). Use resolve/c02725660cea441db4c383af69f1f26f5cd00947/, so the download and refs/main are tied to the same commit, and an upstream push doesn't turn into a hash mismatch on the next uncached build.
The "cheap copy, not a rebuild" comment doesn't match the code. Any miss on the fast path goes through rm -rf "$stageVenv" (L591) and rebuilds the whole venv. Either skip the rebuild when the stamp and staging venv are valid and only the delivered copy is damaged, or fix the comment.
Thanks for the fixes. #2–#6 are addressed: the uv cache and root's Python now live under a root-owned 0700 dir, `refs/main` → `snapshots/<sha>` is correct (sha matches the HF API), `HF_HOME` + `HF_HUB_OFFLINE` are set, the stamp covers all inputs, and the unit is ordered after `prepare-dirs`. Still requesting changes, though: the new code has three bugs that break the unit after its first boot, and #1 isn't fully closed.
### Blocking
**1. `$stageVenv` is used before it's assigned (`hosts/mars/hermes-agent.nix:566`)**
The early-exit check reads `$stageVenv/bin/python`, but `stageVenv=` is only set on L584, and the script runs under `set -u`. On the first boot there's no stamp, so `&&` short-circuits and provisioning works. On every later boot the stamp matches, bash aborts with `stageVenv: unbound variable`, and the unit fails. The self-heal path is unreachable.
Fix: move `stageVenv=$stageDir/venv` up next to the other path variables.
**2. `mv -Tf "$venv.tmp" "$venv"` fails once a venv exists (`:624`)**
rename(2) can't replace a directory that isn't empty (`mv: cannot overwrite '…/venv': Directory not empty`, exit 1). The first install works. Every re-provision (changed wheel, requirements or model, or a damaged delivery) aborts here. The Mnemosyne unit gets away with the same `mv -Tf` because `$pluginDir` is a symlink, not a directory.
**3. `network-online.target` was dropped from `after` (`:528`)**
`wants` still pulls it in, but without ordering, `uv pip install` can run before the network is up on any boot that has to provision. Mnemosyne doesn't need this because it builds from a store path, but this unit downloads at boot. Use:
```nix
after = [ "network-online.target" "hermes-agent-prepare-dirs.service" ];
```
**4. Root still writes through paths uid 986 controls (#1, not fully closed)**
The stamp is safe now, but the delivery step moved the same problem to other paths:
- `printf '%s' "$modelSha" > "$hubDir/refs/main"` (L627) follows symlinks. Link `refs/main` → `/etc/shadow`, trigger a re-provision, and root overwrites shadow with the sha.
- `mkdir -p` / `install` into `$hubDir` / `$snap` resolve through intermediate directories under `kittentts-hf/` that uid 986 owns and can replace with symlinks.
- `rm -rf "$venv.tmp"` → `cp -a` (L622–623) is a race: recreate `kittentts-venv.tmp` as a symlink in between, and root copies into its target.
Suggested fix: root should never write into `hermesHome` at all.
- Stage the full delivered tree, including `kittentts-hf/hub/…` with `refs/main`, under a root-owned 0755 dir. uid 986 can read it but not write it.
- Deliver as the container user:
```sh
setpriv --reuid=${hermesUid} --regid=${hermesGid} --clear-groups sh -c '
rm -rf "$1" "$2" && cp -a "$3" "$1" && cp -a "$4" "$2"
' _ "$venv" "${hermesHome}/kittentts-hf" "$stageVenv" "$stageDir/kittentts-hf"
```
This also fixes #2 above and makes the `chown -R` unnecessary.
### Minor
- **The sanity check doesn't test the offline cache (`:616`).** It only runs `from kittentts import KittenTTS` and never goes through `hf_hub_download`. Point `HF_HOME` at the staged hub tree and actually construct `KittenTTS("KittenML/kitten-tts-mini-0.8")`, so a broken cache layout fails here instead of in the gateway.
- **The model `fetchurl`s still use `resolve/main/` (L95–103).** Use `resolve/c02725660cea441db4c383af69f1f26f5cd00947/`, so the download and `refs/main` are tied to the same commit, and an upstream push doesn't turn into a hash mismatch on the next uncached build.
- **The "cheap copy, not a rebuild" comment doesn't match the code.** Any miss on the fast path goes through `rm -rf "$stageVenv"` (L591) and rebuilds the whole venv. Either skip the rebuild when the stamp and staging venv are valid and only the delivered copy is damaged, or fix the comment.
1: stageVenv declared with the other path vars before the early-exit
reads it (set -u no longer kills the self-heal path on later boots).
2: delivery is rm+cp as the container uid via setpriv — no mv -Tf
rename-replace on a non-empty directory (that failed every
re-provision); chown -R step gone with it.
3: network-online.target back in after= — this unit CAN download at
boot, so uv must not run pre-network.
4: root never writes into hermesHome at all. refs/main, snapshots and
venv staging happen under root-owned staging; delivery drops to uid 986
via setpriv, rm -rf + cp -a from the staging sources. Symlink-takeover
of refs/install targets and the rm/cp race on the delivered tree are
structurally gone.
Minor: sanity check now constructs KittenTTS against the staged hub
tree, not just imports; fetchurls pinned to the commit sha matching
refs/main; fast-path comment matches behavior (staging venv skips
rebuild on damaged delivery).
#1 ($stageVenv used before assignment): all staging paths (stageVenv/stageHub/stageSnap) now block at the top with the other path vars, before the early-exit reads them. set -u is safe on every boot.
#2 (mv -Tf on non-empty dir): gone — delivery is now rm -rf + cp -a executed AS the container uid (setpriv --reuid=986 --regid=983 --clear-groups), which also eats the chown -R step.
#3:network-online.target restored to after alongside prepare-dirs — you're right this unit can download at boot unlike mnemosyne.
#4 (root still writes into hermesHome) — the big one: root now writes NOTHING in hermesHome. The full delivered tree — venv AND the complete hub layout including refs/main — is staged root-owned, then delivered via:
setpriv --reuid=986 --regid=983 --clear-groups \
env HOME=... sh -c 'rm -rf "$1" "$2"; cp -a "$3" "$1"; cp -a "$4" "$2"' _ \
DEST_VENV DEST_HF STAGE_VENV STAGE_HUB
The rm/cp race you called out is structurally gone (root's cp -a $stageVenv reads a root-owned source; the write happens as 986 into 986-owned land). refs/main is staged root-side and arrives by cp, never via a root printf > that could follow a link.
Minors:
Sanity check now does KittenTTS("KittenML/kitten-tts-mini-0.8") with HF_HOME=$stageHub so it walks the full hf_hub_download path against the staged tree.
fetchurls pinned to resolve/c0272566.../ — the download and refs/main can't drift from the same commit.
Fast path updated to match behavior: staging venv is only rebuilt when staging itself is damaged; a damaged delivery is just a copy.
pkgs.setpriv added to the unit path.
Also honest note: the mkdir -p "$stageDir" "$stageSnap" ordering in the previous rev ran before $stageVenv was defined, so your #1 also caught a latent fast-path hole where the delivered-copy cmp referenced $snap from the top-level vars — consistent now.
Round 2 addressed in bef5215.
**#1 ($stageVenv used before assignment):** all staging paths (`stageVenv`/`stageHub`/`stageSnap`) now block at the top with the other path vars, before the early-exit reads them. set -u is safe on every boot.
**#2 (mv -Tf on non-empty dir):** gone — delivery is now `rm -rf` + `cp -a` executed AS the container uid (setpriv `--reuid=986 --regid=983 --clear-groups`), which also eats the chown -R step.
**#3:** `network-online.target` restored to `after` alongside `prepare-dirs` — you're right this unit can download at boot unlike mnemosyne.
**#4 (root still writes into hermesHome) — the big one:** root now writes NOTHING in hermesHome. The full delivered tree — venv AND the complete hub layout including `refs/main` — is staged root-owned, then delivered via:
```sh
setpriv --reuid=986 --regid=983 --clear-groups \
env HOME=... sh -c 'rm -rf "$1" "$2"; cp -a "$3" "$1"; cp -a "$4" "$2"' _ \
DEST_VENV DEST_HF STAGE_VENV STAGE_HUB
```
The rm/cp race you called out is structurally gone (root's `cp -a $stageVenv` reads a root-owned source; the write happens as 986 into 986-owned land). `refs/main` is staged root-side and arrives by cp, never via a root `printf >` that could follow a link.
**Minors:**
- Sanity check now does `KittenTTS("KittenML/kitten-tts-mini-0.8")` with `HF_HOME=$stageHub` so it walks the full `hf_hub_download` path against the staged tree.
- fetchurls pinned to `resolve/c0272566.../` — the download and `refs/main` can't drift from the same commit.
- Fast path updated to match behavior: staging venv is only rebuilt when staging itself is damaged; a damaged *delivery* is just a copy.
- `pkgs.setpriv` added to the unit path.
Also honest note: the `mkdir -p "$stageDir" "$stageSnap"` ordering in the previous rev ran before `$stageVenv` was defined, so your #1 also caught a latent fast-path hole where the delivered-copy cmp referenced `$snap` from the top-level vars — consistent now.
- pkgs.setpriv does not exist (mars failed to eval); setpriv is in util-linux.
- `sh` is not on a unit's PATH; call ${pkgs.runtimeShell} directly.
- Drop `env HOME="$hermesHome"`: unset shell var under `set -u`, and unneeded.
- Staging dir 0700 -> 0755: the uid-986 delivery step could not read it.
- Rebuild the staging venv whenever the stamp misses, else an input bump
delivered the old venv under the new stamp.
- Add diffutils: `cmp` in the fast path was missing, so every boot re-delivered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
uv reads the version from the wheel filename, and the fetchurl store path's
hash prefix made it reject the file ("invalid version"). Expose it via a
linkFarm under kittentts-0.8.1-py3-none-any.whl.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The venv's python is Nix python313 (also inside the container, via the
/nix/store mount), whose loader never searches /usr/lib, so PyPI manylinux
wheels failed with "libstdc++.so.6: cannot open shared object file".
auto-patchelf the staged venv against gcc-lib + zlib; RPATHs are store
paths, so the delivered copy works in the container too. The lib paths join
the input hash so a toolchain bump rebuilds.
Verified locally with mars's exact uv/python313: construct offline from the
staged hub layout and synthesize with voice "Luna".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Follow-up to #6 (Mnemosyne) — voice for Hermes on mars. Reuses the provisioning pattern that review shaped, so this is its own PR rather than a rider on the merged branch (orig
76ebc8c, cherry-picked clean onto master).Config wiring (`tts.provider: kittentts`, `voice: Luna`) happens container-side post-merge — not in this commit, so the module stays purely provisioning.
Caveat unchanged: no Nix eval on my side — brace balance is my only static check beyond live CPU validation.
Requesting changes: two ways for the container to get root on mars, and the model pinning doesn't take effect.
Security
1. Symlink takeover through the provision stamp (
hosts/mars/hermes-agent.nix:562)kittentts-provision.stamplives inhermesHome, whose top level is owned by uid 986. The container can:/etc/shadow$venv/bin/pythonto force a re-provisionOn the next boot, root writes the hash through the link, which truncates the target. Then
chown(no-h) on L563 hands the target to uid 986. The same swap on$snap(L551) redirects root'sinstallto any path.2. Root runs Python from a cache the container can write (
:534)UV_CACHE_DIR=${hermesHome}/kittentts-uv-cacheis writable by uid 986. A planted wheel in that cache gets installed as-is. Root then starts$venv/bin/pythontwice: uv probes the interpreter, and L545 callssysconfig. Any.pthfile shipped in that wheel runs as root. That breaks the "root runs no python from this venv" rule stated in the PR.Suggested fix for both: keep the stamp, the HF snapshot staging and the uv cache in a root-owned directory outside
hermesHome(e.g. under/var/lib/hermes-kittentts). Build the venv there, and only copy or bind the finished result into the container's tree. Never follow a path the container can write while running as root.Correctness
3. Seeded HF cache isn't found (
:519)Files go into
snapshots/kitten, butrefs/mainis never written; only an emptyrefs/directory is created.hf_hub_downloadwith the defaultrevision="main"resolves the snapshot throughrefs/main. WithHF_HUB_OFFLINE=1this raisesLocalEntryNotFoundError. Without it, the model is downloaded again.Fix: use the real commit hash as the snapshot directory name, and write that hash to
refs/main.4. Runtime never uses the seeded cache (
:280)The container env doesn't set
HF_HUB_CACHE(orHF_HOME) to/opt/data/kittentts-hf, and doesn't setHF_HUB_OFFLINE=1. HF falls back to~/.cache/huggingface, so the no-network-at-boot guarantee doesn't hold.5. Stamp only tracks
requirements.txt(:524)Changing the wheel URL,
kitten-misaki-stub.py, or a model URL or hash creates a new store path. The early exit still passes, though, so mars keeps the old install. Key the stamp on the provisioner's own store path, or on a hash of every input.6. Missing ordering on
hermes-agent-prepare-dirs(:503) (minor)The Mnemosyne unit orders itself after
hermes-agent-prepare-dirs; this one doesn't. On a fresh state dir it can createhermesHomeowned by root before the ownership layout is set up. Addafter/requireslike Mnemosyne has.All six addressed in
b2982c4. (Gut gemacht mit dem Takeover-Aufdeckungen — der Stamp-Symlink war ein echter Blöder.)#1+#2 (security) — staging redesign. Venv build, stamp, uv cache and model staging now live in a root-owned
/var/lib/hermes-kittentts(0700), completely outsidehermesHome. Root runs python ONLY from the staging venv — and only after an offlinefrom kittentts import KittenTTSsanity check fails the whole provision (fail-closed). Finished artifacts are cp'd into hermesHome (.tmp+ atomicmv -Tfswap of the delivered venv) and the stamp compares delivered copy vs staging byte-for-byte. No root code path follows anything the container can write: symlink takeover of the stamp impossible (stamp is root-owned outside the container's tree), planted uv-cache wheel impossible (cache is under staging, root-only).#3: snapshot dir is now the real commit SHA
c0272566...withrefs/mainwritten to it, sohf_hub_download(revision="main")resolves offline.#4: container environment now sets
HF_HOME=/opt/data/kittentts-hfandHF_HUB_OFFLINE=1— the sealed cache is actually on the runtime path, and network is refused rather than merely absent.#5: stamp keyed on the full input set —
cat requirements wheel stub onnx voices config | sha256sum. A pin change on ANY of those store paths yields a new hash, invalidating the stamp.#6:
after/requires hermes-agent-prepare-dirsadded, mirroring mnemosyne.One consequence worth flagging: the delivered venv inside hermesHome is now self-healing rather than hardened — luna can delete or modify her copy, and the next boot simply re-delivers from staging (byte-compare on the three model files + venv exec check misses → re-copy). She cannot get root via any of it, and the root-owned staging dir never runs from her tree. That's the tradeoff documented in the unit header comment.
Same standing caveat: no nix eval on my side; static balance + live CPU equivalent was re-verified in the validation container earlier today.
Thanks for the fixes. #2–#6 are addressed: the uv cache and root's Python now live under a root-owned 0700 dir,
refs/main→snapshots/<sha>is correct (sha matches the HF API),HF_HOME+HF_HUB_OFFLINEare set, the stamp covers all inputs, and the unit is ordered afterprepare-dirs. Still requesting changes, though: the new code has three bugs that break the unit after its first boot, and #1 isn't fully closed.Blocking
1.
$stageVenvis used before it's assigned (hosts/mars/hermes-agent.nix:566)The early-exit check reads
$stageVenv/bin/python, butstageVenv=is only set on L584, and the script runs underset -u. On the first boot there's no stamp, so&&short-circuits and provisioning works. On every later boot the stamp matches, bash aborts withstageVenv: unbound variable, and the unit fails. The self-heal path is unreachable.Fix: move
stageVenv=$stageDir/venvup next to the other path variables.2.
mv -Tf "$venv.tmp" "$venv"fails once a venv exists (:624)rename(2) can't replace a directory that isn't empty (
mv: cannot overwrite '…/venv': Directory not empty, exit 1). The first install works. Every re-provision (changed wheel, requirements or model, or a damaged delivery) aborts here. The Mnemosyne unit gets away with the samemv -Tfbecause$pluginDiris a symlink, not a directory.3.
network-online.targetwas dropped fromafter(:528)wantsstill pulls it in, but without ordering,uv pip installcan run before the network is up on any boot that has to provision. Mnemosyne doesn't need this because it builds from a store path, but this unit downloads at boot. Use:4. Root still writes through paths uid 986 controls (#1, not fully closed)
The stamp is safe now, but the delivery step moved the same problem to other paths:
printf '%s' "$modelSha" > "$hubDir/refs/main"(L627) follows symlinks. Linkrefs/main→/etc/shadow, trigger a re-provision, and root overwrites shadow with the sha.mkdir -p/installinto$hubDir/$snapresolve through intermediate directories underkittentts-hf/that uid 986 owns and can replace with symlinks.rm -rf "$venv.tmp"→cp -a(L622–623) is a race: recreatekittentts-venv.tmpas a symlink in between, and root copies into its target.Suggested fix: root should never write into
hermesHomeat all.kittentts-hf/hub/…withrefs/main, under a root-owned 0755 dir. uid 986 can read it but not write it.chown -Runnecessary.Minor
:616). It only runsfrom kittentts import KittenTTSand never goes throughhf_hub_download. PointHF_HOMEat the staged hub tree and actually constructKittenTTS("KittenML/kitten-tts-mini-0.8"), so a broken cache layout fails here instead of in the gateway.fetchurls still useresolve/main/(L95–103). Useresolve/c02725660cea441db4c383af69f1f26f5cd00947/, so the download andrefs/mainare tied to the same commit, and an upstream push doesn't turn into a hash mismatch on the next uncached build.rm -rf "$stageVenv"(L591) and rebuilds the whole venv. Either skip the rebuild when the stamp and staging venv are valid and only the delivered copy is damaged, or fix the comment.Round 2 addressed in
bef5215.#1 ($stageVenv used before assignment): all staging paths (
stageVenv/stageHub/stageSnap) now block at the top with the other path vars, before the early-exit reads them. set -u is safe on every boot.#2 (mv -Tf on non-empty dir): gone — delivery is now
rm -rf+cp -aexecuted AS the container uid (setpriv--reuid=986 --regid=983 --clear-groups), which also eats the chown -R step.#3:
network-online.targetrestored toafteralongsideprepare-dirs— you're right this unit can download at boot unlike mnemosyne.#4 (root still writes into hermesHome) — the big one: root now writes NOTHING in hermesHome. The full delivered tree — venv AND the complete hub layout including
refs/main— is staged root-owned, then delivered via:The rm/cp race you called out is structurally gone (root's
cp -a $stageVenvreads a root-owned source; the write happens as 986 into 986-owned land).refs/mainis staged root-side and arrives by cp, never via a rootprintf >that could follow a link.Minors:
KittenTTS("KittenML/kitten-tts-mini-0.8")withHF_HOME=$stageHubso it walks the fullhf_hub_downloadpath against the staged tree.resolve/c0272566.../— the download andrefs/maincan't drift from the same commit.pkgs.setprivadded to the unit path.Also honest note: the
mkdir -p "$stageDir" "$stageSnap"ordering in the previous rev ran before$stageVenvwas defined, so your #1 also caught a latent fast-path hole where the delivered-copy cmp referenced$snapfrom the top-level vars — consistent now.- pkgs.setpriv does not exist (mars failed to eval); setpriv is in util-linux. - `sh` is not on a unit's PATH; call ${pkgs.runtimeShell} directly. - Drop `env HOME="$hermesHome"`: unset shell var under `set -u`, and unneeded. - Staging dir 0700 -> 0755: the uid-986 delivery step could not read it. - Rebuild the staging venv whenever the stamp misses, else an input bump delivered the old venv under the new stamp. - Add diffutils: `cmp` in the fast path was missing, so every boot re-delivered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>uv reads the version from the wheel filename, and the fetchurl store path's hash prefix made it reject the file ("invalid version"). Expose it via a linkFarm under kittentts-0.8.1-py3-none-any.whl. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>