Compare commits

..
14 Commits
Author SHA1 Message Date
darmanandClaude Opus 5.5 e4ee637ccd feat(terra): luna v3 voice, channel marker and rail panel
- TTS defaults to eleven_v3_conversational over the Text to Dialogue
  socket: performs audio tags and gets her German right. Flash still
  works via the TTS socket.
- Every spoken message is prefixed with a [voice] marker so luna's
  profile on mars can tell the channel apart.
- LunaIndicator/LunaCall replaced by a LunaPanel in the bar rail, with
  captions; headless test updated to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 00:52:03 +02:00
darmanandClaude Opus 5.5 386a46c4fe feat(terra): ROCm blender
pkgsRocm build so Cycles can render on the RX 6800 XT over HIP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 00:52:03 +02:00
darmanandClaude Opus 5.5 f8a7b61280 feat(terra): Claude Desktop with Cowork
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 00:52:03 +02:00
darmanandClaude Opus 5.5 4adb9347b8 feat(jupiter): tailnet-only gitea MCP server
gitea-mcp in HTTP mode on :8093, reachable only over tailscale0. It
holds no token; each client sends its own PAT as a Bearer header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 00:52:03 +02:00
darmanandClaude Opus 5.5 565308a448 chore: bump flake inputs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 00:52:03 +02:00
darmanandClaude Opus 5 b3200d965d docs: record the voice-loop traps
Three that each cost a debugging round, and none of which looks like its
cause: terra's mic levels (every default threshold was above the user's
speech), the noise floor chasing a rising sound, and `try_recv()` popping
a command that an `if let Ok(Cancel)` then throws away — which is why
SUPER+V during her reply did nothing at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 23:38:06 +02:00
darmanandClaude Opus 5 ef1be8a7d9 feat(terra): luna voice-mode readout in quickshell
Top-left under the rail: whether the mic is open, whether she is
thinking, and whether she is talking — the states a 5s reply makes you
want to see. No avatar yet; the meter is a placeholder for it, and the
states and their colours are the part that has to read correctly first.

LunaIndicator is Item-rooted so render.sh can shoot all five states
offscreen (tests/LunaIndicatorHeadless.qml); LunaCall is the thin
PanelWindow. Beside the notification stack rather than among the modals:
it never takes the keyboard and raises no scrim, so it is deliberately
absent from `modalOpen`.

LunaState is read-only — the toggle runs the CLI — and treats a missing
socket as normal, since the daemon is a user service that may start after
the shell or not be installed at all.

Two things measurement decided rather than reasoning. ExclusionMode
.Ignore put it OVER the bar (it ignores other surfaces' zones, not just
its own), and switching to Normal was not enough either: the rail's
exclusive zone still left it at y=8 against a bar occupying 0..147. So
the shell passes bar.contentHeight down, the same value the backdrop
already reads. And the card is one chamfered path rather than a
rectangle with a triangle over its corner, which left the square border
showing underneath and read as an artifact.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 23:38:01 +02:00
darmanandClaude Opus 5 67dd30eb72 feat(terra): hands-free voice mode, toggled with SUPER+V
The key is a toggle rather than push-to-talk, because "voice mode is
active" has to be a state something can display; a held key cannot
sustain one. So the daemon grew a VAD loop: speech starts a capture,
`silenceMs` of quiet ends it. A turn is awaited inline, which is also
what stops her hearing herself — the tick cannot fire while she speaks.

The keybind runs the CLI, not a quickshell GlobalShortcut, so the mic can
still be switched off while the shell is reloading and a QML error can
never leave it open with nothing on screen saying so.

Every threshold had to be resized for terra's mic, which is QUIET:
0.001-0.002 ambient against 0.0045 speaking, about 6dB apart. The VAD
floor of 0.006 and whisper's own 0.015 gate both sat ABOVE the user's
voice, and a 3.5x multiplier cannot separate signals that close — three
independent reasons nothing could ever trigger. Hence vadMult 1.8,
vadFloor 0.0025, and stt::normalize lifting the clip (capped at 30x, so
near-silence is not amplified into words) before whisper sees it.

The floor tracker also had to stop feeding on everything: a rising sound
dragged its own threshold up faster than it rose, and a tone peaking at
0.029 was outrun by a threshold that reached 0.018. Only samples below
the current trigger update it now.

Four diagnostics, because none of the above was visible while it was
failing: `watch` prints the socket protocol, `say` runs a turn from typed
text (a broken mic vs a broken pipeline), `level` is a live meter with
the trigger marked, `devices` lists the inputs. Amplitude is now emitted
on every tick, not only while she speaks — a meter that moves only during
playback cannot answer "is it hearing me".

Also: connect-time replay of mode/state, so restarting the shell while
voice mode is on does not leave it blind until the next event, which in
an idle conversation is never.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 23:37:51 +02:00
darmanandClaude Opus 5 d089743f75 feat(terra): talk to luna with push-to-talk
luna-voiced: mic -> whisper-cpp (local, vulkan) -> Hermes' API on mars ->
ElevenLabs streaming TTS -> speaker, bound to SUPER+V. State, amplitude
and a viseme timeline are published on a unix socket; nothing consumes
that yet, but it is the data source the quickshell avatar will read.

STT stays local on purpose. mars already runs faster-whisper for Telegram
voice notes, but routing the mic through it would put every spoken word
on the network for no latency win.

Shape notes, all of them things the first live runs decided:

  - Viseme offsets are relative to speech_start (emitted when the first
    PCM frame reaches the speaker), never wall clock, so the shell and
    the daemon never have to compare clocks. ElevenLabs sends character
    alignment, not visemes, so the mapping is ours; the glTF morph target
    names are the contract.
  - clean_for_speech is a whitelist. She writes "(¬‿¬)", whose characters
    live in Latin-1 and punctuation blocks rather than any emoji range,
    so enumerating what to drop never converges. Em dashes become commas
    because they are a pause — dropping one ran two clauses together and
    changed what a sentence meant.
  - whisper invents words out of silence ("you", "Thank you."), and a
    held key over a quiet room is long AND empty, so has_speech gates on
    the loudest 100ms window rather than on duration.
  - Beeps on key down/up and a distinct low tone for "heard nothing":
    push-to-talk with no acknowledgement is indistinguishable from broken,
    and a dead mic must not feel like a slow reply.
  - A warmup request at startup: her first reply after the container
    starts costs ~16s against ~5s warm, and the first one is always the
    one you are waiting for.
  - `luna-voice watch` prints the socket protocol and `luna-voice say`
    runs a turn from typed text, which is how a broken mic is told apart
    from a broken pipeline.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 22:45:17 +02:00
darmanandClaude Opus 5 8dc3affdda feat(mars): expose luna over the OpenAI-compatible API server
API_SERVER_ENABLED/HOST/PORT in the env template, bound 0.0.0.0 for the
same reason the dashboard is: the container is --network=host and nothing
opens 8642 in the firewall, so it is reachable over tailscale0 only.

API_SERVER_KEY has to go through .env like the ElevenLabs key did
(461d6cc). Confirmed the same way: with the key set only in the container
env the server came up — so API_SERVER_ENABLED/HOST *did* pass through —
but answered every bearer token with "Invalid gateway API key". The s6
passthrough list carries the plain API_SERVER_* vars and drops the secret
one. The .env write is now a write_env_key helper so both keys share it.

restartUnits gains prepare-dirs, which is what writes the .env line; the
container then has to come back up to read it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 22:45:05 +02:00
darmanandClaude Opus 5 9566624686 secrets(mars,terra): add the Hermes API server key
hermes_api_server_key is the bearer token for Hermes' OpenAI-compatible
API server, generated once and stored in both vaults: mars serves it,
terra is its only client.

elevenlabs_api_key is mirrored into terra for the same reason — terra
calls ElevenLabs directly (the client-direct path Hermes documents)
rather than relaying audio through mars, so both hosts need the value
and a rotation touches two files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 22:44:57 +02:00
darmanandClaude Opus 5 10046bf846 fix(mars): restart hermes when prepare-dirs' script changes
prepare-dirs is an inactive oneshot and switch-to-configuration only
restarts active units, so the .env delivery added in 461d6cc deployed
cleanly and then did nothing — the container kept running with the old
script until something else restarted it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 18:36:21 +02:00
darmanandClaude Opus 5 461d6cc441 fix(mars): deliver the ElevenLabs key through Hermes' .env
The image's s6 init hands the gateway a fixed set of container env vars;
ELEVENLABS_API_KEY is not one of them, so --env-file alone left
text_to_speech reporting it unset even though the var was in the
container config and the resolver read os.environ fine.

prepare-dirs now merges the single line into ${hermesHome}/.env
(HERMES_HOME=/opt/data), as uid 986 so root never writes into hermesHome,
with the key on an fd opened as root so it stays out of argv and the journal.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 18:35:13 +02:00
darmanandClaude Opus 5 1dd8edcbb0 fix(mars): restart hermes when the ElevenLabs key rotates
podman reads --env-file only at container creation, so re-rendering
hermes-agent.env left the running container on its old environment and
TTS kept reporting ELEVENLABS_API_KEY not set after the key landed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 18:10:04 +02:00
31 changed files with 5371 additions and 46 deletions
+3
View File
@@ -17,3 +17,6 @@ keys.txt
# local visual-verification output # local visual-verification output
.artifacts/ .artifacts/
# rust build output
target/
+32
View File
@@ -210,6 +210,38 @@ kept its ssh host key. Run it after ANY change to the kexec paths.
- **`kexec-local` stages on `/var/tmp`, not `/tmp`**: `kexec-run.sh` appends a fresh cpio - **`kexec-local` stages on `/var/tmp`, not `/tmp`**: `kexec-run.sh` appends a fresh cpio
to `kexec/initrd` in place and execs binaries from that dir, so a size-capped or to `kexec/initrd` in place and execs binaries from that dir, so a size-capped or
`noexec` tmpfs gives a half-written initrd or a bare "Permission denied". `noexec` tmpfs gives a half-written initrd or a bare "Permission denied".
- **A sops secret a USER service reads needs `owner`, not a mode.** `luna-voiced`
(`services/desktop/luna-voice.nix`) runs as darman's user unit, and a user manager
cannot read root:root 0400 — `LoadCredential` does not bridge that gap either.
Hence `sops.secrets.<x>.owner = "darman"` in `hosts/terra/secrets.nix`.
- **`elevenlabs_api_key` is in BOTH `secrets/mars.yaml` and `secrets/terra.yaml`** —
same value. terra calls ElevenLabs directly (the client-direct path Hermes
documents) rather than relaying audio through mars, so rotating that key means
editing two files. `hermes_api_server_key` is duplicated for the same reason:
mars serves the API, terra is its only client.
- **terra's mic is QUIET, and every voice threshold is sized for it.** Measured on
the AIR 192|4 (`luna-voice level`): ~0.001–0.002 RMS ambient, only 0.0045–0.005
for normal speech — about 6dB of separation. Consequences, all of which looked
like "the daemon is broken": a VAD multiplier above ~2 can never fire, an
absolute trigger floor of 0.006 sits *above* the user's voice, and whisper's own
gate at 0.015 discarded every word the VAD did capture. Hence `vadMult = 1.8`,
`vadFloor = 0.0025`, and `stt::normalize` scaling the clip up (capped at 30x)
before whisper sees it. Raising the gain on the interface is worth more than any
of these; tune via `homelab.lunaVoice.vad*` rather than editing Rust.
- **`try_recv()` POPS — an `if let Ok(Cancel)` destroys every other command.**
luna-voiced polls for a cancel while she speaks; written as a single `if let`,
any `say`/`toggle`/`mode off` arriving mid-reply was taken off the channel and
dropped, so SUPER+V during her answer did nothing at all and the next turn never
ran. Non-cancel commands are collected and re-sent into the channel after the
turn instead (`requeue` in `turn()`).
- **A rising sound must not update the noise floor.** Feeding every sample into the
floor EMA lets speech drag its own threshold up faster than it rises — measured,
a tone peaking at 0.029 was chased by a threshold that reached 0.018 and never
triggered. Only samples *below* the current trigger update the floor.
- **whisper invents words out of silence** — "you", "Thank you.", "Bye." — and a
held push-to-talk key over a quiet room is long AND empty, so duration alone
does not catch it. `stt::has_speech` gates on the loudest 100ms window instead;
without it every stray keypress sends a hallucinated sentence to luna.
- **terra's greeter is a throwaway Hyprland running `dotfiles/quickshell/greeter.qml`** - **terra's greeter is a throwaway Hyprland running `dotfiles/quickshell/greeter.qml`**
(`services/desktop/quickshell-greeter.nix`). It must exit after login or greetd never (`services/desktop/quickshell-greeter.nix`). It must exit after login or greetd never
starts the session, and with a Lua config `hyprctl dispatch exit` is REJECTED — it needs starts the session, and with a Lua config `hyprctl dispatch exit` is REJECTED — it needs
@@ -4,7 +4,7 @@ import QtQuick.Layouts
import Quickshell import Quickshell
import Quickshell.Wayland import Quickshell.Wayland
import qs.HyprChrome.Theme import qs.HyprChrome.Theme
import qs.HyprChrome.Widgets.Bar.Debug import qs.HyprChrome.Widgets.Bar.Luna
import qs.HyprChrome.Widgets.Bar.Host import qs.HyprChrome.Widgets.Bar.Host
import qs.HyprChrome.Widgets.Bar.Vitals import qs.HyprChrome.Widgets.Bar.Vitals
import qs.HyprChrome.Widgets.Bar.Workspaces import qs.HyprChrome.Widgets.Bar.Workspaces
@@ -87,8 +87,8 @@ PanelWindow {
// //
// The zone still follows the target height, so tiled windows reflow once // The zone still follows the target height, so tiled windows reflow once
// per toggle, at the start, and slide while the panels animate. // per toggle, at the start, and slide while the panels animate.
readonly property real expandedContent: Math.max(hostPanel.expandedHeight, workspacesPanel.expandedHeight, vitalsPanel.expandedHeight, testPanel.expandedHeight, trayPanel.expandedHeight) + window.margin * 2 readonly property real expandedContent: Math.max(hostPanel.expandedHeight, workspacesPanel.expandedHeight, vitalsPanel.expandedHeight, lunaPanel.expandedHeight, trayPanel.expandedHeight) + window.margin * 2
readonly property real contentHeight: Math.max(hostPanel.targetHeight, workspacesPanel.targetHeight, vitalsPanel.targetHeight, testPanel.targetHeight, trayPanel.targetHeight) + window.margin * 2 readonly property real contentHeight: Math.max(hostPanel.targetHeight, workspacesPanel.targetHeight, vitalsPanel.targetHeight, lunaPanel.targetHeight, trayPanel.targetHeight) + window.margin * 2
implicitHeight: Math.round(window.expandedContent) implicitHeight: Math.round(window.expandedContent)
@@ -157,8 +157,8 @@ PanelWindow {
// and a trace at each end. // and a trace at each end.
Item { Layout.fillWidth: true } Item { Layout.fillWidth: true }
TestPanel { LunaPanel {
id: testPanel id: lunaPanel
expanded: window.expanded expanded: window.expanded
toggleOnClick: false toggleOnClick: false
@@ -0,0 +1,200 @@
pragma ComponentBehavior: Bound
import QtQuick
import qs.HyprChrome.Theme
import qs.HyprChrome.Widgets.Bar.Panels
// luna's voice loop on the rail (services/desktop/luna-voice.nix).
//
// Always present, unlike the daemon it watches: the panel is where you look
// to find out whether she is listening, so it has to be there when the answer
// is "no" — including when luna-voiced is not running at all. SUPER+V toggles
// LISTENING, not this.
//
// Collapsed it is the state and nothing else, because that is the one fact
// worth a glance; expanded it carries the last exchange, which is the thing
// you actually want during the seconds her reply takes to arrive.
BarPanel {
id: luna
// Overridable so the headless render can stage every state without a
// daemon; in the rail they track the socket below.
property string daemonState: state.state
property bool active: state.active
property bool connected: state.connected
property real rms: state.rms
property string userText: state.userText
property string lunaText: state.lunaText
// Instantiated here rather than in the shell: the panel is the only
// reader, and it is a second reader that would make this shell state.
readonly property var state: LunaState {}
panelId: "LUN"
title: "LUNA"
meta: luna.active ? "VOICE" : "IDLE"
// Sized to content like the tray, and per DENSITY: collapsed it is one
// word, so reserving the expanded body's width would leave the rail
// carrying a mostly empty panel for the state it spends most of its time
// in. It floats between two spacers, so the width change costs no other
// panel anything.
implicitWidth: luna.expanded
? Math.max(luna.headerMinWidth, luna.padding * 2 + body.implicitWidth)
: Math.max(luna.headerMinWidth, luna.briefLeft + brief.implicitWidth + luna.padding)
readonly property color stateColor: {
if (!luna.connected)
return Theme.disabled;
switch (luna.daemonState) {
case "listening":
return Theme.accent;
case "thinking":
return Theme.muted;
case "speaking":
return Theme.accentSoft;
default:
return luna.active ? Theme.text : Theme.raised;
}
}
readonly property string stateLabel: {
if (!luna.connected)
return "OFFLINE";
if (!luna.active)
return "OFF";
switch (luna.daemonState) {
case "listening":
return "LISTENING";
case "thinking":
return "THINKING";
case "speaking":
return "SPEAKING";
default:
return "READY";
}
}
summary: Row {
id: brief
spacing: 6
// Breathes only while the mic is open, so "she is listening to me
// right now" is legible without reading the word next to it.
Rectangle {
width: 7
height: 7
radius: 4
anchors.verticalCenter: parent.verticalCenter
color: luna.stateColor
SequentialAnimation on opacity {
running: luna.daemonState === "listening"
loops: Animation.Infinite
alwaysRunToEnd: true
NumberAnimation { to: 0.25; duration: 700; easing.type: Easing.InOutQuad }
NumberAnimation { to: 1.0; duration: 700; easing.type: Easing.InOutQuad }
}
}
Text {
anchors.verticalCenter: parent.verticalCenter
text: luna.stateLabel
color: luna.stateColor
font.family: Theme.displayFont
font.pixelSize: 13
font.bold: true
}
}
Column {
id: body
spacing: 6
// Level meter: live whenever voice mode is on, not only once a
// trigger has fired, so a mic too quiet to reach the threshold still
// shows movement instead of looking broken.
Row {
id: meter
width: 260
height: 18
spacing: 2
readonly property int count: 28
readonly property bool sweeping: luna.daemonState === "thinking"
property real phase: 0
NumberAnimation on phase {
running: meter.sweeping
loops: Animation.Infinite
from: 0
to: 1
duration: 1200
}
Repeater {
model: meter.count
Rectangle {
required property int index
width: (meter.width - (meter.count - 1) * meter.spacing) / meter.count
height: {
if (meter.sweeping) {
const pos = index / (meter.count - 1);
return 2 + 14 * Math.max(0, 1 - Math.abs(pos - meter.phase) * 6);
}
if (!luna.active && luna.daemonState !== "speaking")
return 2;
// sqrt opens up the quiet end, which on this mic is
// where all of the speech actually sits.
const level = Math.min(1, Math.sqrt(luna.rms) * 6.0);
const centre = 1 - Math.abs(index / (meter.count - 1) - 0.5) * 1.6;
return 2 + 16 * level * Math.max(0.15, centre);
}
anchors.bottom: parent.bottom
color: luna.stateColor
opacity: meter.sweeping ? 0.9 : 0.5 + 0.5 * (height / 18)
Behavior on height {
enabled: !meter.sweeping
NumberAnimation { duration: 70; easing.type: Easing.OutQuad }
}
}
}
}
Text {
width: 260
visible: luna.userText !== ""
text: `> ${luna.userText}`
color: Theme.muted
font.family: Theme.microFont
font.pixelSize: 10
wrapMode: Text.WordWrap
maximumLineCount: 2
elide: Text.ElideRight
}
Text {
width: 260
visible: luna.lunaText !== ""
text: luna.lunaText
color: Theme.text
font.family: Theme.microFont
font.pixelSize: 11
wrapMode: Text.WordWrap
maximumLineCount: 4
elide: Text.ElideRight
}
Text {
visible: luna.userText === "" && luna.lunaText === ""
text: luna.connected ? "SUPER+V TO TALK" : "DAEMON NOT RUNNING"
color: Theme.muted
font.family: Theme.microFont
font.pixelSize: 9
font.letterSpacing: 1.4
}
}
}
@@ -0,0 +1,106 @@
pragma ComponentBehavior: Bound
import QtQuick
import Quickshell
import Quickshell.Io
// Everything the Luna surfaces know, read off luna-voiced's unix socket
// (services/desktop/luna-voice.nix).
//
// Read-only on purpose: the key that toggles voice mode runs `luna-voice
// toggle` from Hyprland rather than writing here, so the toggle keeps working
// while the shell is reloading — and a QML error can never leave the mic on
// with nothing showing it.
//
// The daemon is a USER service that may start after the shell, restart under
// it, or not be installed at all, so a missing socket is a normal state and
// not an error: `reconnect` retries quietly and `connected` is what the
// widget shows.
QtObject {
id: root
// "idle" | "listening" | "thinking" | "speaking"
property string state: "idle"
// Voice mode itself — the widget is only on screen while this is true.
property bool active: false
property bool connected: socket.connected
// Mic level while listening, her output level while speaking. Drives the
// only genuinely live part of the visuals.
property real rms: 0
property string userText: ""
property string lunaText: ""
property string error: ""
readonly property var socket: Socket {
id: socket
path: `${Quickshell.env("XDG_RUNTIME_DIR") ?? "/run/user/1000"}/luna-voiced.sock`
connected: true
onConnectionStateChanged: {
if (!socket.connected) {
// A daemon that went away is not a daemon that is idle: reset
// rather than leave the last frame of a conversation frozen
// on screen.
root.active = false;
root.state = "idle";
root.rms = 0;
}
}
parser: SplitParser {
splitMarker: "\n"
onRead: line => root.ingest(line)
}
}
// Retry forever, cheaply. Setting `connected` true on an already-open
// socket is a no-op, so this doubles as the initial connect.
readonly property var reconnect: Timer {
interval: 2000
running: !socket.connected
repeat: true
onTriggered: socket.connected = true
}
function ingest(line: string): void {
let msg;
try {
msg = JSON.parse(line);
} catch (e) {
return; // a half-written line is not worth a console error
}
switch (msg.type) {
case "mode":
root.active = msg.active;
if (!msg.active) {
root.state = "idle";
root.rms = 0;
}
break;
case "state":
root.state = msg.value;
// Level is only meaningful during the phase that produces it.
if (msg.value !== "listening" && msg.value !== "speaking")
root.rms = 0;
break;
case "amplitude":
root.rms = msg.rms;
break;
case "caption":
if (msg.speaker === "user") {
// A new user caption starts a turn, so it is what clears her
// last reply — NOT speech_start, which arrives after her
// first caption, not before it.
root.userText = msg.text;
root.lunaText = "";
} else {
root.lunaText = msg.text;
}
break;
case "error":
root.error = msg.message;
break;
}
}
}
@@ -0,0 +1,90 @@
import QtQuick
import qs.HyprChrome.Widgets.Bar.Luna
// The rail's luna panel in both densities and every daemon state. Collapsed
// is the left column (state only), expanded the right (messages) — the two
// things the panel has to get right, side by side.
//
// ./tools/quickshell-preview/render.sh \
// tests/LunaPanelHeadless.qml \
// .artifacts/quickshell-preview/luna-panel.png 640 560
Rectangle {
width: 640
height: 560
color: "#0f1012"
Row {
x: 16
y: 16
spacing: 24
Column {
spacing: 12
LunaPanel {
expanded: false
connected: true
active: false
daemonState: "idle"
}
LunaPanel {
expanded: false
connected: true
active: true
daemonState: "listening"
rms: 0.02
}
LunaPanel {
expanded: false
connected: true
active: true
daemonState: "thinking"
}
LunaPanel {
expanded: false
connected: true
active: true
daemonState: "speaking"
rms: 0.05
}
LunaPanel {
expanded: false
connected: false
}
}
Column {
spacing: 12
LunaPanel {
expanded: true
connected: true
active: false
daemonState: "idle"
}
LunaPanel {
expanded: true
connected: true
active: true
daemonState: "listening"
rms: 0.02
userText: "hey luna, what's the time?"
}
LunaPanel {
expanded: true
connected: true
active: true
daemonState: "speaking"
rms: 0.05
userText: "hey luna, what's the time?"
lunaText: "It's 23:40 on Sunday night, chief. Shouldn't you be asleep?"
}
}
}
}
Generated
+33 -33
View File
@@ -14,11 +14,11 @@
"uv2nix": "uv2nix" "uv2nix": "uv2nix"
}, },
"locked": { "locked": {
"lastModified": 1789662015, "lastModified": 1789888175,
"narHash": "sha256-CveW/4U/zNIRfLpH6vMv8qisfotft8XJC47KqAhViZ0=", "narHash": "sha256-BWzpZ0/HChcL8F2+hl0GoPaIsWBprMrReeOfrhLSk5o=",
"owner": "nix-community", "owner": "nix-community",
"repo": "authentik-nix", "repo": "authentik-nix",
"rev": "9a176c0a1889921f3ed19d29a047bafaf20b9a24", "rev": "477392e1a239228de9a8011786cf673dfbb78c75",
"type": "github" "type": "github"
}, },
"original": { "original": {
@@ -30,16 +30,16 @@
"authentik-src": { "authentik-src": {
"flake": false, "flake": false,
"locked": { "locked": {
"lastModified": 1788978241, "lastModified": 1789678329,
"narHash": "sha256-tAdDYHIur6ewVZlRz6JPtLivqctYLLtD0S8uehuuxSM=", "narHash": "sha256-wt6cjarJI5NXSHgpONjqE9JiltBqkCoPyLLiIS83kuI=",
"owner": "goauthentik", "owner": "goauthentik",
"repo": "authentik", "repo": "authentik",
"rev": "dce85a5b64a429206199e9ffddc602060643f17a", "rev": "e5a0d2f7572cb776eee7a3e9355937ce38973761",
"type": "github" "type": "github"
}, },
"original": { "original": {
"owner": "goauthentik", "owner": "goauthentik",
"ref": "version/2026.8.2", "ref": "version/2026.8.3",
"repo": "authentik", "repo": "authentik",
"type": "github" "type": "github"
} }
@@ -67,11 +67,11 @@
] ]
}, },
"locked": { "locked": {
"lastModified": 1781152676, "lastModified": 1789770686,
"narHash": "sha256-RxWs5ND31KzTG7wvMM+PMfUjyNpmIEr999lqNARaM5o=", "narHash": "sha256-uZkBR7yHdIKUFB5SZdfgh1qkGfI3XmYmI/lTiquxbck=",
"owner": "nix-community", "owner": "nix-community",
"repo": "disko", "repo": "disko",
"rev": "ff8702b4de27f72b4c78573dfb89ec74e36abdf1", "rev": "725ea35e410ad83be4931d1bff7e090eacaf3563",
"type": "github" "type": "github"
}, },
"original": { "original": {
@@ -255,11 +255,11 @@
] ]
}, },
"locked": { "locked": {
"lastModified": 1786747096, "lastModified": 1789357259,
"narHash": "sha256-9QqhmaLVsPhKdMSBaWKjDqeGRn8G4ov4cVuZ6JFwXbo=", "narHash": "sha256-YF2EwlbEEmLgEH0fs7H5sgz9JwwMjoKM2rY5jnGNYwE=",
"owner": "numtide", "owner": "numtide",
"repo": "nix-vm-test", "repo": "nix-vm-test",
"rev": "c8781a0ea2d8417506fff7722eae5a6316461212", "rev": "91825d66623ff030255d98082b5c8af2bbc549ec",
"type": "github" "type": "github"
}, },
"original": { "original": {
@@ -286,11 +286,11 @@
"treefmt-nix": "treefmt-nix" "treefmt-nix": "treefmt-nix"
}, },
"locked": { "locked": {
"lastModified": 1789567195, "lastModified": 1790148129,
"narHash": "sha256-ymvPUaBvMmGvxsPbmrocpxO4XK+H+K12YjgKIyG3Rwk=", "narHash": "sha256-oOp+FNzqKzFfn9+AwcvzBedUyuUniv7eGxQ0jfzFpQ8=",
"owner": "nix-community", "owner": "nix-community",
"repo": "nixos-anywhere", "repo": "nixos-anywhere",
"rev": "1c2f124e970fed2a49bd14ce0a8b4e9bff74d3b4", "rev": "da83557d8b0bce57a52371b888ffd6d58724e55c",
"type": "github" "type": "github"
}, },
"original": { "original": {
@@ -339,11 +339,11 @@
}, },
"nixpkgs": { "nixpkgs": {
"locked": { "locked": {
"lastModified": 1788316716, "lastModified": 1789546076,
"narHash": "sha256-bc7rSpXIdn9QWGNqfWcPZWOhEVF8NoeAZkWq0XWnf/k=", "narHash": "sha256-zVxLZiSnmaaPLwnhj7pwmqe3axBg/C6nG5JZsJMh2g4=",
"owner": "NixOS", "owner": "NixOS",
"repo": "nixpkgs", "repo": "nixpkgs",
"rev": "3ed67ec0a4d3c7ab4ae1f04f8ee8df07bfa506a2", "rev": "b1b875982b17dabde9b4a37f3e229e74913e6db3",
"type": "github" "type": "github"
}, },
"original": { "original": {
@@ -370,11 +370,11 @@
}, },
"nixpkgs-unstable": { "nixpkgs-unstable": {
"locked": { "locked": {
"lastModified": 1789632929, "lastModified": 1790149054,
"narHash": "sha256-RjR8AyvGlWuw16XRj7C1YDEw4E27ciuyPeB2nDMzgTU=", "narHash": "sha256-FfjXg6mAzKQBCY+7V9keGWOXG4DV2Eym/qA987lFl8A=",
"owner": "NixOS", "owner": "NixOS",
"repo": "nixpkgs", "repo": "nixpkgs",
"rev": "a32edd7654519351e48e80372a928df336394670", "rev": "8825bebf6324e0579d012936eff73379af284b6d",
"type": "github" "type": "github"
}, },
"original": { "original": {
@@ -386,11 +386,11 @@
}, },
"nixpkgs_2": { "nixpkgs_2": {
"locked": { "locked": {
"lastModified": 1789654592, "lastModified": 1790113641,
"narHash": "sha256-vrwAiXmz+0hs/IWXZalFG/Ws6xQRcW+gClUDlfvcfjQ=", "narHash": "sha256-D2aaQetafdpHxbwA4ldAxs51UX0b11pXL+JL/5L8VrM=",
"owner": "NixOS", "owner": "NixOS",
"repo": "nixpkgs", "repo": "nixpkgs",
"rev": "ecc58f32d1069a3e3f8a991e2233d365689f748d", "rev": "1bc55b9def8165e82073919945c3239903fe4dc2",
"type": "github" "type": "github"
}, },
"original": { "original": {
@@ -524,11 +524,11 @@
] ]
}, },
"locked": { "locked": {
"lastModified": 1789691124, "lastModified": 1789890976,
"narHash": "sha256-k+I+R6uwHX3VcJ7326qLV6vCahZUgsVl+i8sSU/Stxk=", "narHash": "sha256-GKwH3zpy7tartuJMG0Rv/xUsdetG1QLmTVv8UKgJLmA=",
"owner": "Mic92", "owner": "Mic92",
"repo": "sops-nix", "repo": "sops-nix",
"rev": "1e73e8f7176d65e1b55e324de099bbfff4b2c574", "rev": "7214124c20c1542c90deb54af50e2f53ae02711f",
"type": "github" "type": "github"
}, },
"original": { "original": {
@@ -539,16 +539,16 @@
}, },
"systems": { "systems": {
"locked": { "locked": {
"lastModified": 1689347949, "lastModified": 1680978846,
"narHash": "sha256-12tWmuL2zgBgZkdoB6qXZsgJEH9LR3oUgpaQq2RbI80=", "narHash": "sha256-Gtqg8b/v49BFDpDetjclCYXm8mAnTrUzR0JnE2nv5aw=",
"owner": "nix-systems", "owner": "nix-systems",
"repo": "default-linux", "repo": "x86_64-linux",
"rev": "31732fcf5e8fea42e59c2488ad31a0e651500f68", "rev": "2ecfcac5e15790ba6ce360ceccddb15ad16d08a8",
"type": "github" "type": "github"
}, },
"original": { "original": {
"owner": "nix-systems", "owner": "nix-systems",
"repo": "default-linux", "repo": "x86_64-linux",
"type": "github" "type": "github"
} }
}, },
+1
View File
@@ -24,6 +24,7 @@
../../services/media/seerr.nix ../../services/media/seerr.nix
../../services/media/immich.nix ../../services/media/immich.nix
../../services/dev/gitea.nix ../../services/dev/gitea.nix
../../services/dev/gitea-mcp.nix
../../services/dev/obsidian-livesync.nix ../../services/dev/obsidian-livesync.nix
]; ];
+35 -1
View File
@@ -181,7 +181,7 @@ in
wants = [ "network-online.target" ]; wants = [ "network-online.target" ];
after = [ "network-online.target" ]; after = [ "network-online.target" ];
unitConfig.RequiresMountsFor = [ "/mnt/jupiter" ]; unitConfig.RequiresMountsFor = [ "/mnt/jupiter" ];
path = [ pkgs.git pkgs.tea pkgs.curl pkgs.coreutils pkgs.yq-go ]; path = [ pkgs.git pkgs.tea pkgs.curl pkgs.coreutils pkgs.yq-go pkgs.gnugrep pkgs.util-linux ];
serviceConfig.Type = "oneshot"; serviceConfig.Type = "oneshot";
# Everything here is either local or bounded to ~30s by the probe loop, so # Everything here is either local or bounded to ~30s by the probe loop, so
# anything past two minutes is a hang, not slowness. # anything past two minutes is a hang, not slowness.
@@ -265,6 +265,36 @@ in
chmod --reference="$conf" "$conf.new" chmod --reference="$conf" "$conf.new"
mv -f "$conf.new" "$conf" mv -f "$conf.new" "$conf"
# The image's s6 init hands the gateway a fixed set of container env vars,
# and ELEVENLABS_API_KEY isn't among them (OPENCODE_GO_API_KEY is) — so the
# key only reaches Hermes via .env, which load_env() reads directly.
# Re-asserted every start: a sops rotation must win over a stale line.
# setpriv because root must not write into hermesHome — luna could plant a
# symlink there. The key arrives on an fd opened while still root, so it
# never lands in argv or the journal and her uid needs no /run/secrets access.
#
# API_SERVER_KEY is in the same boat, confirmed the same way: with it set
# only in the container env the server came up (so API_SERVER_ENABLED/HOST
# DID pass through) but answered every bearer token with
# "Invalid gateway API key" — the passthrough list carries the plain
# API_SERVER_* vars and drops the secret one.
write_env_key() {
setpriv --reuid=${hermesUid} --regid=${hermesGid} --clear-groups \
${pkgs.runtimeShell} -c '
set -eu
envf=$1
name=$2
umask 077
key=$(cat)
[ -e "$envf" ] || : > "$envf"
{ grep -v "^$name=" "$envf" || true
printf "%s=%s\n" "$name" "$key"; } > "$envf.new"
mv -f "$envf.new" "$envf"
' _ ${hermesHome}/.env "$1"
}
write_env_key ELEVENLABS_API_KEY < ${config.sops.secrets.elevenlabs_api_key.path}
write_env_key API_SERVER_KEY < ${config.sops.secrets.hermes_api_server_key.path}
if [ -d ${hermesHome}/.config ]; then if [ -d ${hermesHome}/.config ]; then
chown ${hermesUid}:${hermesGid} ${hermesHome}/.config chown ${hermesUid}:${hermesGid} ${hermesHome}/.config
fi fi
@@ -350,6 +380,10 @@ in
]; ];
requires = [ "hermes-agent-prepare-dirs.service" ]; requires = [ "hermes-agent-prepare-dirs.service" ];
unitConfig.RequiresMountsFor = [ "/mnt/jupiter" ]; unitConfig.RequiresMountsFor = [ "/mnt/jupiter" ];
# prepare-dirs is an inactive oneshot, and switch-to-configuration only
# restarts ACTIVE units — so editing its script otherwise lands nowhere
# until something else happens to restart the container.
restartTriggers = [ config.systemd.services.hermes-agent-prepare-dirs.script ];
}; };
# The two Gitea webhook routes, written as config (not via `hermes webhook # The two Gitea webhook routes, written as config (not via `hermes webhook
+20 -2
View File
@@ -29,8 +29,10 @@
sops.secrets.telegram_bot_token = { }; sops.secrets.telegram_bot_token = { };
sops.secrets.hermes_dashboard_oidc_client_secret = { }; sops.secrets.hermes_dashboard_oidc_client_secret = { };
# ElevenLabs TTS/STT. Unlike the webhook secret this must live in the # ElevenLabs TTS/STT. Unlike the webhook secret this must live in the
# container env — luna calls the API herself. # container env — luna calls the API herself. restartUnits because podman
sops.secrets.elevenlabs_api_key = { }; # reads --env-file only at container creation: rotating the key without it
# re-renders the file under a container that never re-reads it.
sops.secrets.elevenlabs_api_key.restartUnits = [ "podman-hermes-agent.service" ];
# Same value as secrets/jupiter.yaml (the sending side), stored WITHOUT a # Same value as secrets/jupiter.yaml (the sending side), stored WITHOUT a
# trailing newline — a stray newline would change the HMAC key and fail # trailing newline — a stray newline would change the HMAC key and fail
# every delivery. Written host-side by hermes-agent-webhook-routes, so it # every delivery. Written host-side by hermes-agent-webhook-routes, so it
@@ -38,6 +40,18 @@
sops.secrets.gitea_hermes_webhook_secret = { sops.secrets.gitea_hermes_webhook_secret = {
restartUnits = [ "hermes-agent-webhook-routes.service" ]; restartUnits = [ "hermes-agent-webhook-routes.service" ];
}; };
# Bearer token for the OpenAI-compatible API server (see the env template
# below). Same value in secrets/terra.yaml, where luna-voiced reads it.
# API_SERVER_HOST=0.0.0.0 for the same reason the dashboard binds it (see
# hermes-agent.nix): the container is --network=host and nothing opens 8642
# in the firewall, so it is reachable over tailscale0 only — LAN-closed.
# prepare-dirs writes it into luna's .env (the container env alone does not
# reach the gateway — see hermes-agent.nix); the container then has to come
# back up to read it.
sops.secrets.hermes_api_server_key.restartUnits = [
"hermes-agent-prepare-dirs.service"
"podman-hermes-agent.service"
];
sops.templates."hermes-agent.env".content = '' sops.templates."hermes-agent.env".content = ''
OPENCODE_GO_API_KEY=${config.sops.placeholder.opencode_go_api_key} OPENCODE_GO_API_KEY=${config.sops.placeholder.opencode_go_api_key}
TELEGRAM_BOT_TOKEN=${config.sops.placeholder.telegram_bot_token} TELEGRAM_BOT_TOKEN=${config.sops.placeholder.telegram_bot_token}
@@ -47,6 +61,10 @@
WEBHOOK_PORT=8644 WEBHOOK_PORT=8644
HERMES_DASHBOARD_OIDC_CLIENT_SECRET=${config.sops.placeholder.hermes_dashboard_oidc_client_secret} HERMES_DASHBOARD_OIDC_CLIENT_SECRET=${config.sops.placeholder.hermes_dashboard_oidc_client_secret}
ELEVENLABS_API_KEY=${config.sops.placeholder.elevenlabs_api_key} ELEVENLABS_API_KEY=${config.sops.placeholder.elevenlabs_api_key}
API_SERVER_ENABLED=true
API_SERVER_HOST=0.0.0.0
API_SERVER_PORT=8642
API_SERVER_KEY=${config.sops.placeholder.hermes_api_server_key}
''; '';
# luna's gitea push token (services/dev/gitea.nix provisions the account + # luna's gitea push token (services/dev/gitea.nix provisions the account +
+21
View File
@@ -18,9 +18,30 @@ in
../../services/monitoring/node-exporter.nix ../../services/monitoring/node-exporter.nix
../../services/desktop/desktop-hyprland.nix ../../services/desktop/desktop-hyprland.nix
../../services/desktop/desktop-apps.nix ../../services/desktop/desktop-apps.nix
../../services/desktop/claude-desktop.nix
../../services/desktop/librechat.nix ../../services/desktop/librechat.nix
../../services/desktop/luna-voice.nix
]; ];
# Talk to luna (Hermes Agent on mars) with SUPER+V — see
# hosts/terra/home/hyprland.nix for the push-to-talk binds.
homelab.lunaVoice = {
enable = true;
hermesKeyFile = config.sops.secrets.hermes_api_server_key.path;
elevenLabsKeyFile = config.sops.secrets.elevenlabs_api_key.path;
# luna's own voice — must stay in sync with `tts.elevenlabs.voice_id` in
# her config.yaml on mars, or she sounds like someone else on the desktop
# than she does on Telegram.
voiceId = "LZAcK8Cx5QjdQhfBsJQZ";
systemPrompt = ''
You are being spoken to out loud and your reply will be read aloud by a
speech synthesiser. Answer in one or two sentences unless asked for
detail. No markdown, no lists, no code blocks — they do not survive the
trip to audio. You may use an audio tag now and then, such as [laughs],
[sighs] or [whispers]; the synthesiser performs them.
'';
};
networking.hostName = "terra"; networking.hostName = "terra";
homelab.greeter = { homelab.greeter = {
+6 -1
View File
@@ -195,6 +195,11 @@ in
# expand / collapse the hyprchrome bar as a whole # expand / collapse the hyprchrome bar as a whole
(bind "SUPER + A" (dsp.global "quickshell:chrome")) (bind "SUPER + A" (dsp.global "quickshell:chrome"))
# toggle luna's hands-free voice mode (services/desktop/luna-voice.nix).
# Runs the CLI rather than a quickshell GlobalShortcut so the mic can
# still be switched off while the shell is reloading.
(bind "SUPER + V" (dsp.exec "luna-voice toggle"))
(bind "SUPER + B" (dsp.exec "vivaldi")) (bind "SUPER + B" (dsp.exec "vivaldi"))
(bind "SUPER + E" (dsp.exec "cosmic-files")) (bind "SUPER + E" (dsp.exec "cosmic-files"))
(bind "SUPER + SHIFT + G" (dsp.workspaceRef "game")) (bind "SUPER + SHIFT + G" (dsp.workspaceRef "game"))
@@ -237,7 +242,7 @@ in
(bind "SUPER + T" (dsp.togglespecial "terminal")) (bind "SUPER + T" (dsp.togglespecial "terminal"))
(bind "SUPER + C" (dsp.togglespecial "communications")) (bind "SUPER + C" (dsp.togglespecial "communications"))
(bind "SUPER + M" (dsp.togglespecial "music")) (bind "SUPER + M" (dsp.togglespecial "music"))
(bind "SUPER + V" (dsp.togglespecial "version_control")) # (bind "SUPER + V" (dsp.togglespecial "version_control"))
# cycle workspaces # cycle workspaces
(bind "SUPER + ALT + up" (dsp.workspaceRef "e+1")) (bind "SUPER + ALT + up" (dsp.workspaceRef "e+1"))
+12
View File
@@ -27,6 +27,18 @@
# tokens. All four are random, generated once with `sops --set` (see # tokens. All four are random, generated once with `sops --set` (see
# CLAUDE.md) — losing/rotating them just invalidates existing sessions and # CLAUDE.md) — losing/rotating them just invalidates existing sessions and
# any saved per-user API keys, nothing else depends on their value. # any saved per-user API keys, nothing else depends on their value.
# luna-voiced (services/desktop/luna-voice.nix) runs as a USER service, so
# these cannot stay root:root 0400 — a user manager cannot read them and
# LoadCredential does not bridge that gap either. Hence owner, not mode.
#
# hermes_api_server_key mirrors secrets/mars.yaml (the serving side);
# elevenlabs_api_key mirrors it too, because client-direct TTS is the
# documented low-latency path — terra calls ElevenLabs itself rather than
# relaying audio through mars. Rotating that key therefore touches BOTH
# files; see CLAUDE.md.
sops.secrets.hermes_api_server_key.owner = "darman";
sops.secrets.elevenlabs_api_key.owner = "darman";
sops.secrets.librechat_creds_key = { }; sops.secrets.librechat_creds_key = { };
sops.secrets.librechat_creds_iv = { }; sops.secrets.librechat_creds_iv = { };
sops.secrets.librechat_jwt_secret = { }; sops.secrets.librechat_jwt_secret = { };
+139
View File
@@ -0,0 +1,139 @@
{ lib
, stdenv
, fetchurl
, dpkg
, autoPatchelfHook
, makeWrapper
, wrapGAppsHook3
, alsa-lib
, at-spi2-atk
, at-spi2-core
, atk
, cairo
, cups
, dbus
, expat
, glib
, gtk3
, libcap_ng
, libdrm
, libgbm
, libseccomp
, libxkbcommon
, nspr
, nss
, pango
, systemd
, libx11
, libxcb
, libxcomposite
, libxdamage
, libxext
, libxfixes
, libxrandr
, libxtst
, libGL
, libayatana-appindicator
, libnotify
, libpulseaudio
, libsecret
, pipewire
, wayland
, qemu_kvm
, xdg-utils
}:
# Anthropic's official Linux build (Debian .deb), patchelf'd in place. The
# bundled Electron binary is kept rather than swapped for nixpkgs' electron:
# app.asar ships a V8 code cache (compile-cache/*.jsc) tied to that exact build.
#
# Cowork looks for qemu on $PATH but for OVMF and virtiofsd only at fixed
# Debian paths; services/desktop/claude-desktop.nix symlinks those into /usr.
#
# Bump: take the highest `Version:` listed in
# https://downloads.claude.ai/claude-desktop/apt/stable/dists/stable/main/binary-amd64/Packages
stdenv.mkDerivation (finalAttrs: {
pname = "claude-desktop";
version = "2.7032.0";
src = fetchurl {
url = "https://downloads.claude.ai/claude-desktop/apt/stable/pool/main/c/claude-desktop/claude-desktop_${finalAttrs.version}_amd64.deb";
hash = "sha256-Hn9FBLylsvay08QSPRRdcnZH538u4tBGhQcR5h59exE=";
};
nativeBuildInputs = [ dpkg autoPatchelfHook makeWrapper wrapGAppsHook3 ];
buildInputs = [
alsa-lib
at-spi2-atk
at-spi2-core
atk
cairo
cups
dbus
expat
glib
gtk3
libcap_ng # bundled virtiofsd
libdrm
libgbm
libseccomp # bundled virtiofsd
libxkbcommon
nspr
nss
pango
stdenv.cc.cc.lib # node-pty
systemd
libx11
libxcb
libxcomposite
libxdamage
libxext
libxfixes
libxrandr
libxtst
];
# dlopen'd by Chromium, so not in any DT_NEEDED.
runtimeDependencies = [
libGL
libayatana-appindicator
libnotify
libpulseaudio
libsecret
pipewire
systemd
wayland
];
dontWrapGApps = true;
installPhase = ''
runHook preInstall
mkdir -p $out/lib $out/bin
cp -r usr/lib/claude-desktop $out/lib/
cp -r usr/share $out/share
rm -r $out/share/lintian
# Setuid is impossible in the store; Chromium falls back to its userns sandbox.
rm $out/lib/claude-desktop/chrome-sandbox
runHook postInstall
'';
postFixup = ''
makeWrapper $out/lib/claude-desktop/claude-desktop $out/bin/claude-desktop \
"''${gappsWrapperArgs[@]}" \
--prefix PATH : ${lib.makeBinPath [ qemu_kvm xdg-utils ]}
'';
meta = {
description = "Desktop application for Claude.ai";
homepage = "https://claude.ai";
license = lib.licenses.unfree;
sourceProvenance = [ lib.sourceTypes.binaryNativeCode ];
platforms = [ "x86_64-linux" ];
mainProgram = "claude-desktop";
};
})
+32
View File
@@ -0,0 +1,32 @@
{ lib
, rustPlatform
, pkg-config
, alsa-lib
}:
# luna-voiced — the push-to-talk voice loop for luna (Hermes Agent on mars).
# Wiring (secrets, whisper model, systemd user unit) lives in
# services/desktop/luna-voice.nix; this only builds the binary.
#
# cargoLock.lockFile rather than a cargoHash: the lockfile is committed, so a
# dependency bump is a visible diff instead of a hash nobody can read.
#
# alsa-lib is cpal's backend even on a pipewire box — pipewire's ALSA
# compatibility layer is what actually serves it, so there is no pipewire
# build dependency here.
rustPlatform.buildRustPackage {
pname = "luna-voice";
version = "0.1.0";
src = ./luna-voice;
cargoLock.lockFile = ./luna-voice/Cargo.lock;
nativeBuildInputs = [ pkg-config ];
buildInputs = [ alsa-lib ];
meta = {
description = "Push-to-talk voice loop for luna (Hermes Agent)";
mainProgram = "luna-voice";
platforms = lib.platforms.linux;
};
}
+2166
View File
File diff suppressed because it is too large Load Diff
+19
View File
@@ -0,0 +1,19 @@
[package]
name = "luna-voice"
version = "0.1.0"
edition = "2021"
[dependencies]
anyhow = "1"
base64 = "0.22"
cpal = "0.15"
futures-util = { version = "0.3", default-features = false, features = ["sink", "std"] }
hound = "3"
reqwest = { version = "0.12", default-features = false, features = ["json", "stream", "rustls-tls"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "sync", "time", "process", "io-util", "signal"] }
tokio-tungstenite = { version = "0.24", default-features = false, features = ["connect", "rustls-tls-webpki-roots"] }
[profile.release]
opt-level = 2
+311
View File
@@ -0,0 +1,311 @@
use anyhow::{anyhow, Result};
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use std::collections::VecDeque;
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use std::sync::{Arc, Mutex};
/// What whisper wants. Everything captured is resampled to this before it
/// reaches disk.
pub const STT_RATE: u32 = 16_000;
/// What we ask ElevenLabs for (pcm_24000), resampled to the device on the way out.
pub const TTS_RATE: u32 = 24_000;
/// Mic. The stream runs for the whole session, not just while recording:
/// barge-in needs a live RMS while she is speaking, and restarting a cpal
/// stream per utterance costs ~100ms of dropped samples on pipewire.
///
/// cpal::Stream is !Send, so it is owned by a parked thread rather than by
/// this handle — which then holds nothing but atomics and a mutex and can be
/// shared across tokio tasks freely.
pub struct Capture {
pub rms: Arc<AtomicU32>,
recording: Arc<AtomicBool>,
buffer: Arc<Mutex<Vec<f32>>>,
rate: u32,
}
impl Capture {
pub fn new() -> Result<Self> {
let rms = Arc::new(AtomicU32::new(0));
let recording = Arc::new(AtomicBool::new(false));
let buffer = Arc::new(Mutex::new(Vec::new()));
let (r, rec, buf) = (rms.clone(), recording.clone(), buffer.clone());
let rate = spawn_stream(move || {
let device = input_device()?;
// Named in the log because "the mic does not work" is usually
// "the default source is the wrong device", and nothing else in
// the pipeline can tell you that.
eprintln!(
"luna-voice: input device {:?}",
device.name().unwrap_or_else(|_| "<unnamed>".into())
);
let config = device.default_input_config()?;
let rate = config.sample_rate().0;
let channels = config.channels() as usize;
let stream = device.build_input_stream(
&config.into(),
move |data: &[f32], _: &cpal::InputCallbackInfo| {
// Downmix to mono up front; nothing downstream cares about stereo.
let mono: Vec<f32> = data
.chunks(channels)
.map(|f| f.iter().sum::<f32>() / channels as f32)
.collect();
let sum: f32 = mono.iter().map(|s| s * s).sum();
let level = (sum / mono.len().max(1) as f32).sqrt();
r.store(level.to_bits(), Ordering::Relaxed);
if rec.load(Ordering::Relaxed) {
if let Ok(mut b) = buf.lock() {
b.extend_from_slice(&mono);
}
}
},
|e| eprintln!("luna-voice: capture error: {e}"),
None,
)?;
stream.play()?;
Ok((stream, rate))
})?;
Ok(Capture { rms, recording, buffer, rate })
}
pub fn level(&self) -> f32 {
f32::from_bits(self.rms.load(Ordering::Relaxed))
}
pub fn start(&self) {
self.buffer.lock().map(|mut b| b.clear()).ok();
self.recording.store(true, Ordering::Relaxed);
}
/// Stops recording and returns what was captured, resampled to STT_RATE.
pub fn stop(&self) -> Vec<f32> {
self.recording.store(false, Ordering::Relaxed);
let raw = self
.buffer
.lock()
.map(|mut b| std::mem::take(&mut *b))
.unwrap_or_default();
resample(&raw, self.rate, STT_RATE)
}
}
/// Speaker. Fed from the TTS websocket; `clear` is what makes barge-in
/// instant rather than "after the current sentence".
pub struct Playback {
queue: Arc<Mutex<VecDeque<f32>>>,
rate: u32,
}
impl Playback {
pub fn new() -> Result<Self> {
let queue: Arc<Mutex<VecDeque<f32>>> = Arc::new(Mutex::new(VecDeque::new()));
let q = queue.clone();
let rate = spawn_stream(move || {
let device = cpal::default_host()
.default_output_device()
.ok_or_else(|| anyhow!("no output device"))?;
let config = device.default_output_config()?;
let rate = config.sample_rate().0;
let channels = config.channels() as usize;
let stream = device.build_output_stream(
&config.into(),
move |data: &mut [f32], _: &cpal::OutputCallbackInfo| {
let mut q = match q.lock() {
Ok(q) => q,
Err(_) => return,
};
for frame in data.chunks_mut(channels) {
let s = q.pop_front().unwrap_or(0.0);
for out in frame.iter_mut() {
*out = s;
}
}
},
|e| eprintln!("luna-voice: playback error: {e}"),
None,
)?;
stream.play()?;
Ok((stream, rate))
})?;
Ok(Playback { queue, rate })
}
pub fn push(&self, pcm: &[f32]) {
let out = resample(pcm, TTS_RATE, self.rate);
if let Ok(mut q) = self.queue.lock() {
q.extend(out);
}
}
pub fn clear(&self) {
self.queue.lock().map(|mut q| q.clear()).ok();
}
pub fn pending(&self) -> usize {
self.queue.lock().map(|q| q.len()).unwrap_or(0)
}
/// Short sine burst, so you know the key registered without looking at
/// anything. Hermes' voice mode has `beep_enabled` for the same reason:
/// push-to-talk with no acknowledgement is indistinguishable from broken.
///
/// Raised-cosine envelope — a bare sine starting at full amplitude clicks.
pub fn beep(&self, freq: f32, ms: u32) {
let n = (self.rate as f32 * ms as f32 / 1000.0) as usize;
let samples: Vec<f32> = (0..n)
.map(|i| {
let t = i as f32 / self.rate as f32;
let progress = i as f32 / n as f32;
let envelope = (std::f32::consts::PI * progress).sin();
(t * freq * std::f32::consts::TAU).sin() * envelope * 0.15
})
.collect();
if let Ok(mut q) = self.queue.lock() {
q.extend(samples);
}
}
}
/// The microphone, honouring LUNA_INPUT_DEVICE (a case-insensitive substring
/// of the device name) before falling back to the system default.
///
/// Worth having because the default source is a system-wide setting that has
/// nothing to do with which input you actually speak into: a box with an
/// audio interface and onboard audio has several, and picking the silent one
/// looks exactly like a broken pipeline.
fn input_device() -> Result<cpal::Device> {
let host = cpal::default_host();
if let Ok(want) = std::env::var("LUNA_INPUT_DEVICE") {
let want = want.to_lowercase();
if let Ok(devices) = host.input_devices() {
for device in devices {
if device
.name()
.map(|n| n.to_lowercase().contains(&want))
.unwrap_or(false)
{
return Ok(device);
}
}
}
eprintln!("luna-voice: no input device matching {want:?}; using the default");
}
host.default_input_device()
.ok_or_else(|| anyhow!("no input device"))
}
/// Every input cpal can see, for `luna-voice devices`.
pub fn list_inputs() -> Result<Vec<String>> {
let host = cpal::default_host();
let default = host.default_input_device().and_then(|d| d.name().ok());
let mut out = Vec::new();
for device in host.input_devices()? {
let name = device.name().unwrap_or_else(|_| "<unnamed>".into());
let marker = if Some(&name) == default.as_ref() { " (default)" } else { "" };
out.push(format!("{name}{marker}"));
}
Ok(out)
}
/// Builds a cpal stream on a dedicated thread that then parks forever,
/// keeping it alive. Returns the device's sample rate, or the build error —
/// a stream that fails here must surface at startup, not as silence later.
fn spawn_stream<F>(build: F) -> Result<u32>
where
F: FnOnce() -> Result<(cpal::Stream, u32)> + Send + 'static,
{
let (tx, rx) = std::sync::mpsc::channel::<Result<u32, String>>();
std::thread::spawn(move || match build() {
Ok((_stream, rate)) => {
let _ = tx.send(Ok(rate));
// Dropping `stream` would stop the device, so the thread holds it
// for the life of the process.
loop {
std::thread::park();
}
#[allow(unreachable_code)]
drop(_stream);
}
Err(e) => {
let _ = tx.send(Err(e.to_string()));
}
});
rx.recv()
.map_err(|_| anyhow!("audio thread died during setup"))?
.map_err(|e| anyhow!(e))
}
/// Linear resampling. Speech at these rates does not justify a windowed sinc,
/// and rubato would add a dependency for inaudible gain.
pub fn resample(input: &[f32], from: u32, to: u32) -> Vec<f32> {
if from == to || input.is_empty() {
return input.to_vec();
}
let ratio = to as f64 / from as f64;
let out_len = (input.len() as f64 * ratio).round() as usize;
(0..out_len)
.map(|i| {
let src = i as f64 / ratio;
let j = src.floor() as usize;
let frac = (src - j as f64) as f32;
let a = input.get(j).copied().unwrap_or(0.0);
let b = input.get(j + 1).copied().unwrap_or(a);
a + (b - a) * frac
})
.collect()
}
/// 16-bit mono WAV, which is the only thing whisper-cli accepts without
/// converting first.
pub fn write_wav(path: &std::path::Path, samples: &[f32], rate: u32) -> Result<()> {
let spec = hound::WavSpec {
channels: 1,
sample_rate: rate,
bits_per_sample: 16,
sample_format: hound::SampleFormat::Int,
};
let mut w = hound::WavWriter::create(path, spec)?;
for s in samples {
w.write_sample((s.clamp(-1.0, 1.0) * i16::MAX as f32) as i16)?;
}
w.finalize()?;
Ok(())
}
/// Signed 16-bit little-endian, as ElevenLabs sends pcm_*.
pub fn pcm16_to_f32(bytes: &[u8]) -> Vec<f32> {
bytes
.chunks_exact(2)
.map(|b| i16::from_le_bytes([b[0], b[1]]) as f32 / i16::MAX as f32)
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn resample_keeps_length_ratio() {
let input = vec![0.0f32; 1000];
assert_eq!(resample(&input, 48_000, 16_000).len(), 333);
assert_eq!(resample(&input, 24_000, 48_000).len(), 2000);
}
#[test]
fn resample_is_identity_at_equal_rates() {
let input = vec![0.1, -0.4, 0.9];
assert_eq!(resample(&input, 16_000, 16_000), input);
}
#[test]
fn pcm16_roundtrips_through_f32() {
let bytes = [0x00, 0x00, 0xff, 0x7f, 0x01, 0x80];
let got = pcm16_to_f32(&bytes);
assert_eq!(got.len(), 3);
assert!((got[0]).abs() < 1e-6);
assert!((got[1] - 1.0).abs() < 1e-3);
assert!((got[2] + 1.0).abs() < 1e-3);
}
}
+123
View File
@@ -0,0 +1,123 @@
use anyhow::{bail, Result};
use futures_util::StreamExt;
use serde::{Deserialize, Serialize};
use tokio::sync::mpsc;
#[derive(Clone, Serialize, Deserialize, Debug)]
pub struct Message {
pub role: String,
pub content: String,
}
impl Message {
pub fn user(content: impl Into<String>) -> Self {
Message { role: "user".into(), content: content.into() }
}
pub fn assistant(content: impl Into<String>) -> Self {
Message { role: "assistant".into(), content: content.into() }
}
pub fn system(content: impl Into<String>) -> Self {
Message { role: "system".into(), content: content.into() }
}
}
/// Streams a reply from Hermes' OpenAI-compatible API, pushing text deltas
/// into `tx` as they arrive.
///
/// /v1/chat/completions with client-side history, not /v1/responses with
/// previous_response_id: the Responses API keeps the transcript server-side,
/// which is nicer, but this shape is the one every OpenAI client exercises so
/// it is the one least likely to differ between Hermes versions. Her memory
/// and skills are unaffected either way — those are not conversation state.
pub async fn stream_reply(
client: &reqwest::Client,
base_url: &str,
key: &str,
model: &str,
history: &[Message],
tx: mpsc::Sender<String>,
) -> Result<String> {
let body = serde_json::json!({
"model": model,
"messages": history,
"stream": true,
});
let resp = client
.post(format!("{}/v1/chat/completions", base_url.trim_end_matches('/')))
.bearer_auth(key)
.json(&body)
.send()
.await?;
if !resp.status().is_success() {
let status = resp.status();
let text = resp.text().await.unwrap_or_default();
bail!("hermes {status}: {}", text.trim());
}
let mut stream = resp.bytes_stream();
let mut pending = String::new();
let mut full = String::new();
while let Some(chunk) = stream.next().await {
pending.push_str(&String::from_utf8_lossy(&chunk?));
// SSE frames are separated by a blank line; a chunk boundary can land
// mid-frame, so only complete frames are consumed.
while let Some(idx) = pending.find("\n\n") {
let frame: String = pending.drain(..idx + 2).collect();
for line in frame.lines() {
let Some(data) = line.strip_prefix("data:") else { continue };
let data = data.trim();
if data == "[DONE]" {
return Ok(full);
}
if let Some(delta) = parse_delta(data) {
full.push_str(&delta);
// A closed receiver means the reply was cancelled (barge-in).
if tx.send(delta).await.is_err() {
return Ok(full);
}
}
}
}
}
Ok(full)
}
/// Pulls choices[0].delta.content out of one SSE data payload. Tool-call
/// frames and role-only frames carry no content and are skipped.
fn parse_delta(data: &str) -> Option<String> {
let v: serde_json::Value = serde_json::from_str(data).ok()?;
let content = v
.get("choices")?
.get(0)?
.get("delta")?
.get("content")?
.as_str()?;
(!content.is_empty()).then(|| content.to_string())
}
#[cfg(test)]
mod tests {
use super::parse_delta;
#[test]
fn extracts_content_delta() {
let d = r#"{"choices":[{"delta":{"content":"hi"}}]}"#;
assert_eq!(parse_delta(d).as_deref(), Some("hi"));
}
#[test]
fn ignores_role_only_and_tool_frames() {
assert!(parse_delta(r#"{"choices":[{"delta":{"role":"assistant"}}]}"#).is_none());
assert!(parse_delta(r#"{"choices":[{"delta":{"tool_calls":[]}}]}"#).is_none());
assert!(parse_delta(r#"{"choices":[{"delta":{"content":""}}]}"#).is_none());
}
#[test]
fn ignores_malformed_json_rather_than_failing_the_stream() {
assert!(parse_delta("not json").is_none());
assert!(parse_delta("{}").is_none());
}
}
+101
View File
@@ -0,0 +1,101 @@
use anyhow::{Context, Result};
use std::path::PathBuf;
/// Everything the daemon needs, all from the environment so the NixOS module
/// is the only place that knows paths. Secrets arrive as *_FILE (sops paths),
/// never as values — an env var is readable from /proc for anything of the
/// user's, a 0400 file is not.
pub struct Config {
pub hermes_url: String,
pub hermes_key: String,
pub model: String,
pub system_prompt: Option<String>,
/// Prefixed to every spoken message so luna can tell this channel apart
/// from Telegram or the dashboard. Her own profile on mars keys off it —
/// which is why it is a per-message marker and not a system prompt: a
/// system prompt is this client's opinion, while the marker survives into
/// whatever history Hermes keeps and still identifies the channel later.
pub voice_marker: Option<String>,
pub eleven_key: String,
pub voice_id: String,
pub tts_model: String,
pub whisper_bin: String,
pub whisper_model: PathBuf,
pub socket_path: PathBuf,
/// Silence that ends an utterance in hands-free mode. Hermes' own default.
pub silence_ms: u64,
/// Level above the running noise floor that starts one.
pub vad_mult: f32,
/// Speech during playback louder than noise_floor * this counts as barge-in.
pub barge_in_mult: f32,
pub barge_in_grace_ms: u64,
}
fn env(key: &str) -> Result<String> {
std::env::var(key).with_context(|| format!("{key} is not set"))
}
fn env_or(key: &str, default: &str) -> String {
std::env::var(key).unwrap_or_else(|_| default.to_string())
}
/// Reads a secret out of a file and trims it. A trailing newline in a bearer
/// token produces a 401 whose message says nothing about whitespace.
fn secret(key: &str) -> Result<String> {
let path = env(key)?;
let raw = std::fs::read_to_string(&path).with_context(|| format!("reading {path}"))?;
Ok(raw.trim().to_string())
}
impl Config {
pub fn from_env() -> Result<Self> {
let runtime = env_or(
"XDG_RUNTIME_DIR",
&format!("/run/user/{}", unsafe { libc_getuid() }),
);
Ok(Config {
hermes_url: env_or("LUNA_HERMES_URL", "http://mars.orbit.sol:8642"),
hermes_key: secret("LUNA_HERMES_KEY_FILE")?,
model: env_or("LUNA_MODEL", "hermes"),
system_prompt: std::env::var("LUNA_SYSTEM_PROMPT").ok(),
// Empty string disables it rather than sending a bare prefix.
voice_marker: match env_or("LUNA_VOICE_MARKER", "[voice]") {
m if m.is_empty() => None,
m => Some(m),
},
eleven_key: secret("LUNA_ELEVENLABS_KEY_FILE")?,
voice_id: env("LUNA_VOICE_ID")?,
tts_model: env_or("LUNA_TTS_MODEL", "eleven_v3_conversational"),
whisper_bin: env_or("LUNA_WHISPER_BIN", "whisper-cli"),
whisper_model: PathBuf::from(env("LUNA_WHISPER_MODEL")?),
socket_path: PathBuf::from(env_or(
"LUNA_SOCKET",
&format!("{runtime}/luna-voiced.sock"),
)),
silence_ms: env_or("LUNA_SILENCE_MS", "2000").parse().unwrap_or(2000),
// 1.8, not Hermes' 3.0-ish: terra's interface has speech only
// ~6dB over its own ambient (0.0045 against 0.0022 measured), and
// no multiplier above ~2 can separate those. The real fix is gain
// on the interface; this makes it work at the gain it has.
vad_mult: env_or("LUNA_VAD_MULT", "1.8").parse().unwrap_or(1.8),
barge_in_mult: env_or("LUNA_BARGE_IN_MULT", "3.0").parse().unwrap_or(3.0),
barge_in_grace_ms: env_or("LUNA_BARGE_IN_GRACE_MS", "500")
.parse()
.unwrap_or(500),
})
}
}
// Avoids pulling in the libc crate for one call.
unsafe fn libc_getuid() -> u32 {
extern "C" {
fn getuid() -> u32;
}
getuid()
}
+239
View File
@@ -0,0 +1,239 @@
use anyhow::Result;
use serde::Serialize;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::{UnixListener, UnixStream};
use tokio::sync::{broadcast, mpsc};
use crate::visemes::Viseme;
/// Daemon -> shell. Newline-delimited JSON; the shell reads it with a
/// SplitParser and never has to spawn anything.
#[derive(Serialize, Clone, Debug)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum Event {
/// Whether voice mode is on at all. Distinct from `state`: the widget is
/// only on screen while this is true, and every state inside it is a
/// phase of an active conversation.
Mode { active: bool },
State { value: &'static str },
/// Emitted when the first PCM frame reaches the speaker. Viseme offsets
/// are relative to THIS, so the two processes never compare clocks.
SpeechStart,
Visemes { seq: Vec<Viseme> },
Amplitude { rms: f32 },
/// What was said, and by whom. Her side is emitted sentence by sentence
/// as it goes to the synthesiser, so subtitles can lead the audio the
/// same way the visemes do.
Caption { speaker: &'static str, text: String },
SpeechStop { reason: &'static str },
Error { message: String },
}
/// Shell -> daemon.
#[derive(Debug, Clone)]
pub enum Command {
Ptt { down: bool },
Cancel,
/// Run a turn from typed text, skipping the microphone entirely. This is
/// how the brain -> TTS -> speaker half gets tested without speaking, and
/// how you tell a broken mic from a broken pipeline.
Say { text: String },
/// Turn hands-free voice mode on, off, or (None) to the opposite of what
/// it is now — the keybind sends None so one key does both.
Mode { on: Option<bool> },
}
pub struct Ipc {
pub events: broadcast::Sender<Event>,
path: PathBuf,
/// Replayed to every client the moment it connects. The shell can be
/// restarted at any time — including while voice mode is on — and a
/// broadcast-only socket would leave it showing nothing until the next
/// event happened to fire, which in an idle conversation is never.
sticky: Arc<Mutex<Vec<Event>>>,
}
impl Ipc {
/// Binds the socket and serves it until dropped. A stale socket from a
/// crashed run is removed first — bind() would fail with EADDRINUSE on a
/// file nothing is listening to.
pub fn bind(path: &Path, commands: mpsc::Sender<Command>) -> Result<Self> {
if path.exists() {
let _ = std::fs::remove_file(path);
}
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
let listener = UnixListener::bind(path)?;
let (events, _) = broadcast::channel(256);
let sticky: Arc<Mutex<Vec<Event>>> = Arc::new(Mutex::new(Vec::new()));
let tx = events.clone();
let accept_sticky = sticky.clone();
tokio::spawn(async move {
loop {
match listener.accept().await {
Ok((stream, _)) => {
let rx = tx.subscribe();
let cmds = commands.clone();
let snapshot = accept_sticky
.lock()
.map(|s| s.clone())
.unwrap_or_default();
tokio::spawn(client(stream, rx, cmds, snapshot));
}
Err(e) => {
eprintln!("luna-voice: accept failed: {e}");
return;
}
}
}
});
Ok(Ipc { events, path: path.to_path_buf(), sticky })
}
pub fn emit(&self, event: Event) {
// Only the two events that describe a lasting condition are kept;
// replaying a viseme timeline or an old caption to a fresh client
// would animate a conversation that already finished.
if matches!(event, Event::Mode { .. } | Event::State { .. }) {
if let Ok(mut sticky) = self.sticky.lock() {
sticky.retain(|e| std::mem::discriminant(e) != std::mem::discriminant(&event));
sticky.push(event.clone());
}
}
// An error here only means nobody is connected, which is normal.
let _ = self.events.send(event);
}
}
impl Drop for Ipc {
fn drop(&mut self) {
let _ = std::fs::remove_file(&self.path);
}
}
async fn client(
stream: UnixStream,
mut rx: broadcast::Receiver<Event>,
commands: mpsc::Sender<Command>,
snapshot: Vec<Event>,
) {
let (read, mut write) = stream.into_split();
let mut lines = BufReader::new(read).lines();
// Failure here must NOT abandon the connection: the one-shot clients
// (`luna-voice on`, `toggle`, …) write a command and close immediately,
// so this write hits a closed socket and errors — and returning on that
// would drop the command unread, which is exactly what it did once.
for event in snapshot {
let Ok(mut line) = serde_json::to_string(&event) else { continue };
line.push('\n');
if write.write_all(line.as_bytes()).await.is_err() {
break;
}
}
loop {
tokio::select! {
event = rx.recv() => {
let Ok(event) = event else { return };
let Ok(mut line) = serde_json::to_string(&event) else { continue };
line.push('\n');
if write.write_all(line.as_bytes()).await.is_err() {
return;
}
}
line = lines.next_line() => {
match line {
Ok(Some(line)) => {
if let Some(cmd) = parse_command(&line) {
let _ = commands.send(cmd).await;
}
}
_ => return,
}
}
}
}
}
fn parse_command(line: &str) -> Option<Command> {
let v: serde_json::Value = serde_json::from_str(line).ok()?;
match v.get("type")?.as_str()? {
"ptt" => Some(Command::Ptt {
down: v.get("down").and_then(|d| d.as_bool()).unwrap_or(false),
}),
"cancel" => Some(Command::Cancel),
"say" => Some(Command::Say {
text: v.get("text")?.as_str()?.to_string(),
}),
"mode" => Some(Command::Mode {
on: v.get("on").and_then(|o| o.as_bool()),
}),
_ => None,
}
}
/// Client side of the same socket, for the `luna-voice ptt` subcommand that
/// Hyprland binds a key to before the quickshell widget exists.
pub async fn send_command(path: &Path, line: &str) -> Result<()> {
let mut stream = UnixStream::connect(path).await?;
stream.write_all(format!("{line}\n").as_bytes()).await?;
stream.flush().await?;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parses_ptt_commands() {
assert!(matches!(
parse_command(r#"{"type":"ptt","down":true}"#),
Some(Command::Ptt { down: true })
));
assert!(matches!(
parse_command(r#"{"type":"ptt","down":false}"#),
Some(Command::Ptt { down: false })
));
}
#[test]
fn ptt_without_down_defaults_to_release() {
assert!(matches!(
parse_command(r#"{"type":"ptt"}"#),
Some(Command::Ptt { down: false })
));
}
#[test]
fn mode_without_a_value_means_toggle() {
assert!(matches!(
parse_command(r#"{"type":"mode"}"#),
Some(Command::Mode { on: None })
));
assert!(matches!(
parse_command(r#"{"type":"mode","on":true}"#),
Some(Command::Mode { on: Some(true) })
));
}
#[test]
fn ignores_unknown_and_malformed() {
assert!(parse_command(r#"{"type":"nope"}"#).is_none());
assert!(parse_command("garbage").is_none());
}
#[test]
fn events_serialise_with_a_type_tag() {
let json = serde_json::to_string(&Event::State { value: "listening" }).unwrap();
assert_eq!(json, r#"{"type":"state","value":"listening"}"#);
let json = serde_json::to_string(&Event::SpeechStart).unwrap();
assert_eq!(json, r#"{"type":"speech_start"}"#);
}
}
+611
View File
@@ -0,0 +1,611 @@
//! luna-voiced — push-to-talk voice loop for luna (the Hermes Agent on mars).
//!
//! mic -> whisper (local, vulkan) -> Hermes /v1/chat/completions -> ElevenLabs
//! streaming TTS -> speaker, with the speaking state, amplitude and viseme
//! timeline published on a unix socket for the quickshell avatar.
//!
//! Nothing here talks to the compositor: push-to-talk arrives over the same
//! socket, from either `luna-voice ptt` or the shell's GlobalShortcut.
mod audio;
mod brain;
mod config;
mod ipc;
mod stt;
mod tts;
mod visemes;
use anyhow::Result;
use serde_json;
use config::Config;
use ipc::{Command, Event};
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::mpsc;
#[tokio::main]
async fn main() -> Result<()> {
let mut args = std::env::args().skip(1);
match args.next().as_deref() {
None | Some("daemon") => daemon().await,
// Fire-and-forget clients so a keybind costs one short-lived process
// and never blocks the compositor.
Some("ptt-down") => client(r#"{"type":"ptt","down":true}"#).await,
Some("ptt-up") => client(r#"{"type":"ptt","down":false}"#).await,
Some("cancel") => client(r#"{"type":"cancel"}"#).await,
Some("toggle") => client(r#"{"type":"mode"}"#).await,
Some("on") => client(r#"{"type":"mode","on":true}"#).await,
Some("off") => client(r#"{"type":"mode","on":false}"#).await,
// The debugging tool: everything the avatar will eventually see,
// printed. `journalctl -fu luna-voiced` shows only what the daemon
// chose to log; this shows the actual protocol.
Some("watch") => watch().await,
// Opens the mic directly, bypassing the daemon: answers "is the
// microphone working at all" without any of the rest being involved.
Some("level") => level().await,
Some("devices") => {
for name in audio::list_inputs()? {
println!("{name}");
}
Ok(())
}
Some("say") => {
let text: Vec<String> = args.collect();
if text.is_empty() {
eprintln!("luna-voice: say what?");
std::process::exit(2);
}
let payload = serde_json::json!({ "type": "say", "text": text.join(" ") });
client(&payload.to_string()).await
}
Some(other) => {
eprintln!("luna-voice: unknown command {other:?}");
eprintln!(
"usage: luna-voice [daemon|toggle|on|off|say <text>|watch|level|devices|cancel]"
);
std::process::exit(2);
}
}
}
/// Level that counts as speech.
///
/// The lower clamp is the number that matters, and it has to sit under the
/// QUIETEST mic worth supporting, not under a typical one: terra's AIR 192|4
/// reads ~0.001 idle and only 0.004–0.005 for normal speech, so a 0.006 floor
/// — which looked conservative — put the trigger permanently above the user's
/// voice. Overshooting downward only costs silent false triggers.
fn root_threshold(floor: f32, mult: f32) -> f32 {
let min = std::env::var("LUNA_VAD_FLOOR")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(0.0025);
(floor * mult).clamp(min, 0.05)
}
/// Tags a spoken line so luna can recognise the channel it came from.
///
/// Prefix rather than suffix: a long utterance can be truncated by whatever
/// history window she is running under, and the front is the end that
/// survives.
fn mark(marker: &Option<String>, text: &str) -> String {
match marker {
Some(m) => format!("{m} {text}"),
None => text.to_string(),
}
}
fn socket_path() -> std::path::PathBuf {
let runtime = std::env::var("XDG_RUNTIME_DIR").unwrap_or_else(|_| "/run/user/1000".into());
std::env::var("LUNA_SOCKET")
.map(std::path::PathBuf::from)
.unwrap_or_else(|_| std::path::Path::new(&runtime).join("luna-voiced.sock"))
}
async fn watch() -> Result<()> {
use tokio::io::{AsyncBufReadExt, BufReader};
let path = socket_path();
let stream = tokio::net::UnixStream::connect(&path)
.await
.map_err(|e| anyhow::anyhow!("{}: {e} (is luna-voiced running?)", path.display()))?;
let mut lines = BufReader::new(stream).lines();
eprintln!("luna-voice: watching {}", path.display());
while let Some(line) = lines.next_line().await? {
println!("{line}");
}
Ok(())
}
/// Live microphone meter with the trigger point marked, so a mic that reads
/// nothing and a threshold that is set too high look different.
async fn level() -> Result<()> {
let capture = audio::Capture::new()?;
let mult: f32 = std::env::var("LUNA_VAD_MULT")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(1.8);
let mut floor: f32 = capture.level().max(0.003);
eprintln!("luna-voice: speak — ctrl-c to stop");
loop {
let level = capture.level();
floor = floor * 0.95 + level * 0.05;
let threshold = root_threshold(floor, mult);
// Full scale 0.02, not 0.05: a quiet interface tops out around
// 0.005 and was drawing a single bar on the wider scale.
const FULL: f32 = 0.02;
let bars = ((level / FULL).min(1.0) * 40.0) as usize;
let mark = ((threshold / FULL).min(1.0) * 40.0) as usize;
let meter: String = (0..40)
.map(|i| {
if i == mark {
'|'
} else if i < bars {
'#'
} else {
'.'
}
})
.collect();
print!(
"\r{meter} rms {level:.4} floor {floor:.4} trigger {threshold:.4} {} ",
if level > threshold { "SPEECH" } else { " " }
);
use std::io::Write;
let _ = std::io::stdout().flush();
tokio::time::sleep(Duration::from_millis(50)).await;
}
}
async fn client(line: &str) -> Result<()> {
ipc::send_command(&socket_path(), line).await
}
/// Errors go to the desktop, not just the journal — terra runs its own
/// notification daemon (HyprChrome/Widgets/Notifications), so this is the one
/// channel that is visible while you are talking rather than debugging.
/// Best-effort: a missing notify-send must never take the daemon down.
fn notify(body: &str) {
let _ = std::process::Command::new("notify-send")
.args(["-a", "luna", "-u", "critical", "luna voice", body])
.spawn();
}
/// Where the loop is. One utterance at a time by construction: the state
/// machine is a single task, so there is no way to start a second reply while
/// one is in flight.
async fn daemon() -> Result<()> {
let cfg = Arc::new(Config::from_env()?);
let capture = audio::Capture::new()?;
let playback = Arc::new(audio::Playback::new()?);
let (cmd_tx, mut cmd_rx) = mpsc::channel::<Command>(16);
// Kept so a turn can put back commands that arrived while she was
// speaking instead of destroying them.
let requeue = cmd_tx.clone();
let ipc = Arc::new(ipc::Ipc::bind(&cfg.socket_path, cmd_tx)?);
eprintln!("luna-voice: listening on {}", cfg.socket_path.display());
let http = reqwest::Client::builder()
// Her replies can involve tool calls; the default 30s would cut those
// off mid-thought.
.timeout(Duration::from_secs(300))
.build()?;
let mut history: Vec<brain::Message> = Vec::new();
if let Some(prompt) = &cfg.system_prompt {
history.push(brain::Message::system(prompt.clone()));
}
ipc.emit(Event::State { value: "idle" });
// Her first reply after the container starts costs ~16s against ~5s warm,
// and the first one is always the one you are sitting there waiting for.
// Spending one throwaway token at login moves that cost off the
// conversation. Failure is ignored on purpose: mars may simply be down,
// which must not stop the daemon from coming up.
tokio::spawn({
let (http, url, key, model) =
(http.clone(), cfg.hermes_url.clone(), cfg.hermes_key.clone(), cfg.model.clone());
async move {
let (tx, mut rx) = mpsc::channel(4);
tokio::spawn(async move { while rx.recv().await.is_some() {} });
let warm = [brain::Message::user("hi")];
match brain::stream_reply(&http, &url, &key, &model, &warm, tx).await {
Ok(_) => eprintln!("luna-voice: warmed up"),
Err(e) => eprintln!("luna-voice: warmup skipped: {e}"),
}
}
});
// Hands-free state. The key toggles `mode`; everything inside it is
// driven by the microphone rather than by keys.
let mut mode = false;
// Running estimate of the room, updated only while not capturing, so a
// loud sentence never raises the bar for the next one.
let mut floor: f32 = 0.01;
let mut capturing = false;
let mut last_voice = tokio::time::Instant::now();
let mut ticker = tokio::time::interval(Duration::from_millis(50));
loop {
let cmd = tokio::select! {
cmd = cmd_rx.recv() => match cmd {
Some(cmd) => Some(cmd),
None => break,
},
// Only ticks while voice mode is on. A turn is awaited inline
// below, so this cannot fire while she is speaking — which is
// also what stops her from hearing herself.
_ = ticker.tick(), if mode => None,
};
let Some(cmd) = cmd else {
let level = capture.level();
// Every tick, not just while she speaks: a meter that only moves
// during playback cannot tell "the mic is dead" apart from "the
// threshold is too high", which is exactly the question you have
// when nothing happens.
ipc.emit(Event::Amplitude { rms: level });
let threshold = root_threshold(floor, cfg.vad_mult);
if !capturing {
// Only samples BELOW the trigger update the floor. Feeding
// every sample in lets a rising sound drag the floor up with
// it, so the threshold outruns the speech that caused it and
// nothing ever fires — measured: a tone peaking at 0.029 was
// chased by a threshold that reached 0.018 on the way up.
if level <= threshold {
floor = floor * 0.95 + level * 0.05;
}
if level > threshold {
capture.start();
capturing = true;
last_voice = tokio::time::Instant::now();
ipc.emit(Event::State { value: "listening" });
}
} else {
if level > threshold {
last_voice = tokio::time::Instant::now();
}
if last_voice.elapsed() > Duration::from_millis(cfg.silence_ms) {
capturing = false;
let samples = capture.stop();
ipc.emit(Event::State { value: "thinking" });
match stt::transcribe(&cfg.whisper_bin, &cfg.whisper_model, &samples).await {
Ok(heard) if !heard.is_empty() => {
run_turn(
&cfg, &http, &ipc, &capture, &playback, &mut cmd_rx,
&requeue, &mut history, heard,
)
.await;
}
Ok(_) => {
// Silent on purpose: in hands-free mode a false
// trigger happens whenever a door closes, and a
// beep for each one would be maddening.
ipc.emit(Event::State { value: "idle" });
}
Err(e) => {
eprintln!("luna-voice: {e:#}");
ipc.emit(Event::Error { message: e.to_string() });
notify(&e.to_string());
ipc.emit(Event::State { value: "idle" });
}
}
}
}
continue;
};
match cmd {
Command::Mode { on } => {
let want = on.unwrap_or(!mode);
if want != mode {
mode = want;
if mode {
// Seeded low: a floor seeded high takes seconds to
// decay, and those are the seconds you spend
// talking to something that is not listening yet.
floor = capture.level().max(0.003);
playback.beep(880.0, 70);
} else {
if capturing {
capture.stop();
capturing = false;
}
playback.clear();
playback.beep(440.0, 90);
}
ipc.emit(Event::Mode { active: mode });
ipc.emit(Event::State { value: "idle" });
eprintln!("luna-voice: voice mode {}", if mode { "on" } else { "off" });
}
}
Command::Ptt { down: true } => {
// Anything still playing is cut off: pressing the key while
// she talks means you want to talk instead.
playback.clear();
capture.start();
playback.beep(880.0, 70);
ipc.emit(Event::State { value: "listening" });
}
Command::Ptt { down: false } => {
let samples = capture.stop();
playback.beep(587.0, 70);
ipc.emit(Event::State { value: "thinking" });
let heard =
match stt::transcribe(&cfg.whisper_bin, &cfg.whisper_model, &samples).await {
Ok(h) => h,
Err(e) => {
eprintln!("luna-voice: {e:#}");
ipc.emit(Event::Error { message: e.to_string() });
notify(&e.to_string());
ipc.emit(Event::State { value: "idle" });
continue;
}
};
if heard.is_empty() {
// Distinct low tone: "I heard nothing" and "I am
// thinking" must not feel the same, or a dead mic reads
// as a slow reply.
playback.beep(220.0, 120);
eprintln!("luna-voice: nothing heard ({} samples)", samples.len());
ipc.emit(Event::SpeechStop { reason: "nothing_heard" });
ipc.emit(Event::State { value: "idle" });
continue;
}
run_turn(&cfg, &http, &ipc, &capture, &playback, &mut cmd_rx, &requeue, &mut history, heard)
.await;
}
Command::Say { text } => {
ipc.emit(Event::State { value: "thinking" });
run_turn(&cfg, &http, &ipc, &capture, &playback, &mut cmd_rx, &requeue, &mut history, text)
.await;
}
Command::Cancel => {
playback.clear();
ipc.emit(Event::SpeechStop { reason: "cancelled" });
ipc.emit(Event::State { value: "idle" });
}
}
}
Ok(())
}
/// Runs one turn and reports failure through every channel that has a
/// listener: journal, socket, desktop notification.
#[cfg(test)]
mod tests {
use super::{mark, root_threshold};
#[test]
fn marker_prefixes_spoken_lines() {
assert_eq!(
mark(&Some("[voice]".into()), "what's the time"),
"[voice] what's the time"
);
}
#[test]
fn no_marker_leaves_the_line_alone() {
assert_eq!(mark(&None, "what's the time"), "what's the time");
}
#[test]
fn threshold_sits_between_terras_room_tone_and_terras_speech() {
// Measured on the AIR 192|4: ~0.001 idle, 0.004–0.005 speaking. The
// trigger has to land strictly between those two, which is the whole
// reason the lower clamp is 0.0025 and not something rounder.
let t = root_threshold(0.001, 3.5);
assert!(t > 0.001, "would trigger on room tone: {t}");
assert!(t < 0.004, "would never trigger on speech: {t}");
}
#[test]
fn threshold_never_climbs_out_of_reach_of_speech() {
// A floor seeded from a loud moment must not lock the mic out.
assert_eq!(root_threshold(0.5, 3.5), 0.05);
}
#[test]
fn threshold_tracks_the_room_in_between() {
assert!((root_threshold(0.01, 3.5) - 0.035).abs() < 1e-6);
}
}
#[allow(clippy::too_many_arguments)]
async fn run_turn(
cfg: &Config,
http: &reqwest::Client,
ipc: &Arc<ipc::Ipc>,
capture: &audio::Capture,
playback: &Arc<audio::Playback>,
cmd_rx: &mut mpsc::Receiver<Command>,
requeue: &mpsc::Sender<Command>,
history: &mut Vec<brain::Message>,
heard: String,
) {
if let Err(e) = turn(cfg, http, ipc, capture, playback, cmd_rx, requeue, history, heard).await {
eprintln!("luna-voice: {e:#}");
ipc.emit(Event::Error { message: e.to_string() });
notify(&e.to_string());
}
ipc.emit(Event::State { value: "idle" });
}
#[allow(clippy::too_many_arguments)]
async fn turn(
cfg: &Config,
http: &reqwest::Client,
ipc: &Arc<ipc::Ipc>,
capture: &audio::Capture,
playback: &Arc<audio::Playback>,
cmd_rx: &mut mpsc::Receiver<Command>,
requeue: &mpsc::Sender<Command>,
history: &mut Vec<brain::Message>,
heard: String,
) -> Result<()> {
eprintln!("luna-voice: heard {heard:?}");
// The caption shows what you SAID; the marker is only for her.
ipc.emit(Event::Caption { speaker: "user", text: heard.clone() });
history.push(brain::Message::user(mark(&cfg.voice_marker, &heard)));
// Commands that arrived while she was speaking, replayed into the
// channel once the turn is over rather than dropped.
let mut deferred: Vec<Command> = Vec::new();
// Noise floor sampled before she starts, so barge-in adapts to the room
// rather than to a constant.
let floor = capture.level().max(0.005);
let (sentence_tx, sentence_rx) = mpsc::channel::<String>(16);
let (event_tx, mut event_rx) = mpsc::channel::<tts::SpeechEvent>(64);
let tts_cfg = tts::TtsConfig {
key: cfg.eleven_key.clone(),
voice_id: cfg.voice_id.clone(),
model: cfg.tts_model.clone(),
};
let keep_tags = tts_cfg.is_v3();
let speech = tokio::spawn({
async move { tts::speak(&tts_cfg, sentence_rx, event_tx).await }
});
// Audio and visemes flow to the speaker/shell on their own task so the
// reply stream is never blocked by playback.
let player = tokio::spawn({
let ipc = ipc.clone();
let playback = playback.clone();
async move {
let mut started = false;
while let Some(ev) = event_rx.recv().await {
match ev {
tts::SpeechEvent::Audio(pcm) => {
if !started {
started = true;
ipc.emit(Event::SpeechStart);
ipc.emit(Event::State { value: "speaking" });
}
playback.push(&pcm);
}
// A chunk whose alignment held nothing visible still
// arrives; forwarding it would make the shell reset the
// mouth mid-word.
tts::SpeechEvent::Visemes(seq) if !seq.is_empty() => {
ipc.emit(Event::Visemes { seq })
}
tts::SpeechEvent::Visemes(_) => {}
tts::SpeechEvent::Done => break,
}
}
}
});
let (delta_tx, mut delta_rx) = mpsc::channel::<String>(64);
let reply = tokio::spawn({
let http = http.clone();
let (url, key, model) = (cfg.hermes_url.clone(), cfg.hermes_key.clone(), cfg.model.clone());
let history = history.clone();
async move { brain::stream_reply(&http, &url, &key, &model, &history, delta_tx).await }
});
let mut splitter = tts::SentenceSplitter::new(keep_tags);
// What ElevenLabs bills: every character sent, audio tags included.
let mut billed = 0usize;
let caption = |sentence: &str| {
let text = tts::strip_tags(sentence);
if !text.is_empty() {
ipc.emit(Event::Caption { speaker: "luna", text });
}
};
while let Some(delta) = delta_rx.recv().await {
for sentence in splitter.push(&delta) {
caption(&sentence);
billed += sentence.chars().count();
if sentence_tx.send(sentence).await.is_err() {
break;
}
}
}
if let Some(tail) = splitter.finish() {
caption(&tail);
billed += tail.chars().count();
let _ = sentence_tx.send(tail).await;
}
drop(sentence_tx);
eprintln!("luna-voice: tts {billed} chars on {}", cfg.tts_model);
let full = reply.await??;
eprintln!("luna-voice: said {:?}", full.trim());
if !full.trim().is_empty() {
history.push(brain::Message::assistant(full));
}
// Wait for the speaker to drain, watching for barge-in. Polling the queue
// is enough: cpal gives no drain callback, and 50ms of latency on "she
// finished" is invisible.
let grace = tokio::time::Instant::now() + Duration::from_millis(cfg.barge_in_grace_ms);
loop {
if speech.is_finished() && playback.pending() == 0 {
break;
}
if tokio::time::Instant::now() > grace
&& capture.level() > floor * cfg.barge_in_mult
{
playback.clear();
ipc.emit(Event::SpeechStop { reason: "barge_in" });
// She is told she was cut off, so the next turn can react to it
// rather than carrying on obliviously.
history.push(brain::Message::system(
"[The user interrupted you mid-sentence.]",
));
break;
}
// Commands arriving mid-playback. try_recv POPS, so anything not
// handled here is destroyed: an `if let Ok(Cancel)` silently ate
// every `say`, `toggle` and `mode off` sent while she was talking,
// which presents as a keypress that does nothing.
match cmd_rx.try_recv() {
Ok(Command::Cancel) => {
playback.clear();
ipc.emit(Event::SpeechStop { reason: "cancelled" });
break;
}
// Leaving voice mode has to shut her up immediately; the mode
// change itself is replayed below.
Ok(cmd @ Command::Mode { .. }) => {
playback.clear();
ipc.emit(Event::SpeechStop { reason: "cancelled" });
deferred.push(cmd);
break;
}
Ok(cmd) => deferred.push(cmd),
Err(_) => {}
}
ipc.emit(Event::Amplitude { rms: capture.level() });
tokio::time::sleep(Duration::from_millis(50)).await;
}
for cmd in deferred {
let _ = requeue.try_send(cmd);
}
// A TTS failure must NOT be silent. The captions come from the text
// stream, not from the synthesiser, so a dead websocket still paints her
// whole reply on screen — "she answered but said nothing" is then
// indistinguishable from a working reply with the volume down, which is
// exactly how it presented.
if speech.is_finished() {
if let Ok(Err(e)) = speech.await {
eprintln!("luna-voice: tts failed: {e:#}");
ipc.emit(Event::Error { message: format!("tts: {e}") });
notify(&format!("voice failed: {e}"));
}
} else {
speech.abort();
}
player.abort();
Ok(())
}
+181
View File
@@ -0,0 +1,181 @@
use crate::audio;
use anyhow::{bail, Result};
use std::path::Path;
use tokio::process::Command;
/// Shells out to whisper-cli rather than linking whisper.cpp: nixpkgs already
/// ships whisper-cpp-vulkan built against this box's GPU stack, and a bundled
/// build would have to rediscover that at cargo time.
pub async fn transcribe(bin: &str, model: &Path, samples: &[f32]) -> Result<String> {
// Whisper hallucinates confident words into silence — "you", "Thank
// you.", "Bye." — which then reach luna as if spoken. Duration alone does
// not catch it (a held key over a quiet room is long AND empty), so gate
// on level too.
if samples.len() < (audio::STT_RATE as usize) / 3 || !has_speech(samples) {
return Ok(String::new());
}
let dir = std::env::var("XDG_RUNTIME_DIR").unwrap_or_else(|_| "/tmp".into());
let wav = Path::new(&dir).join(format!("luna-stt-{}.wav", std::process::id()));
audio::write_wav(&wav, &normalize(samples), audio::STT_RATE)?;
let out = Command::new(bin)
.args(["-m"])
.arg(model)
.args(["-f"])
.arg(&wav)
// -nt: transcript only, no [00:00:00] prefixes. -np: no progress
// banner. Together they make stdout exactly the text.
.args(["-nt", "-np"])
.output()
.await?;
let _ = std::fs::remove_file(&wav);
if !out.status.success() {
bail!(
"whisper-cli failed: {}",
String::from_utf8_lossy(&out.stderr).trim()
);
}
Ok(clean(&String::from_utf8_lossy(&out.stdout)))
}
/// Scales a quiet recording up before whisper sees it.
///
/// Whisper is trained on normalised audio and degrades badly on very quiet
/// input, which is precisely what a low-gain interface produces — speech at a
/// 0.03 peak is intelligible to a person and mush to the model. The gain is
/// capped so a recording of near-silence is not amplified into hiss the model
/// will confidently transcribe as words.
fn normalize(samples: &[f32]) -> Vec<f32> {
const TARGET_PEAK: f32 = 0.6;
const MAX_GAIN: f32 = 30.0;
let peak = samples.iter().fold(0.0f32, |m, s| m.max(s.abs()));
if peak <= f32::EPSILON {
return samples.to_vec();
}
let gain = (TARGET_PEAK / peak).min(MAX_GAIN);
// Never attenuate: a loud recording is already fine, and turning it down
// would only lose resolution in the 16-bit conversion.
if gain <= 1.0 {
return samples.to_vec();
}
samples.iter().map(|s| (s * gain).clamp(-1.0, 1.0)).collect()
}
/// True if any 100ms window is loud enough to be a voice. Peak rather than
/// overall RMS: a short sentence inside a long silent recording averages down
/// to nothing, and that is the common case for a held key.
fn has_speech(samples: &[f32]) -> bool {
// Matches the VAD's lower clamp rather than a "typical" mic: a quiet
// interface puts normal speech at ~0.004, and gating at 0.015 threw away
// every word it captured.
const FLOOR: f32 = 0.0025;
let window = (audio::STT_RATE as usize) / 10;
samples.chunks(window).any(|w| {
let sum: f32 = w.iter().map(|s| s * s).sum();
(sum / w.len().max(1) as f32).sqrt() > FLOOR
})
}
/// whisper emits bracketed non-speech annotations ([BLANK_AUDIO], (wind
/// blowing)) that are not words and must not reach the agent.
fn clean(raw: &str) -> String {
let mut out = String::new();
let mut depth = 0i32;
for c in raw.chars() {
match c {
'[' | '(' => depth += 1,
']' | ')' => depth = (depth - 1).max(0),
c if depth == 0 => out.push(c),
_ => {}
}
}
out.split_whitespace().collect::<Vec<_>>().join(" ")
}
#[cfg(test)]
mod tests {
use super::{clean, has_speech, normalize};
use crate::audio::STT_RATE;
fn peak(v: &[f32]) -> f32 {
v.iter().fold(0.0f32, |m, s| m.max(s.abs()))
}
#[test]
fn normalize_lifts_a_quiet_recording() {
// 0.004 RMS speech peaks around 0.02, which is 30x under target —
// exactly at the gain cap, so it just reaches full scale.
let quiet: Vec<f32> = (0..1000).map(|i| 0.02 * ((i % 7) as f32 - 3.0) / 3.0).collect();
assert!((peak(&normalize(&quiet)) - 0.6).abs() < 0.01);
}
#[test]
fn normalize_leaves_a_loud_recording_alone() {
let loud: Vec<f32> = vec![0.9, -0.8, 0.7];
assert_eq!(normalize(&loud), loud);
}
#[test]
fn normalize_caps_the_gain_so_silence_stays_silent() {
let hiss: Vec<f32> = vec![0.0001, -0.0001];
// 0.6/0.0001 would be 6000x; the cap keeps it at 30x.
assert!(peak(&normalize(&hiss)) <= 0.0001 * 30.0 + 1e-9);
}
#[test]
fn normalize_handles_pure_silence() {
assert_eq!(normalize(&[0.0, 0.0]), vec![0.0, 0.0]);
}
#[test]
fn silence_is_not_speech() {
assert!(!has_speech(&vec![0.0f32; STT_RATE as usize]));
// Terra's measured room tone. Speech on the same mic reaches
// 0.004–0.005, so the gate sits between the two.
assert!(!has_speech(&vec![0.001f32; STT_RATE as usize]));
}
#[test]
fn quiet_speech_still_counts() {
// 0.004 is what normal speech measures on terra's interface; gating
// it out discarded every word the VAD had just captured.
let mut s = vec![0.001f32; STT_RATE as usize];
for v in s.iter_mut().take(STT_RATE as usize / 5) {
*v = 0.004;
}
assert!(has_speech(&s));
}
#[test]
fn a_short_utterance_in_a_long_silence_counts() {
let mut s = vec![0.0f32; STT_RATE as usize];
for v in s.iter_mut().take(STT_RATE as usize / 5) {
*v = 0.2;
}
assert!(has_speech(&s));
}
#[test]
fn strips_bracketed_annotations() {
assert_eq!(clean("[BLANK_AUDIO]"), "");
assert_eq!(clean(" hello (wind blowing) there \n"), "hello there");
assert_eq!(clean("[ music ] real words"), "real words");
}
#[test]
fn collapses_whitespace() {
assert_eq!(clean(" a b \n c "), "a b c");
}
#[test]
fn unbalanced_brackets_do_not_eat_the_rest() {
// The stray closer is dropped (depth floors at 0) but the words
// around it survive — the failure mode that matters is a lone
// bracket swallowing the whole transcript.
assert_eq!(clean("ok] fine"), "ok fine");
assert_eq!(clean("[ oops real words"), "");
}
}
+529
View File
@@ -0,0 +1,529 @@
use crate::audio;
use crate::visemes::{self, Viseme};
use anyhow::{anyhow, Result};
use base64::Engine;
use futures_util::{SinkExt, StreamExt};
use std::time::Duration;
use tokio::sync::mpsc;
use tokio_tungstenite::tungstenite::client::IntoClientRequest;
use tokio_tungstenite::tungstenite::Message as Ws;
pub enum SpeechEvent {
/// Decoded PCM at audio::TTS_RATE.
Audio(Vec<f32>),
/// Mouth shapes, timed from the start of this reply's audio.
Visemes(Vec<Viseme>),
Done,
}
pub struct TtsConfig {
pub key: String,
pub voice_id: String,
pub model: String,
}
impl TtsConfig {
/// v3 is not served on the TTS websocket at all; it streams only over
/// Text to Dialogue, a different protocol.
pub fn is_v3(&self) -> bool {
self.model.starts_with("eleven_v3")
}
}
/// The two sockets send the same audio frames under different field names.
struct Wire {
alignment: &'static [&'static str],
final_key: &'static str,
}
const TTS_WIRE: Wire = Wire { alignment: &["normalizedAlignment", "alignment"], final_key: "isFinal" };
const DIALOGUE_WIRE: Wire = Wire { alignment: &["alignment"], final_key: "is_final" };
/// Dialogue drops the socket after 20s without a client message, and a tool
/// call on mars can stall a reply longer than that.
const KEEP_ALIVE: Duration = Duration::from_secs(10);
type Socket = tokio_tungstenite::WebSocketStream<
tokio_tungstenite::MaybeTlsStream<tokio::net::TcpStream>,
>;
type Sink = futures_util::stream::SplitSink<Socket, Ws>;
type Source = futures_util::stream::SplitStream<Socket>;
/// Speaks sentences as they arrive on `sentences`, over one websocket per
/// reply.
///
/// pcm_24000 rather than the default mp3: there is no decoder in this process
/// and no reason to add one. sync_alignment makes the character timings ride
/// along with the audio frames instead of needing a second request.
pub async fn speak(
cfg: &TtsConfig,
sentences: mpsc::Receiver<String>,
events: mpsc::Sender<SpeechEvent>,
) -> Result<()> {
if cfg.is_v3() {
speak_dialogue(cfg, sentences, events).await
} else {
speak_tts(cfg, sentences, events).await
}
}
async fn connect(url: &str, key: &str) -> Result<(Sink, Source)> {
let mut req = url.into_client_request()?;
// Header, not the ?authorization= query param: a key in a URL ends up in
// every proxy and error log that touches it.
req.headers_mut().insert(
"xi-api-key",
tokio_tungstenite::tungstenite::http::HeaderValue::from_str(key)?,
);
let (stream, _) = tokio_tungstenite::connect_async(req).await?;
Ok(stream.split())
}
async fn send_json(sink: &mut Sink, v: serde_json::Value) -> Result<()> {
sink.send(Ws::Text(v.to_string())).await?;
Ok(())
}
async fn speak_tts(
cfg: &TtsConfig,
mut sentences: mpsc::Receiver<String>,
events: mpsc::Sender<SpeechEvent>,
) -> Result<()> {
let url = format!(
"wss://api.elevenlabs.io/v1/text-to-speech/{}/stream-input\
?model_id={}&output_format=pcm_{}&sync_alignment=true&auto_mode=true",
cfg.voice_id,
cfg.model,
audio::TTS_RATE
);
let (mut sink, source) = connect(&url, &cfg.key).await?;
// The protocol requires an opening frame of a single space before any
// real text; it carries the voice settings for the whole connection.
send_json(
&mut sink,
serde_json::json!({
"text": " ",
"voice_settings": { "stability": 0.4, "similarity_boost": 0.8, "speed": 1.0 },
}),
)
.await?;
let reader = tokio::spawn(read_audio(source, events, &TTS_WIRE));
let sent: Result<()> = async {
while let Some(text) = sentences.recv().await {
// Every text frame must end with a space or the model holds the
// last word back waiting for more input.
let text = format!("{} ", text.trim());
send_json(&mut sink, serde_json::json!({ "text": text })).await?;
}
// Empty string = end of input. Without it the socket sits until the
// inactivity timeout and the last sentence never gets flushed.
send_json(&mut sink, serde_json::json!({ "text": "" })).await
}
.await;
// The server's own error beats the "connection reset" a send hit after it.
reader.await??;
sent
}
async fn speak_dialogue(
cfg: &TtsConfig,
mut sentences: mpsc::Receiver<String>,
events: mpsc::Sender<SpeechEvent>,
) -> Result<()> {
let url = format!(
"wss://api.elevenlabs.io/v1/text-to-dialogue/stream-input\
?model_id={}&output_format=pcm_{}&sync_alignment=true",
cfg.model,
audio::TTS_RATE
);
let (mut sink, source) = connect(&url, &cfg.key).await?;
// No voice_settings: v3 reads only stability, and only as one of three
// presets; the default is its "natural" one.
send_json(&mut sink, serde_json::json!({ "voices": [cfg.voice_id] })).await?;
let reader = tokio::spawn(read_audio(source, events, &DIALOGUE_WIRE));
let sent: Result<()> = async {
loop {
match tokio::time::timeout(KEEP_ALIVE, sentences.recv()).await {
Ok(Some(text)) => {
// Inputs are concatenated verbatim, so the space matters
// here too. flush: the server otherwise holds text until
// ~40 chars and 8 words, so a short first sentence would
// wait for the second.
let text = format!("{} ", text.trim());
send_json(
&mut sink,
serde_json::json!({
"inputs": [{ "text": text, "voice_id": cfg.voice_id }],
"flush": true,
}),
)
.await?
}
Ok(None) => break,
Err(_) => send_json(&mut sink, serde_json::json!({ "keep_alive": true })).await?,
}
}
send_json(&mut sink, serde_json::json!({ "close_socket": true })).await
}
.await;
reader.await??;
sent
}
async fn read_audio(
mut source: Source,
events: mpsc::Sender<SpeechEvent>,
wire: &'static Wire,
) -> Result<()> {
// Alignment times may be per-chunk or cumulative depending on the
// model; `emitted_ms` lets us normalise either into one timeline.
let mut emitted_ms: u32 = 0;
// v3 audio tags come back as characters, and one can straddle chunks.
let mut in_tag = false;
let mut result = Ok(());
while let Some(Ok(msg)) = source.next().await {
let Ws::Text(txt) = msg else { continue };
let Ok(v) = serde_json::from_str::<serde_json::Value>(&txt) else { continue };
if let Some(err) = v.get("error").and_then(|e| e.as_str()) {
let detail = v.get("message").and_then(|m| m.as_str()).unwrap_or("");
result = Err(anyhow!("elevenlabs: {err} {detail}"));
break;
}
if let Some(b64) = v.get("audio").and_then(|a| a.as_str()) {
if let Ok(bytes) = base64::engine::general_purpose::STANDARD.decode(b64) {
let pcm = audio::pcm16_to_f32(&bytes);
let chunk_ms = (pcm.len() as f64 / audio::TTS_RATE as f64 * 1000.0) as u32;
let al = wire.alignment.iter().find_map(|k| v.get(*k).filter(|a| !a.is_null()));
if let Some(t) = al.and_then(|al| alignment_to_timeline(al, emitted_ms, &mut in_tag)) {
let _ = events.send(SpeechEvent::Visemes(t)).await;
}
emitted_ms += chunk_ms;
if events.send(SpeechEvent::Audio(pcm)).await.is_err() {
return Ok(()); // receiver gone: barge-in cancelled this reply
}
}
}
if v.get(wire.final_key).and_then(|f| f.as_bool()).unwrap_or(false) {
break;
}
}
let _ = events.send(SpeechEvent::Done).await;
result
}
fn alignment_to_timeline(
al: &serde_json::Value,
emitted_ms: u32,
in_tag: &mut bool,
) -> Option<Vec<Viseme>> {
let field = |camel: &str, snake: &str| al.get(camel).or_else(|| al.get(snake));
let chars: Vec<String> = al
.get("chars")?
.as_array()?
.iter()
.map(|c| c.as_str().unwrap_or(""))
.map(|c| {
// A tag is a laugh or a sigh, not letters: the mouth rests.
let tagged = *in_tag || c == "[";
*in_tag = tagged && c != "]";
if tagged { " ".to_string() } else { c.to_string() }
})
.collect();
let starts: Vec<i64> = num_array(field("charStartTimesMs", "char_start_times_ms")?)?;
let durs: Vec<i64> = num_array(field("charDurationsMs", "char_durations_ms")?)?;
let starts = absolutize(&starts, emitted_ms);
Some(visemes::timeline(&chars, &starts, &durs))
}
fn num_array(v: &serde_json::Value) -> Option<Vec<i64>> {
Some(v.as_array()?.iter().map(|n| n.as_i64().unwrap_or(0)).collect())
}
/// Normalises chunk-relative and stream-absolute alignments into one
/// timeline. If the first start time is behind audio we have already emitted,
/// the timings must be relative to this chunk, so they get shifted.
fn absolutize(starts: &[i64], emitted_ms: u32) -> Vec<i64> {
let offset = match starts.first() {
Some(&first) if first < emitted_ms as i64 => emitted_ms as i64,
_ => 0,
};
starts.iter().map(|s| s + offset).collect()
}
/// Splits streamed text into speakable sentences, and strips what should
/// never be read aloud. Hermes' own voice mode does the same thing for the
/// same reason: without it she pronounces asterisks.
pub struct SentenceSplitter {
buf: String,
in_think: bool,
/// Whether v3 audio tags survive to the synthesiser or are dropped.
keep_tags: bool,
}
impl SentenceSplitter {
pub fn new(keep_tags: bool) -> Self {
SentenceSplitter { buf: String::new(), in_think: false, keep_tags }
}
/// Feeds a delta, returning any sentences that completed.
pub fn push(&mut self, delta: &str) -> Vec<String> {
self.buf.push_str(delta);
let mut out = Vec::new();
// <think> blocks are reasoning, not speech, and can be long.
loop {
if self.in_think {
match self.buf.find("</think>") {
Some(i) => {
self.buf.drain(..i + "</think>".len());
self.in_think = false;
}
None => {
self.buf.clear();
return out;
}
}
}
match self.buf.find("<think>") {
Some(i) => {
let head: String = self.buf.drain(..i).collect();
self.buf.drain(.."<think>".len());
self.in_think = true;
out.extend(drain_sentences(&mut { head.clone() }, true, self.keep_tags));
}
None => break,
}
}
out.extend(drain_sentences(&mut self.buf, false, self.keep_tags));
out.into_iter().filter(|s| !s.is_empty()).collect()
}
/// Whatever is left when the reply ends.
pub fn finish(&mut self) -> Option<String> {
let rest = clean_for_speech(&std::mem::take(&mut self.buf), self.keep_tags);
(!rest.is_empty()).then_some(rest)
}
}
fn drain_sentences(buf: &mut String, force: bool, keep_tags: bool) -> Vec<String> {
let mut out = Vec::new();
// A sentence is only worth sending once it is long enough to synthesise
// well; ElevenLabs' own guidance is ~20 characters minimum.
while let Some(end) = buf.find(['.', '!', '?', '\n']) {
if end < 20 && !force {
// Abbreviation or a very short clause — keep accumulating.
if buf.len() < 80 {
break;
}
}
let s: String = buf.drain(..end + 1).collect();
let s = clean_for_speech(&s, keep_tags);
if !s.is_empty() {
out.push(s);
}
}
if force && !buf.is_empty() {
let s = clean_for_speech(&std::mem::take(buf), keep_tags);
if !s.is_empty() {
out.push(s);
}
}
out
}
/// Byte index just past a v3 audio tag starting at `i` ("[laughs]",
/// "[clears throat]"), if there is one. Letters and spaces only, so
/// "[1]" or a markdown link stays ordinary noise.
fn tag_end(s: &str, i: usize) -> Option<usize> {
let rest = s[i..].strip_prefix('[')?;
let close = rest.find(']')?;
let body = &rest[..close];
let end = i + 1 + close + 1;
let ok = !body.trim().is_empty()
&& body.len() <= 40
&& body.chars().all(|c| c.is_alphabetic() || c == ' ' || c == '-')
&& !s[end..].starts_with('(');
ok.then_some(end)
}
/// The text as a person reads it: captions show "ha!", not "[laughs]".
pub fn strip_tags(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut skip_to = 0;
for (i, c) in s.char_indices() {
if i < skip_to {
continue;
}
match tag_end(s, i) {
Some(end) => {
skip_to = end;
out.push(' ');
}
None => out.push(c),
}
}
out.split_whitespace().collect::<Vec<_>>().join(" ")
}
/// Markdown, emoji and kaomoji are for eyes.
///
/// A whitelist, not a blacklist: she writes things like "(¬‿¬)", whose
/// characters live in ordinary Latin-1 and punctuation blocks rather than in
/// any emoji range, so enumerating what to drop never converges. Letters
/// stays unicode-aware on purpose — German is in scope, "ö" is a letter.
///
/// Audio tags pass through untouched when `keep_tags`; otherwise they are
/// dropped whole, since a model without them would read "laughs" aloud.
pub fn clean_for_speech(s: &str, keep_tags: bool) -> String {
let mut out = String::with_capacity(s.len());
let mut skip_to = 0;
for (i, c) in s.char_indices() {
if i < skip_to {
continue;
}
if let Some(end) = tag_end(s, i) {
skip_to = end;
out.push(' ');
if keep_tags {
out.push_str(&s[i..end]);
out.push(' ');
}
} else if c.is_alphanumeric() || c.is_whitespace() || is_speakable_punctuation(c) {
out.push(c);
} else if matches!(c, '—' | '–') {
// " — " must not leave " , ": the comma binds to the word before it.
while out.chars().last().is_some_and(char::is_whitespace) {
out.pop();
}
// An em dash is a pause, not a space. Dropping it runs the two
// clauses together and changes what the sentence means; a comma
// is the closest thing the synthesiser understands. ASCII '-'
// deliberately does NOT land here — it is usually intra-word
// ("push-to-talk"), where a comma would be worse than a space.
out.push(',');
} else {
// Not dropped silently: a symbol between two words is usually a
// dash or a bullet, and gluing the words together mispronounces
// both.
out.push(' ');
}
}
out.split_whitespace().collect::<Vec<_>>().join(" ")
}
/// Punctuation a TTS model actually uses for prosody. Everything else is
/// noise once it reaches audio.
fn is_speakable_punctuation(c: char) -> bool {
matches!(c, '.' | ',' | '!' | '?' | ';' | ':' | '\'' | '"')
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn absolutize_shifts_chunk_relative_times() {
assert_eq!(absolutize(&[0, 50], 1000), vec![1000, 1050]);
}
#[test]
fn absolutize_leaves_already_absolute_times() {
assert_eq!(absolutize(&[1000, 1050], 1000), vec![1000, 1050]);
}
#[test]
fn splitter_emits_complete_sentences_only() {
let mut s = SentenceSplitter::new(false);
assert!(s.push("This is a long enough first sentence").is_empty());
let got = s.push(". And more");
assert_eq!(got, vec!["This is a long enough first sentence."]);
}
#[test]
fn splitter_drops_think_blocks() {
let mut s = SentenceSplitter::new(false);
s.push("<think>internal monologue that is long</think>");
let got = s.push("Actually speakable sentence here. ");
assert_eq!(got, vec!["Actually speakable sentence here."]);
}
#[test]
fn splitter_returns_tail_on_finish() {
let mut s = SentenceSplitter::new(false);
s.push("no terminator here");
assert_eq!(s.finish().as_deref(), Some("no terminator here"));
}
#[test]
fn markdown_and_emoji_never_reach_the_voice() {
assert_eq!(clean_for_speech("**bold** `code` 🎉 ok", false), "bold code ok");
}
#[test]
fn kaomoji_and_dashes_are_stripped_without_gluing_words() {
// Her actual first reply over the API.
assert_eq!(
clean_for_speech("Loud and clear, boss — hallo! (¬‿¬)", false),
"Loud and clear, boss, hallo!"
);
}
#[test]
fn em_dash_becomes_a_pause_but_hyphens_do_not() {
assert_eq!(clean_for_speech("I can't confirm — there's nothing", false), "I can't confirm, there's nothing");
assert_eq!(clean_for_speech("push-to-talk", false), "push to talk");
}
#[test]
fn keeps_non_ascii_letters() {
assert_eq!(clean_for_speech("schön, größer", false), "schön, größer");
}
#[test]
fn audio_tags_survive_for_v3_only() {
let s = "[laughs]Fine, boss. [clears throat] Na gut.";
assert_eq!(clean_for_speech(s, true), "[laughs] Fine, boss. [clears throat] Na gut.");
assert_eq!(clean_for_speech(s, false), "Fine, boss. Na gut.");
}
#[test]
fn brackets_that_are_not_tags_stay_noise() {
assert_eq!(clean_for_speech("see [1] and [the docs](x)", true), "see 1 and the docs x");
}
#[test]
fn captions_hide_tags() {
assert_eq!(strip_tags("[sighs] Okay, boss."), "Okay, boss.");
}
#[test]
fn tag_characters_rest_the_mouth_across_chunks() {
let mut in_tag = false;
let chunk = |chars: &[&str], in_tag: &mut bool| {
let n = chars.len() as i64;
let al = serde_json::json!({
"chars": chars,
"char_start_times_ms": (0..n).map(|i| i * 100).collect::<Vec<_>>(),
"char_durations_ms": vec![100; chars.len()],
});
alignment_to_timeline(&al, 0, in_tag).unwrap()
};
let t = chunk(&["[", "l", "a"], &mut in_tag);
assert!(t.iter().all(|v| v.shape == visemes::SILENCE));
assert!(in_tag);
let t = chunk(&["u", "]", "a"], &mut in_tag);
assert!(!in_tag);
assert_eq!(t.last().unwrap().shape, "aa");
}
}
+131
View File
@@ -0,0 +1,131 @@
use serde::Serialize;
/// VRM 1.0's standard mouth expressions, which is what the avatar rig exposes
/// (`aa`/`ih`/`ou`/`ee`/`oh`). A rig with the full ARKit 52 can map these onto
/// jawOpen/mouthFunnel/etc — the shell side owns that, not this.
///
/// These names are a CONTRACT with the balsam-generated morph targets: the
/// glTF target names become the QML ids, so renaming a shape key in Blender
/// breaks the binding silently, with no eval error.
pub const SILENCE: &str = "sil";
#[derive(Debug, Clone, PartialEq, Serialize)]
pub struct Viseme {
pub shape: &'static str,
/// Milliseconds from speech_start, NOT wall clock — the shell starts its
/// own timer when the first PCM frame hits the speaker, so the two clocks
/// never have to agree.
pub at: u32,
pub ms: u32,
}
/// One character to a mouth shape. English orthography is a lie about
/// phonetics, but at 40ms of animation per character nobody can tell: what
/// reads as "talking" is the OPEN/CLOSE rhythm, not vowel accuracy.
fn shape_for(c: char) -> &'static str {
match c.to_ascii_lowercase() {
'a' => "aa",
'e' => "ee",
'i' | 'y' => "ih",
'o' => "oh",
'u' | 'w' => "ou",
// Bilabials close the mouth; giving them silence is what produces the
// visible "p/b/m" beat.
'p' | 'b' | 'm' => SILENCE,
// Everything else is a partially open consonant. "ih" is the least
// committal shape to land on.
c if c.is_alphabetic() => "ih",
_ => SILENCE,
}
}
/// Builds a playable timeline from ElevenLabs' character alignment.
///
/// The three arrays are parallel; a short or missing one means the frame was
/// truncated, so we take the shortest common prefix rather than panicking on
/// an index.
pub fn timeline(chars: &[String], starts: &[i64], durations: &[i64]) -> Vec<Viseme> {
let n = chars.len().min(starts.len()).min(durations.len());
let mut out: Vec<Viseme> = Vec::with_capacity(n);
for i in 0..n {
let c = match chars[i].chars().next() {
Some(c) => c,
None => continue,
};
let shape = shape_for(c);
let at = starts[i].max(0) as u32;
let ms = durations[i].max(0) as u32;
// Merge runs of the same shape: "ll" is one mouth position held, not
// two identical keyframes the shell would re-trigger.
match out.last_mut() {
Some(last) if last.shape == shape && last.at + last.ms >= at => {
last.ms = (at + ms).saturating_sub(last.at);
}
_ => out.push(Viseme { shape, at, ms }),
}
}
// Sub-frame keys cannot be seen and only cost socket bytes.
out.retain(|v| v.ms >= 20);
out
}
#[cfg(test)]
mod tests {
use super::*;
fn s(v: &[&str]) -> Vec<String> {
v.iter().map(|s| s.to_string()).collect()
}
#[test]
fn maps_vowels_to_distinct_shapes() {
let t = timeline(
&s(&["a", "e", "o"]),
&[0, 100, 200],
&[100, 100, 100],
);
assert_eq!(
t.iter().map(|v| v.shape).collect::<Vec<_>>(),
vec!["aa", "ee", "oh"]
);
}
#[test]
fn merges_repeated_shapes_into_one_held_key() {
let t = timeline(&s(&["l", "l", "l"]), &[0, 50, 100], &[50, 50, 50]);
assert_eq!(t.len(), 1);
assert_eq!(t[0].shape, "ih");
assert_eq!(t[0].at, 0);
assert_eq!(t[0].ms, 150);
}
#[test]
fn bilabials_close_the_mouth() {
let t = timeline(&s(&["a", "m", "a"]), &[0, 100, 200], &[100, 100, 100]);
assert_eq!(t[1].shape, SILENCE);
}
#[test]
fn drops_keys_too_short_to_see() {
let t = timeline(&s(&["a", "e"]), &[0, 5], &[5, 100]);
assert_eq!(t.len(), 1);
assert_eq!(t[0].shape, "ee");
}
#[test]
fn tolerates_truncated_arrays() {
let t = timeline(&s(&["a", "e", "o"]), &[0, 100], &[100]);
assert_eq!(t.len(), 1);
}
#[test]
fn punctuation_and_spaces_are_silence() {
let t = timeline(&s(&[" ", ",", "!"]), &[0, 100, 200], &[100, 100, 100]);
assert_eq!(t.len(), 1);
assert_eq!(t[0].shape, SILENCE);
assert_eq!(t[0].ms, 300);
}
}
+3 -2
View File
@@ -9,6 +9,7 @@ gitea_hermes_webhook_secret: ENC[AES256_GCM,data:lV78H0xAehPxusSO/QruOYkt7fkMJrW
couchdb_luna_password: ENC[AES256_GCM,data:V91is2h7UskI1rtwMzQyduNXoDPTYNwa4sw9K9WU+wE=,iv:k976ImKR19+CvGOVsHsrqMaFFtSQxVi6zABCJuQ5AWE=,tag:W2SN4SET/+o4+Wt6BOJ1LA==,type:str] couchdb_luna_password: ENC[AES256_GCM,data:V91is2h7UskI1rtwMzQyduNXoDPTYNwa4sw9K9WU+wE=,iv:k976ImKR19+CvGOVsHsrqMaFFtSQxVi6zABCJuQ5AWE=,tag:W2SN4SET/+o4+Wt6BOJ1LA==,type:str]
obsidian_luna_passphrase: ENC[AES256_GCM,data:fqHtP3g4J40ddYL9lzCixrisdC/DEJJermE=,iv:FU6BGNcBnyP8Rz3dNBk0+K0aAQTDW6/0aVaFm1rBFkk=,tag:Mg1aKqh++2rmU8ROaVIgDw==,type:str] obsidian_luna_passphrase: ENC[AES256_GCM,data:fqHtP3g4J40ddYL9lzCixrisdC/DEJJermE=,iv:FU6BGNcBnyP8Rz3dNBk0+K0aAQTDW6/0aVaFm1rBFkk=,tag:Mg1aKqh++2rmU8ROaVIgDw==,type:str]
elevenlabs_api_key: ENC[AES256_GCM,data:2oJeClO4/SFsbLwRgMe3bFXrmuDX7YOc57VfSjElV5A3vaIQqfN9eHzo/cm0vISxYgUv,iv:eIeKQJ6odDnmGLjHSLV3HbErVnLr7tatOtV83TvB2QA=,tag:XPkXRfDJ2ZK98TYXKeK/CA==,type:str] elevenlabs_api_key: ENC[AES256_GCM,data:2oJeClO4/SFsbLwRgMe3bFXrmuDX7YOc57VfSjElV5A3vaIQqfN9eHzo/cm0vISxYgUv,iv:eIeKQJ6odDnmGLjHSLV3HbErVnLr7tatOtV83TvB2QA=,tag:XPkXRfDJ2ZK98TYXKeK/CA==,type:str]
hermes_api_server_key: ENC[AES256_GCM,data:Au80iziwdryFTCLnggC9DdjjZPrGPKpmRFgkXe/l9RQ7on9nXb1dvQ==,iv:hWvv4fLkhRgVTT2btI+TvpNBHfMOg9y4BQKn+ht7XDo=,tag:2dEZEys8vsTwc7w0ZiBK1w==,type:str]
sops: sops:
age: age:
- enc: | - enc: |
@@ -29,7 +30,7 @@ sops:
oyJ7PS3lW+PxH5AZkeeU7gXO/pz2oDku0aDOds7kaD3n0+qSWicQ+Q== oyJ7PS3lW+PxH5AZkeeU7gXO/pz2oDku0aDOds7kaD3n0+qSWicQ+Q==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1eapjg6tdrr0fuvmgs3q3nlvnjkaxez298qynqqqxt0lpcv0lrsyq7ayxjk recipient: age1eapjg6tdrr0fuvmgs3q3nlvnjkaxez298qynqqqxt0lpcv0lrsyq7ayxjk
lastmodified: "2026-09-20T15:42:01Z" lastmodified: "2026-09-20T20:02:50Z"
mac: ENC[AES256_GCM,data:9xSoG1hJuPxYgY3LVWKlufvYV043PGCXS9Doznsfoof66INq8MnNc//wQa6Y5/IGD36WIskXGGo0TrLOolj0P91B4zQsus7xnJ8TjQTScD9GezW/iQfdVnkFGr72B6H0K+NOnRCm3lz6ahEn+II99ux/Gji8eq8iTaiOMMfjFXE=,iv:6tA41DUyfSrZk8SM2ml6M75INErRVrPZ7Oduy61SgTk=,tag:QFaUPWlld2q9B5KqSgmg4w==,type:str] mac: ENC[AES256_GCM,data:xJVgl094qzW8YyjwDvRw4ymKjUhL3uWtbVN6J25S1Gi2I2siZ17MiUUis3dKXxb3OKq0NODP9/4pJDqWPv1qt88inu3l/6QUQMY6AybbGF7ooiiOSvGhR3jnCHpISptepNLzyHu0Iky6PkTLe5f5ETSolZ5R+Qyd66ZuFmUBjCM=,iv:u8XWCyjJo/ib9VSzi/e1FEbSjod98c1i14LykC7BEtw=,tag:ZYtrw0b3iEI+yaRtATgbkQ==,type:str]
unencrypted_suffix: _unencrypted unencrypted_suffix: _unencrypted
version: 3.13.3 version: 3.13.3
+4 -2
View File
@@ -5,6 +5,8 @@ librechat_creds_key: ENC[AES256_GCM,data:e2Ptf41yHu0KxfzjW4DP04CSJfbtdsJ8bwrgJyv
librechat_creds_iv: ENC[AES256_GCM,data:iuW1Rxfu7Ei8zHVG7wsBTYYznbtErj7DC9B71OiiWGA=,iv:Ngs88C44gcSzxQZU1lMiy5kn9mM/ckuWKRRGOFJeeoQ=,tag:Q9B8vM4YMkIP3RyYVWSQqg==,type:str] librechat_creds_iv: ENC[AES256_GCM,data:iuW1Rxfu7Ei8zHVG7wsBTYYznbtErj7DC9B71OiiWGA=,iv:Ngs88C44gcSzxQZU1lMiy5kn9mM/ckuWKRRGOFJeeoQ=,tag:Q9B8vM4YMkIP3RyYVWSQqg==,type:str]
librechat_jwt_secret: ENC[AES256_GCM,data:f/gljrQPZIXeLHXtqKCCYGEu2pXgbZd69CjyEhSVJ+AMuaVj7DWVclD5IMsEVERBHw5a7OndBSfLvgXRaKN6Gg==,iv:DG5CwCbQLCTv1++APCdFzAiWGKiEJl+9MhKIM2ykJQM=,tag:wdpY6l+5h4UtsluIncsyhA==,type:str] librechat_jwt_secret: ENC[AES256_GCM,data:f/gljrQPZIXeLHXtqKCCYGEu2pXgbZd69CjyEhSVJ+AMuaVj7DWVclD5IMsEVERBHw5a7OndBSfLvgXRaKN6Gg==,iv:DG5CwCbQLCTv1++APCdFzAiWGKiEJl+9MhKIM2ykJQM=,tag:wdpY6l+5h4UtsluIncsyhA==,type:str]
librechat_jwt_refresh_secret: ENC[AES256_GCM,data:N/yAPtaodJX4t2C2A0bZ/LQaVc8SBMgg3KYJrWRyXCM6HAmgHtSQ9nWJO2fl+7A966mMtYvpATYcBIPj/K5nig==,iv:k5ZYi2OKjL9Z+lhVgdNsmGVQm+iCqmF15H3TUTbofWk=,tag:NVTdvlL1lpwVhjo2c2phpg==,type:str] librechat_jwt_refresh_secret: ENC[AES256_GCM,data:N/yAPtaodJX4t2C2A0bZ/LQaVc8SBMgg3KYJrWRyXCM6HAmgHtSQ9nWJO2fl+7A966mMtYvpATYcBIPj/K5nig==,iv:k5ZYi2OKjL9Z+lhVgdNsmGVQm+iCqmF15H3TUTbofWk=,tag:NVTdvlL1lpwVhjo2c2phpg==,type:str]
hermes_api_server_key: ENC[AES256_GCM,data:NU+6zVmUeM9+m5eZCPdCiMLo76GqbsRxNM7EmVM2TmKTOUL2af7Msg==,iv:EMd7JKhQ3ZkFPi2xMGZ+mHHK05qQVDR+ZCbiYcWdhHU=,tag:JaElU4CyvD5qQbehHbM5qQ==,type:str]
elevenlabs_api_key: ENC[AES256_GCM,data:4IBKKMxor1sJ1skh1pgLrQnNX31cgQFiy89O1TZn71QHaq+xY7p+7JLtmGkhbZ10l12m,iv:Ww5RLYuv09xD1deFgM89u0wZ3hZ/0Pndda0kreMx2aQ=,tag:opgIPsDwSc+rLqseqjP6Gg==,type:str]
sops: sops:
age: age:
- enc: | - enc: |
@@ -25,7 +27,7 @@ sops:
sHjKfw8VrrmAR4pQf1dsY+wcyh4FsZxhP3Q+QIVq3eCIXS9PeJkGAg== sHjKfw8VrrmAR4pQf1dsY+wcyh4FsZxhP3Q+QIVq3eCIXS9PeJkGAg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1rfcmu6zh40v4260l9hnf8ajs9vly0s06rx3ey76eu78dp9t7getqyhmkut recipient: age1rfcmu6zh40v4260l9hnf8ajs9vly0s06rx3ey76eu78dp9t7getqyhmkut
lastmodified: "2026-08-21T23:14:15Z" lastmodified: "2026-09-20T20:02:50Z"
mac: ENC[AES256_GCM,data:gD5C3hYFvIUP7zr4GK40LAtM2sskhGErEzdTxVKaRaPkKNYj+pDYd5uSrBxDHv6W2SAY0TtyN3GyNpD0z5b4yaDxiV8ETUOokkj8f9FlTaZk56S6I/21HvaGXW+Cfi1ioFtNGyJeAhSnNOJ6OE0AA9JfcftSJ8BdTe2LienjHwQ=,iv:LtKFum+f1kaE6/XT4Xj0ZEuFrhSXXsnILMgdebGYNDU=,tag:6DxwNjjgze2ldU1g3SdzzA==,type:str] mac: ENC[AES256_GCM,data:fEgOKCOYrpnojw9yl3b7ZgJFCcmAs+DmY7etHgD5KNBtoZ3QgQUiTY5+kXvO5HMoENve+bUfyHymUDNjLlb45lItl9ZkAMKvTxGOe+5t0MR2X8BkB/2tjl/fS7CRBTJ0D4S84mH1hB2V2cfKcnGoKNsQyRJddi6OOey534QeV44=,iv:nQuRLkGklMXwkh8LpMC/Tmz1ewRiv8iKbK6IA88r2fA=,tag:LnzQc1sjrIdZ26AtH1Laxw==,type:str]
unencrypted_suffix: _unencrypted unencrypted_suffix: _unencrypted
version: 3.13.2 version: 3.13.2
+20
View File
@@ -0,0 +1,20 @@
{ pkgs, ... }:
let
claude-desktop = pkgs.callPackage ../../pkgs/claude-desktop.nix { };
in
# Claude Desktop, including Cowork (a QEMU/KVM VM the app runs agent tasks in).
{
environment.systemPackages = [ claude-desktop ];
# The app only probes these Debian paths, never $PATH or an env var.
systemd.tmpfiles.rules = [
"L+ /usr/share/OVMF/OVMF_CODE_4M.fd - - - - ${pkgs.OVMF.fd}/FV/OVMF_CODE.fd"
"L+ /usr/share/OVMF/OVMF_VARS_4M.fd - - - - ${pkgs.OVMF.fd}/FV/OVMF_VARS.fd"
"L+ /usr/libexec/virtiofsd - - - - ${claude-desktop}/lib/claude-desktop/resources/virtiofsd"
];
boot.kernelModules = [ "vhost_vsock" ];
users.users.darman.extraGroups = [ "kvm" ];
}
+3
View File
@@ -16,6 +16,7 @@ in
"steam-unwrapped" "steam-unwrapped"
"steam-run" "steam-run"
"claude-code" "claude-code"
"claude-desktop" # services/desktop/claude-desktop.nix
"proton-pass-cli" "proton-pass-cli"
"vivaldi" "vivaldi"
"mongodb" # librechat's local db (services/desktop/librechat.nix) — SSPL "mongodb" # librechat's local db (services/desktop/librechat.nix) — SSPL
@@ -45,6 +46,8 @@ in
dotnetCorePackages.sdk_10_0 dotnetCorePackages.sdk_10_0
nodejs nodejs
yaak # desktop API client (REST/GraphQL/gRPC) yaak # desktop API client (REST/GraphQL/gRPC)
# pkgsRocm, not plain blender: Cycles needs HIP to render on the RX 6800 XT.
pkgsRocm.blender
]; ];
fonts.packages = [ pkgs.nerd-fonts.departure-mono ]; fonts.packages = [ pkgs.nerd-fonts.departure-mono ];
+169
View File
@@ -0,0 +1,169 @@
{ config, pkgs, lib, ... }:
# Voice loop for luna: SUPER+V opens the mic, talk, hear her answer.
#
# mic -> whisper-cpp (local, vulkan) -> Hermes' OpenAI-compatible API on mars
# -> ElevenLabs streaming TTS -> speaker. State, amplitude, captions and a
# viseme timeline are published on a unix socket; the rail's LunaPanel reads
# them (dotfiles/quickshell/HyprChrome/Widgets/Bar/Luna), and the viseme
# stream is there for the avatar that replaces its meter later.
#
# A USER service, not a system one: it needs darman's pipewire session and his
# XDG_RUNTIME_DIR for the socket. That is also why the two secrets it reads
# are owner-darman in hosts/terra/secrets.nix — a user manager cannot read a
# root-owned 0400 file, and LoadCredential does not bridge that gap.
#
# STT is LOCAL on purpose: mars already runs faster-whisper for Telegram voice
# notes, but routing the mic through it would put every spoken word on the
# network for no latency win. This is the client-direct path Hermes itself
# documents.
let
cfg = config.homelab.lunaVoice;
luna-voice = pkgs.callPackage ../../pkgs/luna-voice.nix { };
# Pinned like the KittenTTS models on mars: a model that re-downloads at
# first use is a service that fails on a fresh install with no network.
whisperModel = pkgs.fetchurl {
url = "https://huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-base.en.bin";
sha256 = "sha256-oDd5yG3zMjB19eeWyyzlAp8A7Ihp7uP9+4l6/jbG0AI=";
};
in
{
options.homelab.lunaVoice = {
enable = lib.mkEnableOption "luna voice daemon";
user = lib.mkOption {
type = lib.types.str;
default = "darman";
description = "User whose session runs the daemon.";
};
hermesUrl = lib.mkOption {
type = lib.types.str;
default = "http://mars.orbit.sol:8642";
description = "Base URL of luna's Hermes API server (tailnet-only).";
};
model = lib.mkOption {
type = lib.types.str;
default = "hermes-agent";
description = "Model name to request; Hermes routes it to her configured provider.";
};
voiceId = lib.mkOption {
type = lib.types.str;
description = "ElevenLabs voice id — luna's voice.";
};
ttsModel = lib.mkOption {
type = lib.types.str;
default = "eleven_v3_conversational";
description = ''
eleven_v3* streams over the Text to Dialogue socket and performs audio
tags ([laughs], [sighs]); it also gets her German right, which Flash
reads with an English accent. Anything else uses the TTS socket, e.g.
"eleven_flash_v2_5" — half the price per character, faster, no tags.
'';
};
hermesKeyFile = lib.mkOption {
type = lib.types.path;
description = "File holding the API_SERVER_KEY bearer token.";
};
elevenLabsKeyFile = lib.mkOption {
type = lib.types.path;
description = "File holding the ElevenLabs API key.";
};
# Mic tuning. Defaults are set for terra's AIR 192|4, which is QUIET:
# ~0.002 ambient against ~0.0045 for normal speech. Raising the gain on
# the interface is worth more than any value here — everything downstream,
# whisper included, works better with headroom.
inputDevice = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
description = "Substring of the input device name; null uses the system default.";
};
vadMult = lib.mkOption {
type = lib.types.float;
default = 1.8;
description = "Speech triggers above noise_floor * this.";
};
vadFloor = lib.mkOption {
type = lib.types.float;
default = 0.0025;
description = "Absolute lower bound on the trigger, whatever the floor says.";
};
silenceMs = lib.mkOption {
type = lib.types.int;
default = 2000;
description = "Silence that ends an utterance.";
};
voiceMarker = lib.mkOption {
type = lib.types.str;
default = "[voice]";
description = ''
Prefixed to every spoken message so luna can tell this channel from
Telegram or the dashboard, and adjust — no kaomoji, no markdown, short
answers. Put the rule in HER profile on mars rather than only in
systemPrompt below: the marker travels with the message, so it still
identifies the channel in whatever history Hermes keeps, and one place
governs her voice manners across every client that sends it.
Empty string disables the marker.
'';
};
systemPrompt = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
description = ''
Prepended to the conversation. Worth setting to something like "You
are being spoken to out loud; keep replies short" — her Telegram
register is too long-form to listen to.
'';
};
};
config = lib.mkIf cfg.enable {
environment.systemPackages = [ luna-voice ];
systemd.user.services.luna-voiced = {
description = "luna voice loop";
wantedBy = [ "graphical-session.target" ];
after = [ "graphical-session.target" "pipewire.service" ];
partOf = [ "graphical-session.target" ];
environment = {
LUNA_HERMES_URL = cfg.hermesUrl;
LUNA_MODEL = cfg.model;
LUNA_VOICE_ID = cfg.voiceId;
LUNA_TTS_MODEL = cfg.ttsModel;
LUNA_HERMES_KEY_FILE = toString cfg.hermesKeyFile;
LUNA_ELEVENLABS_KEY_FILE = toString cfg.elevenLabsKeyFile;
LUNA_WHISPER_BIN = "${pkgs.whisper-cpp-vulkan}/bin/whisper-cli";
LUNA_WHISPER_MODEL = whisperModel;
LUNA_VAD_MULT = toString cfg.vadMult;
LUNA_VAD_FLOOR = toString cfg.vadFloor;
LUNA_SILENCE_MS = toString cfg.silenceMs;
LUNA_VOICE_MARKER = cfg.voiceMarker;
} // lib.optionalAttrs (cfg.systemPrompt != null) {
LUNA_SYSTEM_PROMPT = cfg.systemPrompt;
} // lib.optionalAttrs (cfg.inputDevice != null) {
LUNA_INPUT_DEVICE = cfg.inputDevice;
};
serviceConfig = {
ExecStart = "${luna-voice}/bin/luna-voice daemon";
Restart = "on-failure";
RestartSec = 3;
};
};
};
}
+26
View File
@@ -0,0 +1,26 @@
{ pkgs, ... }:
# gitea-mcp in HTTP mode, tailnet-only at http://jupiter.orbit.sol:8093/mcp.
# It holds no token: each client sends its own PAT as `Authorization: Bearer
# <token>`, so an unauthenticated request can only see what anonymous gitea
# users can.
{
systemd.services.gitea-mcp = {
description = "Gitea MCP server";
after = [ "gitea.service" ];
wants = [ "gitea.service" ];
wantedBy = [ "multi-user.target" ];
environment.GITEA_HOST = "http://localhost:3000";
serviceConfig = {
# Binds every interface (no listen-address flag); tailnet-only because
# tailscale0 is trusted and 8093 stays closed on the LAN.
ExecStart = "${pkgs.gitea-mcp-server}/bin/gitea-mcp -t http -p 8093";
DynamicUser = true;
Restart = "on-failure";
ProtectSystem = "strict";
ProtectHome = true;
PrivateTmp = true;
NoNewPrivileges = true;
};
};
}