feat(terra): talk to luna with push-to-talk

luna-voiced: mic -> whisper-cpp (local, vulkan) -> Hermes' API on mars ->
ElevenLabs streaming TTS -> speaker, bound to SUPER+V. State, amplitude
and a viseme timeline are published on a unix socket; nothing consumes
that yet, but it is the data source the quickshell avatar will read.

STT stays local on purpose. mars already runs faster-whisper for Telegram
voice notes, but routing the mic through it would put every spoken word
on the network for no latency win.

Shape notes, all of them things the first live runs decided:

  - Viseme offsets are relative to speech_start (emitted when the first
    PCM frame reaches the speaker), never wall clock, so the shell and
    the daemon never have to compare clocks. ElevenLabs sends character
    alignment, not visemes, so the mapping is ours; the glTF morph target
    names are the contract.
  - clean_for_speech is a whitelist. She writes "(¬‿¬)", whose characters
    live in Latin-1 and punctuation blocks rather than any emoji range,
    so enumerating what to drop never converges. Em dashes become commas
    because they are a pause — dropping one ran two clauses together and
    changed what a sentence meant.
  - whisper invents words out of silence ("you", "Thank you."), and a
    held key over a quiet room is long AND empty, so has_speech gates on
    the loudest 100ms window rather than on duration.
  - Beeps on key down/up and a distinct low tone for "heard nothing":
    push-to-talk with no acknowledgement is indistinguishable from broken,
    and a dead mic must not feel like a slow reply.
  - A warmup request at startup: her first reply after the container
    starts costs ~16s against ~5s warm, and the first one is always the
    one you are waiting for.
  - `luna-voice watch` prints the socket protocol and `luna-voice say`
    runs a turn from typed text, which is how a broken mic is told apart
    from a broken pipeline.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-20 22:45:17 +02:00
co-authored by Claude Opus 5
parent 8dc3affdda
commit d089743f75
17 changed files with 3934 additions and 1 deletions
+112
View File
@@ -0,0 +1,112 @@
use crate::audio;
use anyhow::{bail, Result};
use std::path::Path;
use tokio::process::Command;
/// Shells out to whisper-cli rather than linking whisper.cpp: nixpkgs already
/// ships whisper-cpp-vulkan built against this box's GPU stack, and a bundled
/// build would have to rediscover that at cargo time.
pub async fn transcribe(bin: &str, model: &Path, samples: &[f32]) -> Result<String> {
// Whisper hallucinates confident words into silence — "you", "Thank
// you.", "Bye." — which then reach luna as if spoken. Duration alone does
// not catch it (a held key over a quiet room is long AND empty), so gate
// on level too.
if samples.len() < (audio::STT_RATE as usize) / 3 || !has_speech(samples) {
return Ok(String::new());
}
let dir = std::env::var("XDG_RUNTIME_DIR").unwrap_or_else(|_| "/tmp".into());
let wav = Path::new(&dir).join(format!("luna-stt-{}.wav", std::process::id()));
audio::write_wav(&wav, samples, audio::STT_RATE)?;
let out = Command::new(bin)
.args(["-m"])
.arg(model)
.args(["-f"])
.arg(&wav)
// -nt: transcript only, no [00:00:00] prefixes. -np: no progress
// banner. Together they make stdout exactly the text.
.args(["-nt", "-np"])
.output()
.await?;
let _ = std::fs::remove_file(&wav);
if !out.status.success() {
bail!(
"whisper-cli failed: {}",
String::from_utf8_lossy(&out.stderr).trim()
);
}
Ok(clean(&String::from_utf8_lossy(&out.stdout)))
}
/// True if any 100ms window is loud enough to be a voice. Peak rather than
/// overall RMS: a short sentence inside a long silent recording averages down
/// to nothing, and that is the common case for a held key.
fn has_speech(samples: &[f32]) -> bool {
const FLOOR: f32 = 0.015;
let window = (audio::STT_RATE as usize) / 10;
samples.chunks(window).any(|w| {
let sum: f32 = w.iter().map(|s| s * s).sum();
(sum / w.len().max(1) as f32).sqrt() > FLOOR
})
}
/// whisper emits bracketed non-speech annotations ([BLANK_AUDIO], (wind
/// blowing)) that are not words and must not reach the agent.
fn clean(raw: &str) -> String {
let mut out = String::new();
let mut depth = 0i32;
for c in raw.chars() {
match c {
'[' | '(' => depth += 1,
']' | ')' => depth = (depth - 1).max(0),
c if depth == 0 => out.push(c),
_ => {}
}
}
out.split_whitespace().collect::<Vec<_>>().join(" ")
}
#[cfg(test)]
mod tests {
use super::{clean, has_speech};
use crate::audio::STT_RATE;
#[test]
fn silence_is_not_speech() {
assert!(!has_speech(&vec![0.0f32; STT_RATE as usize]));
// Room tone: audible on a meter, not a voice.
assert!(!has_speech(&vec![0.004f32; STT_RATE as usize]));
}
#[test]
fn a_short_utterance_in_a_long_silence_counts() {
let mut s = vec![0.0f32; STT_RATE as usize];
for v in s.iter_mut().take(STT_RATE as usize / 5) {
*v = 0.2;
}
assert!(has_speech(&s));
}
#[test]
fn strips_bracketed_annotations() {
assert_eq!(clean("[BLANK_AUDIO]"), "");
assert_eq!(clean(" hello (wind blowing) there \n"), "hello there");
assert_eq!(clean("[ music ] real words"), "real words");
}
#[test]
fn collapses_whitespace() {
assert_eq!(clean(" a b \n c "), "a b c");
}
#[test]
fn unbalanced_brackets_do_not_eat_the_rest() {
// The stray closer is dropped (depth floors at 0) but the words
// around it survive — the failure mode that matters is a lone
// bracket swallowing the whole transcript.
assert_eq!(clean("ok] fine"), "ok fine");
assert_eq!(clean("[ oops real words"), "");
}
}