feat(terra): hands-free voice mode, toggled with SUPER+V
The key is a toggle rather than push-to-talk, because "voice mode is active" has to be a state something can display; a held key cannot sustain one. So the daemon grew a VAD loop: speech starts a capture, `silenceMs` of quiet ends it. A turn is awaited inline, which is also what stops her hearing herself — the tick cannot fire while she speaks. The keybind runs the CLI, not a quickshell GlobalShortcut, so the mic can still be switched off while the shell is reloading and a QML error can never leave it open with nothing on screen saying so. Every threshold had to be resized for terra's mic, which is QUIET: 0.001-0.002 ambient against 0.0045 speaking, about 6dB apart. The VAD floor of 0.006 and whisper's own 0.015 gate both sat ABOVE the user's voice, and a 3.5x multiplier cannot separate signals that close — three independent reasons nothing could ever trigger. Hence vadMult 1.8, vadFloor 0.0025, and stt::normalize lifting the clip (capped at 30x, so near-silence is not amplified into words) before whisper sees it. The floor tracker also had to stop feeding on everything: a rising sound dragged its own threshold up faster than it rose, and a tone peaking at 0.029 was outrun by a threshold that reached 0.018. Only samples below the current trigger update it now. Four diagnostics, because none of the above was visible while it was failing: `watch` prints the socket protocol, `say` runs a turn from typed text (a broken mic vs a broken pipeline), `level` is a live meter with the trigger marked, `devices` lists the inputs. Amplitude is now emitted on every tick, not only while she speaks — a meter that moves only during playback cannot answer "is it hearing me". Also: connect-time replay of mode/state, so restarting the shell while voice mode is on does not leave it blind until the next event, which in an idle conversation is never. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -17,7 +17,7 @@ pub async fn transcribe(bin: &str, model: &Path, samples: &[f32]) -> Result<Stri
|
||||
|
||||
let dir = std::env::var("XDG_RUNTIME_DIR").unwrap_or_else(|_| "/tmp".into());
|
||||
let wav = Path::new(&dir).join(format!("luna-stt-{}.wav", std::process::id()));
|
||||
audio::write_wav(&wav, samples, audio::STT_RATE)?;
|
||||
audio::write_wav(&wav, &normalize(samples), audio::STT_RATE)?;
|
||||
|
||||
let out = Command::new(bin)
|
||||
.args(["-m"])
|
||||
@@ -40,11 +40,38 @@ pub async fn transcribe(bin: &str, model: &Path, samples: &[f32]) -> Result<Stri
|
||||
Ok(clean(&String::from_utf8_lossy(&out.stdout)))
|
||||
}
|
||||
|
||||
/// Scales a quiet recording up before whisper sees it.
|
||||
///
|
||||
/// Whisper is trained on normalised audio and degrades badly on very quiet
|
||||
/// input, which is precisely what a low-gain interface produces — speech at a
|
||||
/// 0.03 peak is intelligible to a person and mush to the model. The gain is
|
||||
/// capped so a recording of near-silence is not amplified into hiss the model
|
||||
/// will confidently transcribe as words.
|
||||
fn normalize(samples: &[f32]) -> Vec<f32> {
|
||||
const TARGET_PEAK: f32 = 0.6;
|
||||
const MAX_GAIN: f32 = 30.0;
|
||||
|
||||
let peak = samples.iter().fold(0.0f32, |m, s| m.max(s.abs()));
|
||||
if peak <= f32::EPSILON {
|
||||
return samples.to_vec();
|
||||
}
|
||||
let gain = (TARGET_PEAK / peak).min(MAX_GAIN);
|
||||
// Never attenuate: a loud recording is already fine, and turning it down
|
||||
// would only lose resolution in the 16-bit conversion.
|
||||
if gain <= 1.0 {
|
||||
return samples.to_vec();
|
||||
}
|
||||
samples.iter().map(|s| (s * gain).clamp(-1.0, 1.0)).collect()
|
||||
}
|
||||
|
||||
/// True if any 100ms window is loud enough to be a voice. Peak rather than
|
||||
/// overall RMS: a short sentence inside a long silent recording averages down
|
||||
/// to nothing, and that is the common case for a held key.
|
||||
fn has_speech(samples: &[f32]) -> bool {
|
||||
const FLOOR: f32 = 0.015;
|
||||
// Matches the VAD's lower clamp rather than a "typical" mic: a quiet
|
||||
// interface puts normal speech at ~0.004, and gating at 0.015 threw away
|
||||
// every word it captured.
|
||||
const FLOOR: f32 = 0.0025;
|
||||
let window = (audio::STT_RATE as usize) / 10;
|
||||
samples.chunks(window).any(|w| {
|
||||
let sum: f32 = w.iter().map(|s| s * s).sum();
|
||||
@@ -70,14 +97,56 @@ fn clean(raw: &str) -> String {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{clean, has_speech};
|
||||
use super::{clean, has_speech, normalize};
|
||||
use crate::audio::STT_RATE;
|
||||
|
||||
fn peak(v: &[f32]) -> f32 {
|
||||
v.iter().fold(0.0f32, |m, s| m.max(s.abs()))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn normalize_lifts_a_quiet_recording() {
|
||||
// 0.004 RMS speech peaks around 0.02, which is 30x under target —
|
||||
// exactly at the gain cap, so it just reaches full scale.
|
||||
let quiet: Vec<f32> = (0..1000).map(|i| 0.02 * ((i % 7) as f32 - 3.0) / 3.0).collect();
|
||||
assert!((peak(&normalize(&quiet)) - 0.6).abs() < 0.01);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn normalize_leaves_a_loud_recording_alone() {
|
||||
let loud: Vec<f32> = vec![0.9, -0.8, 0.7];
|
||||
assert_eq!(normalize(&loud), loud);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn normalize_caps_the_gain_so_silence_stays_silent() {
|
||||
let hiss: Vec<f32> = vec![0.0001, -0.0001];
|
||||
// 0.6/0.0001 would be 6000x; the cap keeps it at 30x.
|
||||
assert!(peak(&normalize(&hiss)) <= 0.0001 * 30.0 + 1e-9);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn normalize_handles_pure_silence() {
|
||||
assert_eq!(normalize(&[0.0, 0.0]), vec![0.0, 0.0]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn silence_is_not_speech() {
|
||||
assert!(!has_speech(&vec![0.0f32; STT_RATE as usize]));
|
||||
// Room tone: audible on a meter, not a voice.
|
||||
assert!(!has_speech(&vec![0.004f32; STT_RATE as usize]));
|
||||
// Terra's measured room tone. Speech on the same mic reaches
|
||||
// 0.004–0.005, so the gate sits between the two.
|
||||
assert!(!has_speech(&vec![0.001f32; STT_RATE as usize]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn quiet_speech_still_counts() {
|
||||
// 0.004 is what normal speech measures on terra's interface; gating
|
||||
// it out discarded every word the VAD had just captured.
|
||||
let mut s = vec![0.001f32; STT_RATE as usize];
|
||||
for v in s.iter_mut().take(STT_RATE as usize / 5) {
|
||||
*v = 0.004;
|
||||
}
|
||||
assert!(has_speech(&s));
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
Reference in New Issue
Block a user