feat(terra): talk to luna with push-to-talk
luna-voiced: mic -> whisper-cpp (local, vulkan) -> Hermes' API on mars ->
ElevenLabs streaming TTS -> speaker, bound to SUPER+V. State, amplitude
and a viseme timeline are published on a unix socket; nothing consumes
that yet, but it is the data source the quickshell avatar will read.
STT stays local on purpose. mars already runs faster-whisper for Telegram
voice notes, but routing the mic through it would put every spoken word
on the network for no latency win.
Shape notes, all of them things the first live runs decided:
- Viseme offsets are relative to speech_start (emitted when the first
PCM frame reaches the speaker), never wall clock, so the shell and
the daemon never have to compare clocks. ElevenLabs sends character
alignment, not visemes, so the mapping is ours; the glTF morph target
names are the contract.
- clean_for_speech is a whitelist. She writes "(¬‿¬)", whose characters
live in Latin-1 and punctuation blocks rather than any emoji range,
so enumerating what to drop never converges. Em dashes become commas
because they are a pause — dropping one ran two clauses together and
changed what a sentence meant.
- whisper invents words out of silence ("you", "Thank you."), and a
held key over a quiet room is long AND empty, so has_speech gates on
the loudest 100ms window rather than on duration.
- Beeps on key down/up and a distinct low tone for "heard nothing":
push-to-talk with no acknowledgement is indistinguishable from broken,
and a dead mic must not feel like a slow reply.
- A warmup request at startup: her first reply after the container
starts costs ~16s against ~5s warm, and the first one is always the
one you are waiting for.
- `luna-voice watch` prints the socket protocol and `luna-voice say`
runs a turn from typed text, which is how a broken mic is told apart
from a broken pipeline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -17,3 +17,6 @@ keys.txt
|
|||||||
|
|
||||||
# local visual-verification output
|
# local visual-verification output
|
||||||
.artifacts/
|
.artifacts/
|
||||||
|
|
||||||
|
# rust build output
|
||||||
|
target/
|
||||||
|
|||||||
@@ -210,6 +210,19 @@ kept its ssh host key. Run it after ANY change to the kexec paths.
|
|||||||
- **`kexec-local` stages on `/var/tmp`, not `/tmp`**: `kexec-run.sh` appends a fresh cpio
|
- **`kexec-local` stages on `/var/tmp`, not `/tmp`**: `kexec-run.sh` appends a fresh cpio
|
||||||
to `kexec/initrd` in place and execs binaries from that dir, so a size-capped or
|
to `kexec/initrd` in place and execs binaries from that dir, so a size-capped or
|
||||||
`noexec` tmpfs gives a half-written initrd or a bare "Permission denied".
|
`noexec` tmpfs gives a half-written initrd or a bare "Permission denied".
|
||||||
|
- **A sops secret a USER service reads needs `owner`, not a mode.** `luna-voiced`
|
||||||
|
(`services/desktop/luna-voice.nix`) runs as darman's user unit, and a user manager
|
||||||
|
cannot read root:root 0400 — `LoadCredential` does not bridge that gap either.
|
||||||
|
Hence `sops.secrets.<x>.owner = "darman"` in `hosts/terra/secrets.nix`.
|
||||||
|
- **`elevenlabs_api_key` is in BOTH `secrets/mars.yaml` and `secrets/terra.yaml`** —
|
||||||
|
same value. terra calls ElevenLabs directly (the client-direct path Hermes
|
||||||
|
documents) rather than relaying audio through mars, so rotating that key means
|
||||||
|
editing two files. `hermes_api_server_key` is duplicated for the same reason:
|
||||||
|
mars serves the API, terra is its only client.
|
||||||
|
- **whisper invents words out of silence** — "you", "Thank you.", "Bye." — and a
|
||||||
|
held push-to-talk key over a quiet room is long AND empty, so duration alone
|
||||||
|
does not catch it. `stt::has_speech` gates on the loudest 100ms window instead;
|
||||||
|
without it every stray keypress sends a hallucinated sentence to luna.
|
||||||
- **terra's greeter is a throwaway Hyprland running `dotfiles/quickshell/greeter.qml`**
|
- **terra's greeter is a throwaway Hyprland running `dotfiles/quickshell/greeter.qml`**
|
||||||
(`services/desktop/quickshell-greeter.nix`). It must exit after login or greetd never
|
(`services/desktop/quickshell-greeter.nix`). It must exit after login or greetd never
|
||||||
starts the session, and with a Lua config `hyprctl dispatch exit` is REJECTED — it needs
|
starts the session, and with a Lua config `hyprctl dispatch exit` is REJECTED — it needs
|
||||||
|
|||||||
@@ -19,8 +19,27 @@ in
|
|||||||
../../services/desktop/desktop-hyprland.nix
|
../../services/desktop/desktop-hyprland.nix
|
||||||
../../services/desktop/desktop-apps.nix
|
../../services/desktop/desktop-apps.nix
|
||||||
../../services/desktop/librechat.nix
|
../../services/desktop/librechat.nix
|
||||||
|
../../services/desktop/luna-voice.nix
|
||||||
];
|
];
|
||||||
|
|
||||||
|
# Talk to luna (Hermes Agent on mars) with SUPER+V — see
|
||||||
|
# hosts/terra/home/hyprland.nix for the push-to-talk binds.
|
||||||
|
homelab.lunaVoice = {
|
||||||
|
enable = true;
|
||||||
|
hermesKeyFile = config.sops.secrets.hermes_api_server_key.path;
|
||||||
|
elevenLabsKeyFile = config.sops.secrets.elevenlabs_api_key.path;
|
||||||
|
# luna's own voice — must stay in sync with `tts.elevenlabs.voice_id` in
|
||||||
|
# her config.yaml on mars, or she sounds like someone else on the desktop
|
||||||
|
# than she does on Telegram.
|
||||||
|
voiceId = "LZAcK8Cx5QjdQhfBsJQZ";
|
||||||
|
systemPrompt = ''
|
||||||
|
You are being spoken to out loud and your reply will be read aloud by a
|
||||||
|
speech synthesiser. Answer in one or two sentences unless asked for
|
||||||
|
detail. No markdown, no lists, no code blocks — they do not survive the
|
||||||
|
trip to audio.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
networking.hostName = "terra";
|
networking.hostName = "terra";
|
||||||
|
|
||||||
homelab.greeter = {
|
homelab.greeter = {
|
||||||
|
|||||||
@@ -195,6 +195,12 @@ in
|
|||||||
# expand / collapse the hyprchrome bar as a whole
|
# expand / collapse the hyprchrome bar as a whole
|
||||||
(bind "SUPER + A" (dsp.global "quickshell:chrome"))
|
(bind "SUPER + A" (dsp.global "quickshell:chrome"))
|
||||||
|
|
||||||
|
# push-to-talk for luna (services/desktop/luna-voice.nix). Two binds,
|
||||||
|
# not one: the daemon records between the press and the release, so
|
||||||
|
# the release edge is what ends the utterance.
|
||||||
|
(bind "SUPER + V" (dsp.exec "luna-voice ptt-down"))
|
||||||
|
(bindo "SUPER + V" (dsp.exec "luna-voice ptt-up") { release = true; })
|
||||||
|
|
||||||
(bind "SUPER + B" (dsp.exec "vivaldi"))
|
(bind "SUPER + B" (dsp.exec "vivaldi"))
|
||||||
(bind "SUPER + E" (dsp.exec "cosmic-files"))
|
(bind "SUPER + E" (dsp.exec "cosmic-files"))
|
||||||
(bind "SUPER + SHIFT + G" (dsp.workspaceRef "game"))
|
(bind "SUPER + SHIFT + G" (dsp.workspaceRef "game"))
|
||||||
@@ -237,7 +243,7 @@ in
|
|||||||
(bind "SUPER + T" (dsp.togglespecial "terminal"))
|
(bind "SUPER + T" (dsp.togglespecial "terminal"))
|
||||||
(bind "SUPER + C" (dsp.togglespecial "communications"))
|
(bind "SUPER + C" (dsp.togglespecial "communications"))
|
||||||
(bind "SUPER + M" (dsp.togglespecial "music"))
|
(bind "SUPER + M" (dsp.togglespecial "music"))
|
||||||
(bind "SUPER + V" (dsp.togglespecial "version_control"))
|
# (bind "SUPER + V" (dsp.togglespecial "version_control"))
|
||||||
|
|
||||||
# cycle workspaces
|
# cycle workspaces
|
||||||
(bind "SUPER + ALT + up" (dsp.workspaceRef "e+1"))
|
(bind "SUPER + ALT + up" (dsp.workspaceRef "e+1"))
|
||||||
|
|||||||
@@ -27,6 +27,18 @@
|
|||||||
# tokens. All four are random, generated once with `sops --set` (see
|
# tokens. All four are random, generated once with `sops --set` (see
|
||||||
# CLAUDE.md) — losing/rotating them just invalidates existing sessions and
|
# CLAUDE.md) — losing/rotating them just invalidates existing sessions and
|
||||||
# any saved per-user API keys, nothing else depends on their value.
|
# any saved per-user API keys, nothing else depends on their value.
|
||||||
|
# luna-voiced (services/desktop/luna-voice.nix) runs as a USER service, so
|
||||||
|
# these cannot stay root:root 0400 — a user manager cannot read them and
|
||||||
|
# LoadCredential does not bridge that gap either. Hence owner, not mode.
|
||||||
|
#
|
||||||
|
# hermes_api_server_key mirrors secrets/mars.yaml (the serving side);
|
||||||
|
# elevenlabs_api_key mirrors it too, because client-direct TTS is the
|
||||||
|
# documented low-latency path — terra calls ElevenLabs itself rather than
|
||||||
|
# relaying audio through mars. Rotating that key therefore touches BOTH
|
||||||
|
# files; see CLAUDE.md.
|
||||||
|
sops.secrets.hermes_api_server_key.owner = "darman";
|
||||||
|
sops.secrets.elevenlabs_api_key.owner = "darman";
|
||||||
|
|
||||||
sops.secrets.librechat_creds_key = { };
|
sops.secrets.librechat_creds_key = { };
|
||||||
sops.secrets.librechat_creds_iv = { };
|
sops.secrets.librechat_creds_iv = { };
|
||||||
sops.secrets.librechat_jwt_secret = { };
|
sops.secrets.librechat_jwt_secret = { };
|
||||||
|
|||||||
@@ -0,0 +1,32 @@
|
|||||||
|
{ lib
|
||||||
|
, rustPlatform
|
||||||
|
, pkg-config
|
||||||
|
, alsa-lib
|
||||||
|
}:
|
||||||
|
|
||||||
|
# luna-voiced — the push-to-talk voice loop for luna (Hermes Agent on mars).
|
||||||
|
# Wiring (secrets, whisper model, systemd user unit) lives in
|
||||||
|
# services/desktop/luna-voice.nix; this only builds the binary.
|
||||||
|
#
|
||||||
|
# cargoLock.lockFile rather than a cargoHash: the lockfile is committed, so a
|
||||||
|
# dependency bump is a visible diff instead of a hash nobody can read.
|
||||||
|
#
|
||||||
|
# alsa-lib is cpal's backend even on a pipewire box — pipewire's ALSA
|
||||||
|
# compatibility layer is what actually serves it, so there is no pipewire
|
||||||
|
# build dependency here.
|
||||||
|
rustPlatform.buildRustPackage {
|
||||||
|
pname = "luna-voice";
|
||||||
|
version = "0.1.0";
|
||||||
|
|
||||||
|
src = ./luna-voice;
|
||||||
|
cargoLock.lockFile = ./luna-voice/Cargo.lock;
|
||||||
|
|
||||||
|
nativeBuildInputs = [ pkg-config ];
|
||||||
|
buildInputs = [ alsa-lib ];
|
||||||
|
|
||||||
|
meta = {
|
||||||
|
description = "Push-to-talk voice loop for luna (Hermes Agent)";
|
||||||
|
mainProgram = "luna-voice";
|
||||||
|
platforms = lib.platforms.linux;
|
||||||
|
};
|
||||||
|
}
|
||||||
Generated
+2166
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,19 @@
|
|||||||
|
[package]
|
||||||
|
name = "luna-voice"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2021"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
anyhow = "1"
|
||||||
|
base64 = "0.22"
|
||||||
|
cpal = "0.15"
|
||||||
|
futures-util = { version = "0.3", default-features = false, features = ["sink", "std"] }
|
||||||
|
hound = "3"
|
||||||
|
reqwest = { version = "0.12", default-features = false, features = ["json", "stream", "rustls-tls"] }
|
||||||
|
serde = { version = "1", features = ["derive"] }
|
||||||
|
serde_json = "1"
|
||||||
|
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "sync", "time", "process", "io-util", "signal"] }
|
||||||
|
tokio-tungstenite = { version = "0.24", default-features = false, features = ["connect", "rustls-tls-webpki-roots"] }
|
||||||
|
|
||||||
|
[profile.release]
|
||||||
|
opt-level = 2
|
||||||
@@ -0,0 +1,265 @@
|
|||||||
|
use anyhow::{anyhow, Result};
|
||||||
|
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
|
||||||
|
use std::collections::VecDeque;
|
||||||
|
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
|
||||||
|
use std::sync::{Arc, Mutex};
|
||||||
|
|
||||||
|
/// What whisper wants. Everything captured is resampled to this before it
|
||||||
|
/// reaches disk.
|
||||||
|
pub const STT_RATE: u32 = 16_000;
|
||||||
|
/// What we ask ElevenLabs for (pcm_24000), resampled to the device on the way out.
|
||||||
|
pub const TTS_RATE: u32 = 24_000;
|
||||||
|
|
||||||
|
/// Mic. The stream runs for the whole session, not just while recording:
|
||||||
|
/// barge-in needs a live RMS while she is speaking, and restarting a cpal
|
||||||
|
/// stream per utterance costs ~100ms of dropped samples on pipewire.
|
||||||
|
///
|
||||||
|
/// cpal::Stream is !Send, so it is owned by a parked thread rather than by
|
||||||
|
/// this handle — which then holds nothing but atomics and a mutex and can be
|
||||||
|
/// shared across tokio tasks freely.
|
||||||
|
pub struct Capture {
|
||||||
|
pub rms: Arc<AtomicU32>,
|
||||||
|
recording: Arc<AtomicBool>,
|
||||||
|
buffer: Arc<Mutex<Vec<f32>>>,
|
||||||
|
rate: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Capture {
|
||||||
|
pub fn new() -> Result<Self> {
|
||||||
|
let rms = Arc::new(AtomicU32::new(0));
|
||||||
|
let recording = Arc::new(AtomicBool::new(false));
|
||||||
|
let buffer = Arc::new(Mutex::new(Vec::new()));
|
||||||
|
|
||||||
|
let (r, rec, buf) = (rms.clone(), recording.clone(), buffer.clone());
|
||||||
|
let rate = spawn_stream(move || {
|
||||||
|
let device = cpal::default_host()
|
||||||
|
.default_input_device()
|
||||||
|
.ok_or_else(|| anyhow!("no input device"))?;
|
||||||
|
let config = device.default_input_config()?;
|
||||||
|
let rate = config.sample_rate().0;
|
||||||
|
let channels = config.channels() as usize;
|
||||||
|
let stream = device.build_input_stream(
|
||||||
|
&config.into(),
|
||||||
|
move |data: &[f32], _: &cpal::InputCallbackInfo| {
|
||||||
|
// Downmix to mono up front; nothing downstream cares about stereo.
|
||||||
|
let mono: Vec<f32> = data
|
||||||
|
.chunks(channels)
|
||||||
|
.map(|f| f.iter().sum::<f32>() / channels as f32)
|
||||||
|
.collect();
|
||||||
|
let sum: f32 = mono.iter().map(|s| s * s).sum();
|
||||||
|
let level = (sum / mono.len().max(1) as f32).sqrt();
|
||||||
|
r.store(level.to_bits(), Ordering::Relaxed);
|
||||||
|
if rec.load(Ordering::Relaxed) {
|
||||||
|
if let Ok(mut b) = buf.lock() {
|
||||||
|
b.extend_from_slice(&mono);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|e| eprintln!("luna-voice: capture error: {e}"),
|
||||||
|
None,
|
||||||
|
)?;
|
||||||
|
stream.play()?;
|
||||||
|
Ok((stream, rate))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
Ok(Capture { rms, recording, buffer, rate })
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn level(&self) -> f32 {
|
||||||
|
f32::from_bits(self.rms.load(Ordering::Relaxed))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn start(&self) {
|
||||||
|
self.buffer.lock().map(|mut b| b.clear()).ok();
|
||||||
|
self.recording.store(true, Ordering::Relaxed);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Stops recording and returns what was captured, resampled to STT_RATE.
|
||||||
|
pub fn stop(&self) -> Vec<f32> {
|
||||||
|
self.recording.store(false, Ordering::Relaxed);
|
||||||
|
let raw = self
|
||||||
|
.buffer
|
||||||
|
.lock()
|
||||||
|
.map(|mut b| std::mem::take(&mut *b))
|
||||||
|
.unwrap_or_default();
|
||||||
|
resample(&raw, self.rate, STT_RATE)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Speaker. Fed from the TTS websocket; `clear` is what makes barge-in
|
||||||
|
/// instant rather than "after the current sentence".
|
||||||
|
pub struct Playback {
|
||||||
|
queue: Arc<Mutex<VecDeque<f32>>>,
|
||||||
|
rate: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Playback {
|
||||||
|
pub fn new() -> Result<Self> {
|
||||||
|
let queue: Arc<Mutex<VecDeque<f32>>> = Arc::new(Mutex::new(VecDeque::new()));
|
||||||
|
let q = queue.clone();
|
||||||
|
let rate = spawn_stream(move || {
|
||||||
|
let device = cpal::default_host()
|
||||||
|
.default_output_device()
|
||||||
|
.ok_or_else(|| anyhow!("no output device"))?;
|
||||||
|
let config = device.default_output_config()?;
|
||||||
|
let rate = config.sample_rate().0;
|
||||||
|
let channels = config.channels() as usize;
|
||||||
|
let stream = device.build_output_stream(
|
||||||
|
&config.into(),
|
||||||
|
move |data: &mut [f32], _: &cpal::OutputCallbackInfo| {
|
||||||
|
let mut q = match q.lock() {
|
||||||
|
Ok(q) => q,
|
||||||
|
Err(_) => return,
|
||||||
|
};
|
||||||
|
for frame in data.chunks_mut(channels) {
|
||||||
|
let s = q.pop_front().unwrap_or(0.0);
|
||||||
|
for out in frame.iter_mut() {
|
||||||
|
*out = s;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|e| eprintln!("luna-voice: playback error: {e}"),
|
||||||
|
None,
|
||||||
|
)?;
|
||||||
|
stream.play()?;
|
||||||
|
Ok((stream, rate))
|
||||||
|
})?;
|
||||||
|
Ok(Playback { queue, rate })
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn push(&self, pcm: &[f32]) {
|
||||||
|
let out = resample(pcm, TTS_RATE, self.rate);
|
||||||
|
if let Ok(mut q) = self.queue.lock() {
|
||||||
|
q.extend(out);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn clear(&self) {
|
||||||
|
self.queue.lock().map(|mut q| q.clear()).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn pending(&self) -> usize {
|
||||||
|
self.queue.lock().map(|q| q.len()).unwrap_or(0)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Short sine burst, so you know the key registered without looking at
|
||||||
|
/// anything. Hermes' voice mode has `beep_enabled` for the same reason:
|
||||||
|
/// push-to-talk with no acknowledgement is indistinguishable from broken.
|
||||||
|
///
|
||||||
|
/// Raised-cosine envelope — a bare sine starting at full amplitude clicks.
|
||||||
|
pub fn beep(&self, freq: f32, ms: u32) {
|
||||||
|
let n = (self.rate as f32 * ms as f32 / 1000.0) as usize;
|
||||||
|
let samples: Vec<f32> = (0..n)
|
||||||
|
.map(|i| {
|
||||||
|
let t = i as f32 / self.rate as f32;
|
||||||
|
let progress = i as f32 / n as f32;
|
||||||
|
let envelope = (std::f32::consts::PI * progress).sin();
|
||||||
|
(t * freq * std::f32::consts::TAU).sin() * envelope * 0.15
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
if let Ok(mut q) = self.queue.lock() {
|
||||||
|
q.extend(samples);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Builds a cpal stream on a dedicated thread that then parks forever,
|
||||||
|
/// keeping it alive. Returns the device's sample rate, or the build error —
|
||||||
|
/// a stream that fails here must surface at startup, not as silence later.
|
||||||
|
fn spawn_stream<F>(build: F) -> Result<u32>
|
||||||
|
where
|
||||||
|
F: FnOnce() -> Result<(cpal::Stream, u32)> + Send + 'static,
|
||||||
|
{
|
||||||
|
let (tx, rx) = std::sync::mpsc::channel::<Result<u32, String>>();
|
||||||
|
std::thread::spawn(move || match build() {
|
||||||
|
Ok((_stream, rate)) => {
|
||||||
|
let _ = tx.send(Ok(rate));
|
||||||
|
// Dropping `stream` would stop the device, so the thread holds it
|
||||||
|
// for the life of the process.
|
||||||
|
loop {
|
||||||
|
std::thread::park();
|
||||||
|
}
|
||||||
|
#[allow(unreachable_code)]
|
||||||
|
drop(_stream);
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
let _ = tx.send(Err(e.to_string()));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
rx.recv()
|
||||||
|
.map_err(|_| anyhow!("audio thread died during setup"))?
|
||||||
|
.map_err(|e| anyhow!(e))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Linear resampling. Speech at these rates does not justify a windowed sinc,
|
||||||
|
/// and rubato would add a dependency for inaudible gain.
|
||||||
|
pub fn resample(input: &[f32], from: u32, to: u32) -> Vec<f32> {
|
||||||
|
if from == to || input.is_empty() {
|
||||||
|
return input.to_vec();
|
||||||
|
}
|
||||||
|
let ratio = to as f64 / from as f64;
|
||||||
|
let out_len = (input.len() as f64 * ratio).round() as usize;
|
||||||
|
(0..out_len)
|
||||||
|
.map(|i| {
|
||||||
|
let src = i as f64 / ratio;
|
||||||
|
let j = src.floor() as usize;
|
||||||
|
let frac = (src - j as f64) as f32;
|
||||||
|
let a = input.get(j).copied().unwrap_or(0.0);
|
||||||
|
let b = input.get(j + 1).copied().unwrap_or(a);
|
||||||
|
a + (b - a) * frac
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// 16-bit mono WAV, which is the only thing whisper-cli accepts without
|
||||||
|
/// converting first.
|
||||||
|
pub fn write_wav(path: &std::path::Path, samples: &[f32], rate: u32) -> Result<()> {
|
||||||
|
let spec = hound::WavSpec {
|
||||||
|
channels: 1,
|
||||||
|
sample_rate: rate,
|
||||||
|
bits_per_sample: 16,
|
||||||
|
sample_format: hound::SampleFormat::Int,
|
||||||
|
};
|
||||||
|
let mut w = hound::WavWriter::create(path, spec)?;
|
||||||
|
for s in samples {
|
||||||
|
w.write_sample((s.clamp(-1.0, 1.0) * i16::MAX as f32) as i16)?;
|
||||||
|
}
|
||||||
|
w.finalize()?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Signed 16-bit little-endian, as ElevenLabs sends pcm_*.
|
||||||
|
pub fn pcm16_to_f32(bytes: &[u8]) -> Vec<f32> {
|
||||||
|
bytes
|
||||||
|
.chunks_exact(2)
|
||||||
|
.map(|b| i16::from_le_bytes([b[0], b[1]]) as f32 / i16::MAX as f32)
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn resample_keeps_length_ratio() {
|
||||||
|
let input = vec![0.0f32; 1000];
|
||||||
|
assert_eq!(resample(&input, 48_000, 16_000).len(), 333);
|
||||||
|
assert_eq!(resample(&input, 24_000, 48_000).len(), 2000);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn resample_is_identity_at_equal_rates() {
|
||||||
|
let input = vec![0.1, -0.4, 0.9];
|
||||||
|
assert_eq!(resample(&input, 16_000, 16_000), input);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pcm16_roundtrips_through_f32() {
|
||||||
|
let bytes = [0x00, 0x00, 0xff, 0x7f, 0x01, 0x80];
|
||||||
|
let got = pcm16_to_f32(&bytes);
|
||||||
|
assert_eq!(got.len(), 3);
|
||||||
|
assert!((got[0]).abs() < 1e-6);
|
||||||
|
assert!((got[1] - 1.0).abs() < 1e-3);
|
||||||
|
assert!((got[2] + 1.0).abs() < 1e-3);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,123 @@
|
|||||||
|
use anyhow::{bail, Result};
|
||||||
|
use futures_util::StreamExt;
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use tokio::sync::mpsc;
|
||||||
|
|
||||||
|
#[derive(Clone, Serialize, Deserialize, Debug)]
|
||||||
|
pub struct Message {
|
||||||
|
pub role: String,
|
||||||
|
pub content: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Message {
|
||||||
|
pub fn user(content: impl Into<String>) -> Self {
|
||||||
|
Message { role: "user".into(), content: content.into() }
|
||||||
|
}
|
||||||
|
pub fn assistant(content: impl Into<String>) -> Self {
|
||||||
|
Message { role: "assistant".into(), content: content.into() }
|
||||||
|
}
|
||||||
|
pub fn system(content: impl Into<String>) -> Self {
|
||||||
|
Message { role: "system".into(), content: content.into() }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Streams a reply from Hermes' OpenAI-compatible API, pushing text deltas
|
||||||
|
/// into `tx` as they arrive.
|
||||||
|
///
|
||||||
|
/// /v1/chat/completions with client-side history, not /v1/responses with
|
||||||
|
/// previous_response_id: the Responses API keeps the transcript server-side,
|
||||||
|
/// which is nicer, but this shape is the one every OpenAI client exercises so
|
||||||
|
/// it is the one least likely to differ between Hermes versions. Her memory
|
||||||
|
/// and skills are unaffected either way — those are not conversation state.
|
||||||
|
pub async fn stream_reply(
|
||||||
|
client: &reqwest::Client,
|
||||||
|
base_url: &str,
|
||||||
|
key: &str,
|
||||||
|
model: &str,
|
||||||
|
history: &[Message],
|
||||||
|
tx: mpsc::Sender<String>,
|
||||||
|
) -> Result<String> {
|
||||||
|
let body = serde_json::json!({
|
||||||
|
"model": model,
|
||||||
|
"messages": history,
|
||||||
|
"stream": true,
|
||||||
|
});
|
||||||
|
|
||||||
|
let resp = client
|
||||||
|
.post(format!("{}/v1/chat/completions", base_url.trim_end_matches('/')))
|
||||||
|
.bearer_auth(key)
|
||||||
|
.json(&body)
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if !resp.status().is_success() {
|
||||||
|
let status = resp.status();
|
||||||
|
let text = resp.text().await.unwrap_or_default();
|
||||||
|
bail!("hermes {status}: {}", text.trim());
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut stream = resp.bytes_stream();
|
||||||
|
let mut pending = String::new();
|
||||||
|
let mut full = String::new();
|
||||||
|
|
||||||
|
while let Some(chunk) = stream.next().await {
|
||||||
|
pending.push_str(&String::from_utf8_lossy(&chunk?));
|
||||||
|
// SSE frames are separated by a blank line; a chunk boundary can land
|
||||||
|
// mid-frame, so only complete frames are consumed.
|
||||||
|
while let Some(idx) = pending.find("\n\n") {
|
||||||
|
let frame: String = pending.drain(..idx + 2).collect();
|
||||||
|
for line in frame.lines() {
|
||||||
|
let Some(data) = line.strip_prefix("data:") else { continue };
|
||||||
|
let data = data.trim();
|
||||||
|
if data == "[DONE]" {
|
||||||
|
return Ok(full);
|
||||||
|
}
|
||||||
|
if let Some(delta) = parse_delta(data) {
|
||||||
|
full.push_str(&delta);
|
||||||
|
// A closed receiver means the reply was cancelled (barge-in).
|
||||||
|
if tx.send(delta).await.is_err() {
|
||||||
|
return Ok(full);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(full)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Pulls choices[0].delta.content out of one SSE data payload. Tool-call
|
||||||
|
/// frames and role-only frames carry no content and are skipped.
|
||||||
|
fn parse_delta(data: &str) -> Option<String> {
|
||||||
|
let v: serde_json::Value = serde_json::from_str(data).ok()?;
|
||||||
|
let content = v
|
||||||
|
.get("choices")?
|
||||||
|
.get(0)?
|
||||||
|
.get("delta")?
|
||||||
|
.get("content")?
|
||||||
|
.as_str()?;
|
||||||
|
(!content.is_empty()).then(|| content.to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::parse_delta;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn extracts_content_delta() {
|
||||||
|
let d = r#"{"choices":[{"delta":{"content":"hi"}}]}"#;
|
||||||
|
assert_eq!(parse_delta(d).as_deref(), Some("hi"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ignores_role_only_and_tool_frames() {
|
||||||
|
assert!(parse_delta(r#"{"choices":[{"delta":{"role":"assistant"}}]}"#).is_none());
|
||||||
|
assert!(parse_delta(r#"{"choices":[{"delta":{"tool_calls":[]}}]}"#).is_none());
|
||||||
|
assert!(parse_delta(r#"{"choices":[{"delta":{"content":""}}]}"#).is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ignores_malformed_json_rather_than_failing_the_stream() {
|
||||||
|
assert!(parse_delta("not json").is_none());
|
||||||
|
assert!(parse_delta("{}").is_none());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
use anyhow::{Context, Result};
|
||||||
|
use std::path::PathBuf;
|
||||||
|
|
||||||
|
/// Everything the daemon needs, all from the environment so the NixOS module
|
||||||
|
/// is the only place that knows paths. Secrets arrive as *_FILE (sops paths),
|
||||||
|
/// never as values — an env var is readable from /proc for anything of the
|
||||||
|
/// user's, a 0400 file is not.
|
||||||
|
pub struct Config {
|
||||||
|
pub hermes_url: String,
|
||||||
|
pub hermes_key: String,
|
||||||
|
pub model: String,
|
||||||
|
pub system_prompt: Option<String>,
|
||||||
|
|
||||||
|
pub eleven_key: String,
|
||||||
|
pub voice_id: String,
|
||||||
|
pub tts_model: String,
|
||||||
|
|
||||||
|
pub whisper_bin: String,
|
||||||
|
pub whisper_model: PathBuf,
|
||||||
|
|
||||||
|
pub socket_path: PathBuf,
|
||||||
|
/// Speech during playback louder than noise_floor * this counts as barge-in.
|
||||||
|
pub barge_in_mult: f32,
|
||||||
|
pub barge_in_grace_ms: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
fn env(key: &str) -> Result<String> {
|
||||||
|
std::env::var(key).with_context(|| format!("{key} is not set"))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn env_or(key: &str, default: &str) -> String {
|
||||||
|
std::env::var(key).unwrap_or_else(|_| default.to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reads a secret out of a file and trims it. A trailing newline in a bearer
|
||||||
|
/// token produces a 401 whose message says nothing about whitespace.
|
||||||
|
fn secret(key: &str) -> Result<String> {
|
||||||
|
let path = env(key)?;
|
||||||
|
let raw = std::fs::read_to_string(&path).with_context(|| format!("reading {path}"))?;
|
||||||
|
Ok(raw.trim().to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Config {
|
||||||
|
pub fn from_env() -> Result<Self> {
|
||||||
|
let runtime = env_or(
|
||||||
|
"XDG_RUNTIME_DIR",
|
||||||
|
&format!("/run/user/{}", unsafe { libc_getuid() }),
|
||||||
|
);
|
||||||
|
Ok(Config {
|
||||||
|
hermes_url: env_or("LUNA_HERMES_URL", "http://mars.orbit.sol:8642"),
|
||||||
|
hermes_key: secret("LUNA_HERMES_KEY_FILE")?,
|
||||||
|
model: env_or("LUNA_MODEL", "hermes"),
|
||||||
|
system_prompt: std::env::var("LUNA_SYSTEM_PROMPT").ok(),
|
||||||
|
|
||||||
|
eleven_key: secret("LUNA_ELEVENLABS_KEY_FILE")?,
|
||||||
|
voice_id: env("LUNA_VOICE_ID")?,
|
||||||
|
tts_model: env_or("LUNA_TTS_MODEL", "eleven_flash_v2_5"),
|
||||||
|
|
||||||
|
whisper_bin: env_or("LUNA_WHISPER_BIN", "whisper-cli"),
|
||||||
|
whisper_model: PathBuf::from(env("LUNA_WHISPER_MODEL")?),
|
||||||
|
|
||||||
|
socket_path: PathBuf::from(env_or(
|
||||||
|
"LUNA_SOCKET",
|
||||||
|
&format!("{runtime}/luna-voiced.sock"),
|
||||||
|
)),
|
||||||
|
barge_in_mult: env_or("LUNA_BARGE_IN_MULT", "3.0").parse().unwrap_or(3.0),
|
||||||
|
barge_in_grace_ms: env_or("LUNA_BARGE_IN_GRACE_MS", "500")
|
||||||
|
.parse()
|
||||||
|
.unwrap_or(500),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Avoids pulling in the libc crate for one call.
|
||||||
|
unsafe fn libc_getuid() -> u32 {
|
||||||
|
extern "C" {
|
||||||
|
fn getuid() -> u32;
|
||||||
|
}
|
||||||
|
getuid()
|
||||||
|
}
|
||||||
@@ -0,0 +1,183 @@
|
|||||||
|
use anyhow::Result;
|
||||||
|
use serde::Serialize;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
|
||||||
|
use tokio::net::{UnixListener, UnixStream};
|
||||||
|
use tokio::sync::{broadcast, mpsc};
|
||||||
|
|
||||||
|
use crate::visemes::Viseme;
|
||||||
|
|
||||||
|
/// Daemon -> shell. Newline-delimited JSON; the shell reads it with a
|
||||||
|
/// SplitParser and never has to spawn anything.
|
||||||
|
#[derive(Serialize, Clone, Debug)]
|
||||||
|
#[serde(tag = "type", rename_all = "snake_case")]
|
||||||
|
pub enum Event {
|
||||||
|
State { value: &'static str },
|
||||||
|
/// Emitted when the first PCM frame reaches the speaker. Viseme offsets
|
||||||
|
/// are relative to THIS, so the two processes never compare clocks.
|
||||||
|
SpeechStart,
|
||||||
|
Visemes { seq: Vec<Viseme> },
|
||||||
|
Amplitude { rms: f32 },
|
||||||
|
/// What was said, and by whom. Her side is emitted sentence by sentence
|
||||||
|
/// as it goes to the synthesiser, so subtitles can lead the audio the
|
||||||
|
/// same way the visemes do.
|
||||||
|
Caption { speaker: &'static str, text: String },
|
||||||
|
SpeechStop { reason: &'static str },
|
||||||
|
Error { message: String },
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Shell -> daemon.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub enum Command {
|
||||||
|
Ptt { down: bool },
|
||||||
|
Cancel,
|
||||||
|
/// Run a turn from typed text, skipping the microphone entirely. This is
|
||||||
|
/// how the brain -> TTS -> speaker half gets tested without speaking, and
|
||||||
|
/// how you tell a broken mic from a broken pipeline.
|
||||||
|
Say { text: String },
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct Ipc {
|
||||||
|
pub events: broadcast::Sender<Event>,
|
||||||
|
path: PathBuf,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Ipc {
|
||||||
|
/// Binds the socket and serves it until dropped. A stale socket from a
|
||||||
|
/// crashed run is removed first — bind() would fail with EADDRINUSE on a
|
||||||
|
/// file nothing is listening to.
|
||||||
|
pub fn bind(path: &Path, commands: mpsc::Sender<Command>) -> Result<Self> {
|
||||||
|
if path.exists() {
|
||||||
|
let _ = std::fs::remove_file(path);
|
||||||
|
}
|
||||||
|
if let Some(parent) = path.parent() {
|
||||||
|
std::fs::create_dir_all(parent)?;
|
||||||
|
}
|
||||||
|
let listener = UnixListener::bind(path)?;
|
||||||
|
let (events, _) = broadcast::channel(256);
|
||||||
|
|
||||||
|
let tx = events.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
loop {
|
||||||
|
match listener.accept().await {
|
||||||
|
Ok((stream, _)) => {
|
||||||
|
let rx = tx.subscribe();
|
||||||
|
let cmds = commands.clone();
|
||||||
|
tokio::spawn(client(stream, rx, cmds));
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("luna-voice: accept failed: {e}");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
Ok(Ipc { events, path: path.to_path_buf() })
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn emit(&self, event: Event) {
|
||||||
|
// An error here only means nobody is connected, which is normal.
|
||||||
|
let _ = self.events.send(event);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Drop for Ipc {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
let _ = std::fs::remove_file(&self.path);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn client(
|
||||||
|
stream: UnixStream,
|
||||||
|
mut rx: broadcast::Receiver<Event>,
|
||||||
|
commands: mpsc::Sender<Command>,
|
||||||
|
) {
|
||||||
|
let (read, mut write) = stream.into_split();
|
||||||
|
let mut lines = BufReader::new(read).lines();
|
||||||
|
|
||||||
|
loop {
|
||||||
|
tokio::select! {
|
||||||
|
event = rx.recv() => {
|
||||||
|
let Ok(event) = event else { return };
|
||||||
|
let Ok(mut line) = serde_json::to_string(&event) else { continue };
|
||||||
|
line.push('\n');
|
||||||
|
if write.write_all(line.as_bytes()).await.is_err() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
line = lines.next_line() => {
|
||||||
|
match line {
|
||||||
|
Ok(Some(line)) => {
|
||||||
|
if let Some(cmd) = parse_command(&line) {
|
||||||
|
let _ = commands.send(cmd).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_ => return,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_command(line: &str) -> Option<Command> {
|
||||||
|
let v: serde_json::Value = serde_json::from_str(line).ok()?;
|
||||||
|
match v.get("type")?.as_str()? {
|
||||||
|
"ptt" => Some(Command::Ptt {
|
||||||
|
down: v.get("down").and_then(|d| d.as_bool()).unwrap_or(false),
|
||||||
|
}),
|
||||||
|
"cancel" => Some(Command::Cancel),
|
||||||
|
"say" => Some(Command::Say {
|
||||||
|
text: v.get("text")?.as_str()?.to_string(),
|
||||||
|
}),
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Client side of the same socket, for the `luna-voice ptt` subcommand that
|
||||||
|
/// Hyprland binds a key to before the quickshell widget exists.
|
||||||
|
pub async fn send_command(path: &Path, line: &str) -> Result<()> {
|
||||||
|
let mut stream = UnixStream::connect(path).await?;
|
||||||
|
stream.write_all(format!("{line}\n").as_bytes()).await?;
|
||||||
|
stream.flush().await?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parses_ptt_commands() {
|
||||||
|
assert!(matches!(
|
||||||
|
parse_command(r#"{"type":"ptt","down":true}"#),
|
||||||
|
Some(Command::Ptt { down: true })
|
||||||
|
));
|
||||||
|
assert!(matches!(
|
||||||
|
parse_command(r#"{"type":"ptt","down":false}"#),
|
||||||
|
Some(Command::Ptt { down: false })
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ptt_without_down_defaults_to_release() {
|
||||||
|
assert!(matches!(
|
||||||
|
parse_command(r#"{"type":"ptt"}"#),
|
||||||
|
Some(Command::Ptt { down: false })
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ignores_unknown_and_malformed() {
|
||||||
|
assert!(parse_command(r#"{"type":"nope"}"#).is_none());
|
||||||
|
assert!(parse_command("garbage").is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn events_serialise_with_a_type_tag() {
|
||||||
|
let json = serde_json::to_string(&Event::State { value: "listening" }).unwrap();
|
||||||
|
assert_eq!(json, r#"{"type":"state","value":"listening"}"#);
|
||||||
|
let json = serde_json::to_string(&Event::SpeechStart).unwrap();
|
||||||
|
assert_eq!(json, r#"{"type":"speech_start"}"#);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,336 @@
|
|||||||
|
//! luna-voiced — push-to-talk voice loop for luna (the Hermes Agent on mars).
|
||||||
|
//!
|
||||||
|
//! mic -> whisper (local, vulkan) -> Hermes /v1/chat/completions -> ElevenLabs
|
||||||
|
//! streaming TTS -> speaker, with the speaking state, amplitude and viseme
|
||||||
|
//! timeline published on a unix socket for the quickshell avatar.
|
||||||
|
//!
|
||||||
|
//! Nothing here talks to the compositor: push-to-talk arrives over the same
|
||||||
|
//! socket, from either `luna-voice ptt` or the shell's GlobalShortcut.
|
||||||
|
|
||||||
|
mod audio;
|
||||||
|
mod brain;
|
||||||
|
mod config;
|
||||||
|
mod ipc;
|
||||||
|
mod stt;
|
||||||
|
mod tts;
|
||||||
|
mod visemes;
|
||||||
|
|
||||||
|
use anyhow::Result;
|
||||||
|
use serde_json;
|
||||||
|
use config::Config;
|
||||||
|
use ipc::{Command, Event};
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::time::Duration;
|
||||||
|
use tokio::sync::mpsc;
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() -> Result<()> {
|
||||||
|
let mut args = std::env::args().skip(1);
|
||||||
|
match args.next().as_deref() {
|
||||||
|
None | Some("daemon") => daemon().await,
|
||||||
|
// Fire-and-forget clients so a keybind costs one short-lived process
|
||||||
|
// and never blocks the compositor.
|
||||||
|
Some("ptt-down") => client(r#"{"type":"ptt","down":true}"#).await,
|
||||||
|
Some("ptt-up") => client(r#"{"type":"ptt","down":false}"#).await,
|
||||||
|
Some("cancel") => client(r#"{"type":"cancel"}"#).await,
|
||||||
|
// The debugging tool: everything the avatar will eventually see,
|
||||||
|
// printed. `journalctl -fu luna-voiced` shows only what the daemon
|
||||||
|
// chose to log; this shows the actual protocol.
|
||||||
|
Some("watch") => watch().await,
|
||||||
|
Some("say") => {
|
||||||
|
let text: Vec<String> = args.collect();
|
||||||
|
if text.is_empty() {
|
||||||
|
eprintln!("luna-voice: say what?");
|
||||||
|
std::process::exit(2);
|
||||||
|
}
|
||||||
|
let payload = serde_json::json!({ "type": "say", "text": text.join(" ") });
|
||||||
|
client(&payload.to_string()).await
|
||||||
|
}
|
||||||
|
Some(other) => {
|
||||||
|
eprintln!("luna-voice: unknown command {other:?}");
|
||||||
|
eprintln!("usage: luna-voice [daemon|ptt-down|ptt-up|cancel|watch]");
|
||||||
|
std::process::exit(2);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn socket_path() -> std::path::PathBuf {
|
||||||
|
let runtime = std::env::var("XDG_RUNTIME_DIR").unwrap_or_else(|_| "/run/user/1000".into());
|
||||||
|
std::env::var("LUNA_SOCKET")
|
||||||
|
.map(std::path::PathBuf::from)
|
||||||
|
.unwrap_or_else(|_| std::path::Path::new(&runtime).join("luna-voiced.sock"))
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn watch() -> Result<()> {
|
||||||
|
use tokio::io::{AsyncBufReadExt, BufReader};
|
||||||
|
let path = socket_path();
|
||||||
|
let stream = tokio::net::UnixStream::connect(&path)
|
||||||
|
.await
|
||||||
|
.map_err(|e| anyhow::anyhow!("{}: {e} (is luna-voiced running?)", path.display()))?;
|
||||||
|
let mut lines = BufReader::new(stream).lines();
|
||||||
|
eprintln!("luna-voice: watching {}", path.display());
|
||||||
|
while let Some(line) = lines.next_line().await? {
|
||||||
|
println!("{line}");
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn client(line: &str) -> Result<()> {
|
||||||
|
ipc::send_command(&socket_path(), line).await
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Errors go to the desktop, not just the journal — terra runs its own
|
||||||
|
/// notification daemon (HyprChrome/Widgets/Notifications), so this is the one
|
||||||
|
/// channel that is visible while you are talking rather than debugging.
|
||||||
|
/// Best-effort: a missing notify-send must never take the daemon down.
|
||||||
|
fn notify(body: &str) {
|
||||||
|
let _ = std::process::Command::new("notify-send")
|
||||||
|
.args(["-a", "luna", "-u", "critical", "luna voice", body])
|
||||||
|
.spawn();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Where the loop is. One utterance at a time by construction: the state
|
||||||
|
/// machine is a single task, so there is no way to start a second reply while
|
||||||
|
/// one is in flight.
|
||||||
|
async fn daemon() -> Result<()> {
|
||||||
|
let cfg = Arc::new(Config::from_env()?);
|
||||||
|
let capture = audio::Capture::new()?;
|
||||||
|
let playback = Arc::new(audio::Playback::new()?);
|
||||||
|
|
||||||
|
let (cmd_tx, mut cmd_rx) = mpsc::channel::<Command>(16);
|
||||||
|
let ipc = Arc::new(ipc::Ipc::bind(&cfg.socket_path, cmd_tx)?);
|
||||||
|
eprintln!("luna-voice: listening on {}", cfg.socket_path.display());
|
||||||
|
|
||||||
|
let http = reqwest::Client::builder()
|
||||||
|
// Her replies can involve tool calls; the default 30s would cut those
|
||||||
|
// off mid-thought.
|
||||||
|
.timeout(Duration::from_secs(300))
|
||||||
|
.build()?;
|
||||||
|
|
||||||
|
let mut history: Vec<brain::Message> = Vec::new();
|
||||||
|
if let Some(prompt) = &cfg.system_prompt {
|
||||||
|
history.push(brain::Message::system(prompt.clone()));
|
||||||
|
}
|
||||||
|
|
||||||
|
ipc.emit(Event::State { value: "idle" });
|
||||||
|
|
||||||
|
// Her first reply after the container starts costs ~16s against ~5s warm,
|
||||||
|
// and the first one is always the one you are sitting there waiting for.
|
||||||
|
// Spending one throwaway token at login moves that cost off the
|
||||||
|
// conversation. Failure is ignored on purpose: mars may simply be down,
|
||||||
|
// which must not stop the daemon from coming up.
|
||||||
|
tokio::spawn({
|
||||||
|
let (http, url, key, model) =
|
||||||
|
(http.clone(), cfg.hermes_url.clone(), cfg.hermes_key.clone(), cfg.model.clone());
|
||||||
|
async move {
|
||||||
|
let (tx, mut rx) = mpsc::channel(4);
|
||||||
|
tokio::spawn(async move { while rx.recv().await.is_some() {} });
|
||||||
|
let warm = [brain::Message::user("hi")];
|
||||||
|
match brain::stream_reply(&http, &url, &key, &model, &warm, tx).await {
|
||||||
|
Ok(_) => eprintln!("luna-voice: warmed up"),
|
||||||
|
Err(e) => eprintln!("luna-voice: warmup skipped: {e}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
while let Some(cmd) = cmd_rx.recv().await {
|
||||||
|
match cmd {
|
||||||
|
Command::Ptt { down: true } => {
|
||||||
|
// Anything still playing is cut off: pressing the key while
|
||||||
|
// she talks means you want to talk instead.
|
||||||
|
playback.clear();
|
||||||
|
capture.start();
|
||||||
|
playback.beep(880.0, 70);
|
||||||
|
ipc.emit(Event::State { value: "listening" });
|
||||||
|
}
|
||||||
|
Command::Ptt { down: false } => {
|
||||||
|
let samples = capture.stop();
|
||||||
|
playback.beep(587.0, 70);
|
||||||
|
ipc.emit(Event::State { value: "thinking" });
|
||||||
|
|
||||||
|
let heard =
|
||||||
|
match stt::transcribe(&cfg.whisper_bin, &cfg.whisper_model, &samples).await {
|
||||||
|
Ok(h) => h,
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("luna-voice: {e:#}");
|
||||||
|
ipc.emit(Event::Error { message: e.to_string() });
|
||||||
|
notify(&e.to_string());
|
||||||
|
ipc.emit(Event::State { value: "idle" });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
if heard.is_empty() {
|
||||||
|
// Distinct low tone: "I heard nothing" and "I am
|
||||||
|
// thinking" must not feel the same, or a dead mic reads
|
||||||
|
// as a slow reply.
|
||||||
|
playback.beep(220.0, 120);
|
||||||
|
eprintln!("luna-voice: nothing heard ({} samples)", samples.len());
|
||||||
|
ipc.emit(Event::SpeechStop { reason: "nothing_heard" });
|
||||||
|
ipc.emit(Event::State { value: "idle" });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
run_turn(&cfg, &http, &ipc, &capture, &playback, &mut cmd_rx, &mut history, heard)
|
||||||
|
.await;
|
||||||
|
}
|
||||||
|
Command::Say { text } => {
|
||||||
|
ipc.emit(Event::State { value: "thinking" });
|
||||||
|
run_turn(&cfg, &http, &ipc, &capture, &playback, &mut cmd_rx, &mut history, text)
|
||||||
|
.await;
|
||||||
|
}
|
||||||
|
Command::Cancel => {
|
||||||
|
playback.clear();
|
||||||
|
ipc.emit(Event::SpeechStop { reason: "cancelled" });
|
||||||
|
ipc.emit(Event::State { value: "idle" });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Runs one turn and reports failure through every channel that has a
|
||||||
|
/// listener: journal, socket, desktop notification.
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
|
async fn run_turn(
|
||||||
|
cfg: &Config,
|
||||||
|
http: &reqwest::Client,
|
||||||
|
ipc: &Arc<ipc::Ipc>,
|
||||||
|
capture: &audio::Capture,
|
||||||
|
playback: &Arc<audio::Playback>,
|
||||||
|
cmd_rx: &mut mpsc::Receiver<Command>,
|
||||||
|
history: &mut Vec<brain::Message>,
|
||||||
|
heard: String,
|
||||||
|
) {
|
||||||
|
if let Err(e) = turn(cfg, http, ipc, capture, playback, cmd_rx, history, heard).await {
|
||||||
|
eprintln!("luna-voice: {e:#}");
|
||||||
|
ipc.emit(Event::Error { message: e.to_string() });
|
||||||
|
notify(&e.to_string());
|
||||||
|
}
|
||||||
|
ipc.emit(Event::State { value: "idle" });
|
||||||
|
}
|
||||||
|
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
|
async fn turn(
|
||||||
|
cfg: &Config,
|
||||||
|
http: &reqwest::Client,
|
||||||
|
ipc: &Arc<ipc::Ipc>,
|
||||||
|
capture: &audio::Capture,
|
||||||
|
playback: &Arc<audio::Playback>,
|
||||||
|
cmd_rx: &mut mpsc::Receiver<Command>,
|
||||||
|
history: &mut Vec<brain::Message>,
|
||||||
|
heard: String,
|
||||||
|
) -> Result<()> {
|
||||||
|
eprintln!("luna-voice: heard {heard:?}");
|
||||||
|
ipc.emit(Event::Caption { speaker: "user", text: heard.clone() });
|
||||||
|
history.push(brain::Message::user(heard));
|
||||||
|
|
||||||
|
// Noise floor sampled before she starts, so barge-in adapts to the room
|
||||||
|
// rather than to a constant.
|
||||||
|
let floor = capture.level().max(0.005);
|
||||||
|
|
||||||
|
let (sentence_tx, sentence_rx) = mpsc::channel::<String>(16);
|
||||||
|
let (event_tx, mut event_rx) = mpsc::channel::<tts::SpeechEvent>(64);
|
||||||
|
|
||||||
|
let speech = tokio::spawn({
|
||||||
|
let tts_cfg = tts::TtsConfig {
|
||||||
|
key: cfg.eleven_key.clone(),
|
||||||
|
voice_id: cfg.voice_id.clone(),
|
||||||
|
model: cfg.tts_model.clone(),
|
||||||
|
};
|
||||||
|
async move { tts::speak(&tts_cfg, sentence_rx, event_tx).await }
|
||||||
|
});
|
||||||
|
|
||||||
|
// Audio and visemes flow to the speaker/shell on their own task so the
|
||||||
|
// reply stream is never blocked by playback.
|
||||||
|
let player = tokio::spawn({
|
||||||
|
let ipc = ipc.clone();
|
||||||
|
let playback = playback.clone();
|
||||||
|
async move {
|
||||||
|
let mut started = false;
|
||||||
|
while let Some(ev) = event_rx.recv().await {
|
||||||
|
match ev {
|
||||||
|
tts::SpeechEvent::Audio(pcm) => {
|
||||||
|
if !started {
|
||||||
|
started = true;
|
||||||
|
ipc.emit(Event::SpeechStart);
|
||||||
|
ipc.emit(Event::State { value: "speaking" });
|
||||||
|
}
|
||||||
|
playback.push(&pcm);
|
||||||
|
}
|
||||||
|
// A chunk whose alignment held nothing visible still
|
||||||
|
// arrives; forwarding it would make the shell reset the
|
||||||
|
// mouth mid-word.
|
||||||
|
tts::SpeechEvent::Visemes(seq) if !seq.is_empty() => {
|
||||||
|
ipc.emit(Event::Visemes { seq })
|
||||||
|
}
|
||||||
|
tts::SpeechEvent::Visemes(_) => {}
|
||||||
|
tts::SpeechEvent::Done => break,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
let (delta_tx, mut delta_rx) = mpsc::channel::<String>(64);
|
||||||
|
let reply = tokio::spawn({
|
||||||
|
let http = http.clone();
|
||||||
|
let (url, key, model) = (cfg.hermes_url.clone(), cfg.hermes_key.clone(), cfg.model.clone());
|
||||||
|
let history = history.clone();
|
||||||
|
async move { brain::stream_reply(&http, &url, &key, &model, &history, delta_tx).await }
|
||||||
|
});
|
||||||
|
|
||||||
|
let mut splitter = tts::SentenceSplitter::new();
|
||||||
|
while let Some(delta) = delta_rx.recv().await {
|
||||||
|
for sentence in splitter.push(&delta) {
|
||||||
|
ipc.emit(Event::Caption { speaker: "luna", text: sentence.clone() });
|
||||||
|
if sentence_tx.send(sentence).await.is_err() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(tail) = splitter.finish() {
|
||||||
|
ipc.emit(Event::Caption { speaker: "luna", text: tail.clone() });
|
||||||
|
let _ = sentence_tx.send(tail).await;
|
||||||
|
}
|
||||||
|
drop(sentence_tx);
|
||||||
|
|
||||||
|
let full = reply.await??;
|
||||||
|
eprintln!("luna-voice: said {:?}", full.trim());
|
||||||
|
if !full.trim().is_empty() {
|
||||||
|
history.push(brain::Message::assistant(full));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wait for the speaker to drain, watching for barge-in. Polling the queue
|
||||||
|
// is enough: cpal gives no drain callback, and 50ms of latency on "she
|
||||||
|
// finished" is invisible.
|
||||||
|
let grace = tokio::time::Instant::now() + Duration::from_millis(cfg.barge_in_grace_ms);
|
||||||
|
loop {
|
||||||
|
if speech.is_finished() && playback.pending() == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if tokio::time::Instant::now() > grace
|
||||||
|
&& capture.level() > floor * cfg.barge_in_mult
|
||||||
|
{
|
||||||
|
playback.clear();
|
||||||
|
ipc.emit(Event::SpeechStop { reason: "barge_in" });
|
||||||
|
// She is told she was cut off, so the next turn can react to it
|
||||||
|
// rather than carrying on obliviously.
|
||||||
|
history.push(brain::Message::system(
|
||||||
|
"[The user interrupted you mid-sentence.]",
|
||||||
|
));
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
// A cancel arriving mid-playback must be honoured now, not after.
|
||||||
|
if let Ok(Command::Cancel) = cmd_rx.try_recv() {
|
||||||
|
playback.clear();
|
||||||
|
ipc.emit(Event::SpeechStop { reason: "cancelled" });
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
ipc.emit(Event::Amplitude { rms: capture.level() });
|
||||||
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
speech.abort();
|
||||||
|
player.abort();
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
use crate::audio;
|
||||||
|
use anyhow::{bail, Result};
|
||||||
|
use std::path::Path;
|
||||||
|
use tokio::process::Command;
|
||||||
|
|
||||||
|
/// Shells out to whisper-cli rather than linking whisper.cpp: nixpkgs already
|
||||||
|
/// ships whisper-cpp-vulkan built against this box's GPU stack, and a bundled
|
||||||
|
/// build would have to rediscover that at cargo time.
|
||||||
|
pub async fn transcribe(bin: &str, model: &Path, samples: &[f32]) -> Result<String> {
|
||||||
|
// Whisper hallucinates confident words into silence — "you", "Thank
|
||||||
|
// you.", "Bye." — which then reach luna as if spoken. Duration alone does
|
||||||
|
// not catch it (a held key over a quiet room is long AND empty), so gate
|
||||||
|
// on level too.
|
||||||
|
if samples.len() < (audio::STT_RATE as usize) / 3 || !has_speech(samples) {
|
||||||
|
return Ok(String::new());
|
||||||
|
}
|
||||||
|
|
||||||
|
let dir = std::env::var("XDG_RUNTIME_DIR").unwrap_or_else(|_| "/tmp".into());
|
||||||
|
let wav = Path::new(&dir).join(format!("luna-stt-{}.wav", std::process::id()));
|
||||||
|
audio::write_wav(&wav, samples, audio::STT_RATE)?;
|
||||||
|
|
||||||
|
let out = Command::new(bin)
|
||||||
|
.args(["-m"])
|
||||||
|
.arg(model)
|
||||||
|
.args(["-f"])
|
||||||
|
.arg(&wav)
|
||||||
|
// -nt: transcript only, no [00:00:00] prefixes. -np: no progress
|
||||||
|
// banner. Together they make stdout exactly the text.
|
||||||
|
.args(["-nt", "-np"])
|
||||||
|
.output()
|
||||||
|
.await?;
|
||||||
|
let _ = std::fs::remove_file(&wav);
|
||||||
|
|
||||||
|
if !out.status.success() {
|
||||||
|
bail!(
|
||||||
|
"whisper-cli failed: {}",
|
||||||
|
String::from_utf8_lossy(&out.stderr).trim()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
Ok(clean(&String::from_utf8_lossy(&out.stdout)))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True if any 100ms window is loud enough to be a voice. Peak rather than
|
||||||
|
/// overall RMS: a short sentence inside a long silent recording averages down
|
||||||
|
/// to nothing, and that is the common case for a held key.
|
||||||
|
fn has_speech(samples: &[f32]) -> bool {
|
||||||
|
const FLOOR: f32 = 0.015;
|
||||||
|
let window = (audio::STT_RATE as usize) / 10;
|
||||||
|
samples.chunks(window).any(|w| {
|
||||||
|
let sum: f32 = w.iter().map(|s| s * s).sum();
|
||||||
|
(sum / w.len().max(1) as f32).sqrt() > FLOOR
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// whisper emits bracketed non-speech annotations ([BLANK_AUDIO], (wind
|
||||||
|
/// blowing)) that are not words and must not reach the agent.
|
||||||
|
fn clean(raw: &str) -> String {
|
||||||
|
let mut out = String::new();
|
||||||
|
let mut depth = 0i32;
|
||||||
|
for c in raw.chars() {
|
||||||
|
match c {
|
||||||
|
'[' | '(' => depth += 1,
|
||||||
|
']' | ')' => depth = (depth - 1).max(0),
|
||||||
|
c if depth == 0 => out.push(c),
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out.split_whitespace().collect::<Vec<_>>().join(" ")
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::{clean, has_speech};
|
||||||
|
use crate::audio::STT_RATE;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn silence_is_not_speech() {
|
||||||
|
assert!(!has_speech(&vec![0.0f32; STT_RATE as usize]));
|
||||||
|
// Room tone: audible on a meter, not a voice.
|
||||||
|
assert!(!has_speech(&vec![0.004f32; STT_RATE as usize]));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_short_utterance_in_a_long_silence_counts() {
|
||||||
|
let mut s = vec![0.0f32; STT_RATE as usize];
|
||||||
|
for v in s.iter_mut().take(STT_RATE as usize / 5) {
|
||||||
|
*v = 0.2;
|
||||||
|
}
|
||||||
|
assert!(has_speech(&s));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn strips_bracketed_annotations() {
|
||||||
|
assert_eq!(clean("[BLANK_AUDIO]"), "");
|
||||||
|
assert_eq!(clean(" hello (wind blowing) there \n"), "hello there");
|
||||||
|
assert_eq!(clean("[ music ] real words"), "real words");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn collapses_whitespace() {
|
||||||
|
assert_eq!(clean(" a b \n c "), "a b c");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn unbalanced_brackets_do_not_eat_the_rest() {
|
||||||
|
// The stray closer is dropped (depth floors at 0) but the words
|
||||||
|
// around it survive — the failure mode that matters is a lone
|
||||||
|
// bracket swallowing the whole transcript.
|
||||||
|
assert_eq!(clean("ok] fine"), "ok fine");
|
||||||
|
assert_eq!(clean("[ oops real words"), "");
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,319 @@
|
|||||||
|
use crate::audio;
|
||||||
|
use crate::visemes::{self, Viseme};
|
||||||
|
use anyhow::Result;
|
||||||
|
use base64::Engine;
|
||||||
|
use futures_util::{SinkExt, StreamExt};
|
||||||
|
use tokio::sync::mpsc;
|
||||||
|
use tokio_tungstenite::tungstenite::client::IntoClientRequest;
|
||||||
|
use tokio_tungstenite::tungstenite::Message as Ws;
|
||||||
|
|
||||||
|
pub enum SpeechEvent {
|
||||||
|
/// Decoded PCM at audio::TTS_RATE.
|
||||||
|
Audio(Vec<f32>),
|
||||||
|
/// Mouth shapes, timed from the start of this reply's audio.
|
||||||
|
Visemes(Vec<Viseme>),
|
||||||
|
Done,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct TtsConfig {
|
||||||
|
pub key: String,
|
||||||
|
pub voice_id: String,
|
||||||
|
pub model: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Speaks sentences as they arrive on `sentences`, over one websocket per
|
||||||
|
/// reply.
|
||||||
|
///
|
||||||
|
/// pcm_24000 rather than the default mp3: there is no decoder in this process
|
||||||
|
/// and no reason to add one. sync_alignment makes the character timings ride
|
||||||
|
/// along with the audio frames instead of needing a second request.
|
||||||
|
pub async fn speak(
|
||||||
|
cfg: &TtsConfig,
|
||||||
|
mut sentences: mpsc::Receiver<String>,
|
||||||
|
events: mpsc::Sender<SpeechEvent>,
|
||||||
|
) -> Result<()> {
|
||||||
|
let url = format!(
|
||||||
|
"wss://api.elevenlabs.io/v1/text-to-speech/{}/stream-input\
|
||||||
|
?model_id={}&output_format=pcm_{}&sync_alignment=true&auto_mode=true",
|
||||||
|
cfg.voice_id,
|
||||||
|
cfg.model,
|
||||||
|
audio::TTS_RATE
|
||||||
|
);
|
||||||
|
let mut req = url.into_client_request()?;
|
||||||
|
// Header, not the ?authorization= query param: a key in a URL ends up in
|
||||||
|
// every proxy and error log that touches it.
|
||||||
|
req.headers_mut().insert(
|
||||||
|
"xi-api-key",
|
||||||
|
tokio_tungstenite::tungstenite::http::HeaderValue::from_str(&cfg.key)?,
|
||||||
|
);
|
||||||
|
|
||||||
|
let (stream, _) = tokio_tungstenite::connect_async(req).await?;
|
||||||
|
let (mut sink, mut source) = stream.split();
|
||||||
|
|
||||||
|
// The protocol requires an opening frame of a single space before any
|
||||||
|
// real text; it carries the voice settings for the whole connection.
|
||||||
|
sink.send(Ws::Text(
|
||||||
|
serde_json::json!({
|
||||||
|
"text": " ",
|
||||||
|
"voice_settings": { "stability": 0.4, "similarity_boost": 0.8, "speed": 1.0 },
|
||||||
|
})
|
||||||
|
.to_string(),
|
||||||
|
))
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
let reader = tokio::spawn(async move {
|
||||||
|
// Alignment times may be per-chunk or cumulative depending on the
|
||||||
|
// model; `emitted_ms` lets us normalise either into one timeline.
|
||||||
|
let mut emitted_ms: u32 = 0;
|
||||||
|
while let Some(Ok(msg)) = source.next().await {
|
||||||
|
let Ws::Text(txt) = msg else { continue };
|
||||||
|
let Ok(v) = serde_json::from_str::<serde_json::Value>(&txt) else { continue };
|
||||||
|
|
||||||
|
if let Some(b64) = v.get("audio").and_then(|a| a.as_str()) {
|
||||||
|
if let Ok(bytes) = base64::engine::general_purpose::STANDARD.decode(b64) {
|
||||||
|
let pcm = audio::pcm16_to_f32(&bytes);
|
||||||
|
let chunk_ms = (pcm.len() as f64 / audio::TTS_RATE as f64 * 1000.0) as u32;
|
||||||
|
if let Some(al) = v.get("normalizedAlignment").or_else(|| v.get("alignment")) {
|
||||||
|
if let Some(t) = alignment_to_timeline(al, emitted_ms) {
|
||||||
|
let _ = events.send(SpeechEvent::Visemes(t)).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
emitted_ms += chunk_ms;
|
||||||
|
if events.send(SpeechEvent::Audio(pcm)).await.is_err() {
|
||||||
|
return; // receiver gone: barge-in cancelled this reply
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if v.get("isFinal").and_then(|f| f.as_bool()).unwrap_or(false) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let _ = events.send(SpeechEvent::Done).await;
|
||||||
|
});
|
||||||
|
|
||||||
|
while let Some(text) = sentences.recv().await {
|
||||||
|
// Every text frame must end with a space or the model holds the last
|
||||||
|
// word back waiting for more input.
|
||||||
|
let text = format!("{} ", text.trim());
|
||||||
|
sink.send(Ws::Text(
|
||||||
|
serde_json::json!({ "text": text }).to_string(),
|
||||||
|
))
|
||||||
|
.await?;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Empty string = end of input. Without it the socket sits until the
|
||||||
|
// inactivity timeout and the last sentence never gets flushed.
|
||||||
|
sink.send(Ws::Text(serde_json::json!({ "text": "" }).to_string()))
|
||||||
|
.await?;
|
||||||
|
let _ = reader.await;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn alignment_to_timeline(al: &serde_json::Value, emitted_ms: u32) -> Option<Vec<Viseme>> {
|
||||||
|
let chars: Vec<String> = al
|
||||||
|
.get("chars")?
|
||||||
|
.as_array()?
|
||||||
|
.iter()
|
||||||
|
.map(|c| c.as_str().unwrap_or("").to_string())
|
||||||
|
.collect();
|
||||||
|
let starts: Vec<i64> = num_array(al.get("charStartTimesMs")?)?;
|
||||||
|
let durs: Vec<i64> = num_array(al.get("charDurationsMs")?)?;
|
||||||
|
let starts = absolutize(&starts, emitted_ms);
|
||||||
|
Some(visemes::timeline(&chars, &starts, &durs))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn num_array(v: &serde_json::Value) -> Option<Vec<i64>> {
|
||||||
|
Some(v.as_array()?.iter().map(|n| n.as_i64().unwrap_or(0)).collect())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Normalises chunk-relative and stream-absolute alignments into one
|
||||||
|
/// timeline. If the first start time is behind audio we have already emitted,
|
||||||
|
/// the timings must be relative to this chunk, so they get shifted.
|
||||||
|
fn absolutize(starts: &[i64], emitted_ms: u32) -> Vec<i64> {
|
||||||
|
let offset = match starts.first() {
|
||||||
|
Some(&first) if first < emitted_ms as i64 => emitted_ms as i64,
|
||||||
|
_ => 0,
|
||||||
|
};
|
||||||
|
starts.iter().map(|s| s + offset).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Splits streamed text into speakable sentences, and strips what should
|
||||||
|
/// never be read aloud. Hermes' own voice mode does the same thing for the
|
||||||
|
/// same reason: without it she pronounces asterisks.
|
||||||
|
pub struct SentenceSplitter {
|
||||||
|
buf: String,
|
||||||
|
in_think: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl SentenceSplitter {
|
||||||
|
pub fn new() -> Self {
|
||||||
|
SentenceSplitter { buf: String::new(), in_think: false }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Feeds a delta, returning any sentences that completed.
|
||||||
|
pub fn push(&mut self, delta: &str) -> Vec<String> {
|
||||||
|
self.buf.push_str(delta);
|
||||||
|
let mut out = Vec::new();
|
||||||
|
|
||||||
|
// <think> blocks are reasoning, not speech, and can be long.
|
||||||
|
loop {
|
||||||
|
if self.in_think {
|
||||||
|
match self.buf.find("</think>") {
|
||||||
|
Some(i) => {
|
||||||
|
self.buf.drain(..i + "</think>".len());
|
||||||
|
self.in_think = false;
|
||||||
|
}
|
||||||
|
None => {
|
||||||
|
self.buf.clear();
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
match self.buf.find("<think>") {
|
||||||
|
Some(i) => {
|
||||||
|
let head: String = self.buf.drain(..i).collect();
|
||||||
|
self.buf.drain(.."<think>".len());
|
||||||
|
self.in_think = true;
|
||||||
|
out.extend(drain_sentences(&mut { head.clone() }, true));
|
||||||
|
}
|
||||||
|
None => break,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
out.extend(drain_sentences(&mut self.buf, false));
|
||||||
|
out.into_iter().filter(|s| !s.is_empty()).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whatever is left when the reply ends.
|
||||||
|
pub fn finish(&mut self) -> Option<String> {
|
||||||
|
let rest = clean_for_speech(&std::mem::take(&mut self.buf));
|
||||||
|
(!rest.is_empty()).then_some(rest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn drain_sentences(buf: &mut String, force: bool) -> Vec<String> {
|
||||||
|
let mut out = Vec::new();
|
||||||
|
// A sentence is only worth sending once it is long enough to synthesise
|
||||||
|
// well; ElevenLabs' own guidance is ~20 characters minimum.
|
||||||
|
while let Some(end) = buf.find(['.', '!', '?', '\n']) {
|
||||||
|
if end < 20 && !force {
|
||||||
|
// Abbreviation or a very short clause — keep accumulating.
|
||||||
|
if buf.len() < 80 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let s: String = buf.drain(..end + 1).collect();
|
||||||
|
let s = clean_for_speech(&s);
|
||||||
|
if !s.is_empty() {
|
||||||
|
out.push(s);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if force && !buf.is_empty() {
|
||||||
|
let s = clean_for_speech(&std::mem::take(buf));
|
||||||
|
if !s.is_empty() {
|
||||||
|
out.push(s);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Markdown, emoji and kaomoji are for eyes.
|
||||||
|
///
|
||||||
|
/// A whitelist, not a blacklist: she writes things like "(¬‿¬)", whose
|
||||||
|
/// characters live in ordinary Latin-1 and punctuation blocks rather than in
|
||||||
|
/// any emoji range, so enumerating what to drop never converges. Letters
|
||||||
|
/// stays unicode-aware on purpose — German is in scope, "ö" is a letter.
|
||||||
|
pub fn clean_for_speech(s: &str) -> String {
|
||||||
|
let mut out = String::with_capacity(s.len());
|
||||||
|
for c in s.chars() {
|
||||||
|
if c.is_alphanumeric() || c.is_whitespace() || is_speakable_punctuation(c) {
|
||||||
|
out.push(c);
|
||||||
|
} else if matches!(c, '—' | '–') {
|
||||||
|
// " — " must not leave " , ": the comma binds to the word before it.
|
||||||
|
while out.chars().last().is_some_and(char::is_whitespace) {
|
||||||
|
out.pop();
|
||||||
|
}
|
||||||
|
// An em dash is a pause, not a space. Dropping it runs the two
|
||||||
|
// clauses together and changes what the sentence means; a comma
|
||||||
|
// is the closest thing the synthesiser understands. ASCII '-'
|
||||||
|
// deliberately does NOT land here — it is usually intra-word
|
||||||
|
// ("push-to-talk"), where a comma would be worse than a space.
|
||||||
|
out.push(',');
|
||||||
|
} else {
|
||||||
|
// Not dropped silently: a symbol between two words is usually a
|
||||||
|
// dash or a bullet, and gluing the words together mispronounces
|
||||||
|
// both.
|
||||||
|
out.push(' ');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out.split_whitespace().collect::<Vec<_>>().join(" ")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Punctuation a TTS model actually uses for prosody. Everything else is
|
||||||
|
/// noise once it reaches audio.
|
||||||
|
fn is_speakable_punctuation(c: char) -> bool {
|
||||||
|
matches!(c, '.' | ',' | '!' | '?' | ';' | ':' | '\'' | '"')
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn absolutize_shifts_chunk_relative_times() {
|
||||||
|
assert_eq!(absolutize(&[0, 50], 1000), vec![1000, 1050]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn absolutize_leaves_already_absolute_times() {
|
||||||
|
assert_eq!(absolutize(&[1000, 1050], 1000), vec![1000, 1050]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn splitter_emits_complete_sentences_only() {
|
||||||
|
let mut s = SentenceSplitter::new();
|
||||||
|
assert!(s.push("This is a long enough first sentence").is_empty());
|
||||||
|
let got = s.push(". And more");
|
||||||
|
assert_eq!(got, vec!["This is a long enough first sentence."]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn splitter_drops_think_blocks() {
|
||||||
|
let mut s = SentenceSplitter::new();
|
||||||
|
s.push("<think>internal monologue that is long</think>");
|
||||||
|
let got = s.push("Actually speakable sentence here. ");
|
||||||
|
assert_eq!(got, vec!["Actually speakable sentence here."]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn splitter_returns_tail_on_finish() {
|
||||||
|
let mut s = SentenceSplitter::new();
|
||||||
|
s.push("no terminator here");
|
||||||
|
assert_eq!(s.finish().as_deref(), Some("no terminator here"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn markdown_and_emoji_never_reach_the_voice() {
|
||||||
|
assert_eq!(clean_for_speech("**bold** `code` 🎉 ok"), "bold code ok");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn kaomoji_and_dashes_are_stripped_without_gluing_words() {
|
||||||
|
// Her actual first reply over the API.
|
||||||
|
assert_eq!(
|
||||||
|
clean_for_speech("Loud and clear, boss — hallo! (¬‿¬)"),
|
||||||
|
"Loud and clear, boss, hallo!"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn em_dash_becomes_a_pause_but_hyphens_do_not() {
|
||||||
|
assert_eq!(clean_for_speech("I can't confirm — there's nothing"), "I can't confirm, there's nothing");
|
||||||
|
assert_eq!(clean_for_speech("push-to-talk"), "push to talk");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn keeps_non_ascii_letters() {
|
||||||
|
assert_eq!(clean_for_speech("schön, größer"), "schön, größer");
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
use serde::Serialize;
|
||||||
|
|
||||||
|
/// VRM 1.0's standard mouth expressions, which is what the avatar rig exposes
|
||||||
|
/// (`aa`/`ih`/`ou`/`ee`/`oh`). A rig with the full ARKit 52 can map these onto
|
||||||
|
/// jawOpen/mouthFunnel/etc — the shell side owns that, not this.
|
||||||
|
///
|
||||||
|
/// These names are a CONTRACT with the balsam-generated morph targets: the
|
||||||
|
/// glTF target names become the QML ids, so renaming a shape key in Blender
|
||||||
|
/// breaks the binding silently, with no eval error.
|
||||||
|
pub const SILENCE: &str = "sil";
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, PartialEq, Serialize)]
|
||||||
|
pub struct Viseme {
|
||||||
|
pub shape: &'static str,
|
||||||
|
/// Milliseconds from speech_start, NOT wall clock — the shell starts its
|
||||||
|
/// own timer when the first PCM frame hits the speaker, so the two clocks
|
||||||
|
/// never have to agree.
|
||||||
|
pub at: u32,
|
||||||
|
pub ms: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One character to a mouth shape. English orthography is a lie about
|
||||||
|
/// phonetics, but at 40ms of animation per character nobody can tell: what
|
||||||
|
/// reads as "talking" is the OPEN/CLOSE rhythm, not vowel accuracy.
|
||||||
|
fn shape_for(c: char) -> &'static str {
|
||||||
|
match c.to_ascii_lowercase() {
|
||||||
|
'a' => "aa",
|
||||||
|
'e' => "ee",
|
||||||
|
'i' | 'y' => "ih",
|
||||||
|
'o' => "oh",
|
||||||
|
'u' | 'w' => "ou",
|
||||||
|
// Bilabials close the mouth; giving them silence is what produces the
|
||||||
|
// visible "p/b/m" beat.
|
||||||
|
'p' | 'b' | 'm' => SILENCE,
|
||||||
|
// Everything else is a partially open consonant. "ih" is the least
|
||||||
|
// committal shape to land on.
|
||||||
|
c if c.is_alphabetic() => "ih",
|
||||||
|
_ => SILENCE,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Builds a playable timeline from ElevenLabs' character alignment.
|
||||||
|
///
|
||||||
|
/// The three arrays are parallel; a short or missing one means the frame was
|
||||||
|
/// truncated, so we take the shortest common prefix rather than panicking on
|
||||||
|
/// an index.
|
||||||
|
pub fn timeline(chars: &[String], starts: &[i64], durations: &[i64]) -> Vec<Viseme> {
|
||||||
|
let n = chars.len().min(starts.len()).min(durations.len());
|
||||||
|
let mut out: Vec<Viseme> = Vec::with_capacity(n);
|
||||||
|
|
||||||
|
for i in 0..n {
|
||||||
|
let c = match chars[i].chars().next() {
|
||||||
|
Some(c) => c,
|
||||||
|
None => continue,
|
||||||
|
};
|
||||||
|
let shape = shape_for(c);
|
||||||
|
let at = starts[i].max(0) as u32;
|
||||||
|
let ms = durations[i].max(0) as u32;
|
||||||
|
|
||||||
|
// Merge runs of the same shape: "ll" is one mouth position held, not
|
||||||
|
// two identical keyframes the shell would re-trigger.
|
||||||
|
match out.last_mut() {
|
||||||
|
Some(last) if last.shape == shape && last.at + last.ms >= at => {
|
||||||
|
last.ms = (at + ms).saturating_sub(last.at);
|
||||||
|
}
|
||||||
|
_ => out.push(Viseme { shape, at, ms }),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sub-frame keys cannot be seen and only cost socket bytes.
|
||||||
|
out.retain(|v| v.ms >= 20);
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
fn s(v: &[&str]) -> Vec<String> {
|
||||||
|
v.iter().map(|s| s.to_string()).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn maps_vowels_to_distinct_shapes() {
|
||||||
|
let t = timeline(
|
||||||
|
&s(&["a", "e", "o"]),
|
||||||
|
&[0, 100, 200],
|
||||||
|
&[100, 100, 100],
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
t.iter().map(|v| v.shape).collect::<Vec<_>>(),
|
||||||
|
vec!["aa", "ee", "oh"]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn merges_repeated_shapes_into_one_held_key() {
|
||||||
|
let t = timeline(&s(&["l", "l", "l"]), &[0, 50, 100], &[50, 50, 50]);
|
||||||
|
assert_eq!(t.len(), 1);
|
||||||
|
assert_eq!(t[0].shape, "ih");
|
||||||
|
assert_eq!(t[0].at, 0);
|
||||||
|
assert_eq!(t[0].ms, 150);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn bilabials_close_the_mouth() {
|
||||||
|
let t = timeline(&s(&["a", "m", "a"]), &[0, 100, 200], &[100, 100, 100]);
|
||||||
|
assert_eq!(t[1].shape, SILENCE);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn drops_keys_too_short_to_see() {
|
||||||
|
let t = timeline(&s(&["a", "e"]), &[0, 5], &[5, 100]);
|
||||||
|
assert_eq!(t.len(), 1);
|
||||||
|
assert_eq!(t[0].shape, "ee");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn tolerates_truncated_arrays() {
|
||||||
|
let t = timeline(&s(&["a", "e", "o"]), &[0, 100], &[100]);
|
||||||
|
assert_eq!(t.len(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn punctuation_and_spaces_are_silence() {
|
||||||
|
let t = timeline(&s(&[" ", ",", "!"]), &[0, 100, 200], &[100, 100, 100]);
|
||||||
|
assert_eq!(t.len(), 1);
|
||||||
|
assert_eq!(t[0].shape, SILENCE);
|
||||||
|
assert_eq!(t[0].ms, 300);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
{ config, pkgs, lib, ... }:
|
||||||
|
|
||||||
|
# Voice loop for luna: hold a key, talk, hear her answer.
|
||||||
|
#
|
||||||
|
# mic -> whisper-cpp (local, vulkan) -> Hermes' OpenAI-compatible API on mars
|
||||||
|
# -> ElevenLabs streaming TTS -> speaker. State, amplitude and the viseme
|
||||||
|
# timeline are published on a unix socket for the quickshell avatar to read
|
||||||
|
# (nothing consumes it yet — the widget is the next phase).
|
||||||
|
#
|
||||||
|
# A USER service, not a system one: it needs darman's pipewire session and his
|
||||||
|
# XDG_RUNTIME_DIR for the socket. That is also why the two secrets it reads
|
||||||
|
# are owner-darman in hosts/terra/secrets.nix — a user manager cannot read a
|
||||||
|
# root-owned 0400 file, and LoadCredential does not bridge that gap.
|
||||||
|
#
|
||||||
|
# STT is LOCAL on purpose: mars already runs faster-whisper for Telegram voice
|
||||||
|
# notes, but routing the mic through it would put every spoken word on the
|
||||||
|
# network for no latency win. This is the client-direct path Hermes itself
|
||||||
|
# documents.
|
||||||
|
|
||||||
|
let
|
||||||
|
cfg = config.homelab.lunaVoice;
|
||||||
|
luna-voice = pkgs.callPackage ../../pkgs/luna-voice.nix { };
|
||||||
|
|
||||||
|
# Pinned like the KittenTTS models on mars: a model that re-downloads at
|
||||||
|
# first use is a service that fails on a fresh install with no network.
|
||||||
|
whisperModel = pkgs.fetchurl {
|
||||||
|
url = "https://huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-base.en.bin";
|
||||||
|
sha256 = "sha256-oDd5yG3zMjB19eeWyyzlAp8A7Ihp7uP9+4l6/jbG0AI=";
|
||||||
|
};
|
||||||
|
in
|
||||||
|
{
|
||||||
|
options.homelab.lunaVoice = {
|
||||||
|
enable = lib.mkEnableOption "luna voice daemon";
|
||||||
|
|
||||||
|
user = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
default = "darman";
|
||||||
|
description = "User whose session runs the daemon.";
|
||||||
|
};
|
||||||
|
|
||||||
|
hermesUrl = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
default = "http://mars.orbit.sol:8642";
|
||||||
|
description = "Base URL of luna's Hermes API server (tailnet-only).";
|
||||||
|
};
|
||||||
|
|
||||||
|
model = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
default = "hermes-agent";
|
||||||
|
description = "Model name to request; Hermes routes it to her configured provider.";
|
||||||
|
};
|
||||||
|
|
||||||
|
voiceId = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
description = "ElevenLabs voice id — luna's voice.";
|
||||||
|
};
|
||||||
|
|
||||||
|
ttsModel = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
default = "eleven_flash_v2_5";
|
||||||
|
description = "~75ms model, the one ElevenLabs recommends for live conversation.";
|
||||||
|
};
|
||||||
|
|
||||||
|
hermesKeyFile = lib.mkOption {
|
||||||
|
type = lib.types.path;
|
||||||
|
description = "File holding the API_SERVER_KEY bearer token.";
|
||||||
|
};
|
||||||
|
|
||||||
|
elevenLabsKeyFile = lib.mkOption {
|
||||||
|
type = lib.types.path;
|
||||||
|
description = "File holding the ElevenLabs API key.";
|
||||||
|
};
|
||||||
|
|
||||||
|
systemPrompt = lib.mkOption {
|
||||||
|
type = lib.types.nullOr lib.types.str;
|
||||||
|
default = null;
|
||||||
|
description = ''
|
||||||
|
Prepended to the conversation. Worth setting to something like "You
|
||||||
|
are being spoken to out loud; keep replies short" — her Telegram
|
||||||
|
register is too long-form to listen to.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
config = lib.mkIf cfg.enable {
|
||||||
|
environment.systemPackages = [ luna-voice ];
|
||||||
|
|
||||||
|
systemd.user.services.luna-voiced = {
|
||||||
|
description = "luna voice loop";
|
||||||
|
wantedBy = [ "graphical-session.target" ];
|
||||||
|
after = [ "graphical-session.target" "pipewire.service" ];
|
||||||
|
partOf = [ "graphical-session.target" ];
|
||||||
|
|
||||||
|
environment = {
|
||||||
|
LUNA_HERMES_URL = cfg.hermesUrl;
|
||||||
|
LUNA_MODEL = cfg.model;
|
||||||
|
LUNA_VOICE_ID = cfg.voiceId;
|
||||||
|
LUNA_TTS_MODEL = cfg.ttsModel;
|
||||||
|
LUNA_HERMES_KEY_FILE = toString cfg.hermesKeyFile;
|
||||||
|
LUNA_ELEVENLABS_KEY_FILE = toString cfg.elevenLabsKeyFile;
|
||||||
|
LUNA_WHISPER_BIN = "${pkgs.whisper-cpp-vulkan}/bin/whisper-cli";
|
||||||
|
LUNA_WHISPER_MODEL = whisperModel;
|
||||||
|
} // lib.optionalAttrs (cfg.systemPrompt != null) {
|
||||||
|
LUNA_SYSTEM_PROMPT = cfg.systemPrompt;
|
||||||
|
};
|
||||||
|
|
||||||
|
serviceConfig = {
|
||||||
|
ExecStart = "${luna-voice}/bin/luna-voice daemon";
|
||||||
|
Restart = "on-failure";
|
||||||
|
RestartSec = 3;
|
||||||
|
};
|
||||||
|
};
|
||||||
|
};
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user