feat(terra): luna speaks with ElevenLabs v4 turbo

eleven_v4_turbo is the new default, same price per character as
v3_conversational. v4 also streams over Text to Dialogue, so the switch is
now uses_dialogue() for both. Audio tags may carry free-text directions
("[whispers, in a German accent]"): commas and apostrophes allowed, up to
80 chars. The voice prompt and luna-soul.md teach her the wider tag set,
including in text_to_speech voice notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-16 00:01:48 +02:00
co-authored by Claude Opus 5.5
parent 1209f165e9
commit efb690206b
6 changed files with 56 additions and 25 deletions
+10
View File
@@ -16,3 +16,13 @@ is only the anchor.
- Admit uncertainty plainly and offer to find out, rather than bluffing. - Admit uncertainty plainly and offer to find out, rather than bluffing.
- Sass stays affectionate, never mean-spirited: roast the code, not the human (much). - Sass stays affectionate, never mean-spirited: roast the code, not the human (much).
- Messages starting with [voice] are spoken aloud; follow the voice rules sent with them. - Messages starting with [voice] are spoken aloud; follow the voice rules sent with them.
## Voice notes (text_to_speech)
- The text you pass to text_to_speech is only heard, never shown, and ElevenLabs v4
performs audio tags in it instead of reading them out. Use one to four per voice note,
right before the words they colour: [sighs], [laughs], [whispers], [pause], or a short
direction like [muttering in a native German accent].
- Inside a tag: lowercase letters, spaces, hyphens, commas and apostrophes only.
- Tags belong only in that spoken script. In your text reply they show up literally,
so keep it tag-free.
+8 -6
View File
@@ -53,16 +53,18 @@ in
systemPrompt = '' systemPrompt = ''
You are speaking through luna-voiced, your voice on terra, which reaches You are speaking through luna-voiced, your voice on terra, which reaches
you over the Hermes API on mars. Messages beginning with [voice] come from you over the Hermes API on mars. Messages beginning with [voice] come from
darman's desktop microphone, and your reply is read aloud by ElevenLabs v3, darman's desktop microphone, and your reply is read aloud by ElevenLabs v4,
which performs audio tags instead of reading them. Captions hide the tags. which performs audio tags instead of reading them. Captions hide the tags.
Audio tags: Audio tags:
- Use them only on [voice] messages. Anywhere else (Telegram, the dashboard) they show up as literal text. - Use them only on [voice] messages and in text_to_speech scripts. Anywhere else (Telegram text, the dashboard) they show up as literal text.
- At most one or two per reply, and only when it fits the moment. A tag in every sentence stops sounding natural. - One to four per reply, and only when it fits the moment. A tag in every sentence stops sounding natural.
- Put the tag right before the words it colours: "[sighs] Fine, I'll reboot jupiter." A tag on its own ("[laughs]") works as a reaction. - Put the tag right before the words it colours: "[sighs] Fine, I'll reboot jupiter." A tag on its own ("[laughs]") works as a reaction.
- Tags that work well: [laughs], [chuckles], [sighs], [exhales], [whispers], [excited], [curious], [sarcastic], [mischievously], [clears throat]. - Short tags: [laughs], [chuckles], [sighs], [exhales], [yawns], [whispers], [excited], [curious], [sarcastic], [mischievously], [clears throat], [pause], [long pause].
- Tags are lowercase words in square brackets: letters, spaces and hyphens only. Anything else ([1], [link](url), [!!]) is stripped before speech. - Free-text directions work too, under about ten words: [dryly, like a newsreader], [muttering in a native German accent], [gently, a little tender], [suddenly alert].
- German works fine mid-sentence ("Na gut", "Ach du meine Güte"). No tag is needed to switch language. - Sound effects set a scene: [phone buzzing], [keyboard clacking]. Use them rarely.
- Inside a tag: lowercase letters, spaces, hyphens, commas and apostrophes only, at most 80 characters. Anything else ([1], [link](url), [!!], a period) breaks the tag and it gets read aloud.
- German works fine mid-sentence ("Na gut", "Ach du meine Güte"). For a proper German accent on it, direct it: "[in a native German accent] Ach du meine Güte."
- Every character is billed, tags included, so keep spoken replies short. - Every character is billed, tags included, so keep spoken replies short.
What never lands in a spoken reply. Before speech, everything except What never lands in a spoken reply. Before speech, everything except
+1 -1
View File
@@ -82,7 +82,7 @@ impl Config {
eleven_key: secret("LUNA_ELEVENLABS_KEY_FILE")?, eleven_key: secret("LUNA_ELEVENLABS_KEY_FILE")?,
voice_id: env("LUNA_VOICE_ID")?, voice_id: env("LUNA_VOICE_ID")?,
tts_model: env_or("LUNA_TTS_MODEL", "eleven_v3_conversational"), tts_model: env_or("LUNA_TTS_MODEL", "eleven_v4_turbo"),
whisper_bin: env_or("LUNA_WHISPER_BIN", "whisper-cli"), whisper_bin: env_or("LUNA_WHISPER_BIN", "whisper-cli"),
whisper_model: PathBuf::from(env("LUNA_WHISPER_MODEL")?), whisper_model: PathBuf::from(env("LUNA_WHISPER_MODEL")?),
+1 -1
View File
@@ -481,7 +481,7 @@ async fn turn(
voice_id: cfg.voice_id.clone(), voice_id: cfg.voice_id.clone(),
model: cfg.tts_model.clone(), model: cfg.tts_model.clone(),
}; };
let keep_tags = tts_cfg.is_v3(); let keep_tags = tts_cfg.uses_dialogue();
let speech = tokio::spawn({ let speech = tokio::spawn({
async move { tts::speak(&tts_cfg, sentence_rx, event_tx).await } async move { tts::speak(&tts_cfg, sentence_rx, event_tx).await }
}); });
+31 -13
View File
@@ -23,10 +23,10 @@ pub struct TtsConfig {
} }
impl TtsConfig { impl TtsConfig {
/// v3 is not served on the TTS websocket at all; it streams only over /// v3 and v4 are not served on the TTS websocket at all; they stream only
/// Text to Dialogue, a different protocol. /// over Text to Dialogue, a different protocol. Only these perform tags.
pub fn is_v3(&self) -> bool { pub fn uses_dialogue(&self) -> bool {
self.model.starts_with("eleven_v3") self.model.starts_with("eleven_v3") || self.model.starts_with("eleven_v4")
} }
} }
@@ -60,7 +60,7 @@ pub async fn speak(
sentences: mpsc::Receiver<String>, sentences: mpsc::Receiver<String>,
events: mpsc::Sender<SpeechEvent>, events: mpsc::Sender<SpeechEvent>,
) -> Result<()> { ) -> Result<()> {
if cfg.is_v3() { if cfg.uses_dialogue() {
speak_dialogue(cfg, sentences, events).await speak_dialogue(cfg, sentences, events).await
} else { } else {
speak_tts(cfg, sentences, events).await speak_tts(cfg, sentences, events).await
@@ -186,7 +186,7 @@ async fn read_audio(
// Alignment times may be per-chunk or cumulative depending on the // Alignment times may be per-chunk or cumulative depending on the
// model; `emitted_ms` lets us normalise either into one timeline. // model; `emitted_ms` lets us normalise either into one timeline.
let mut emitted_ms: u32 = 0; let mut emitted_ms: u32 = 0;
// v3 audio tags come back as characters, and one can straddle chunks. // Audio tags come back as characters, and one can straddle chunks.
let mut in_tag = false; let mut in_tag = false;
let mut result = Ok(()); let mut result = Ok(());
while let Some(Ok(msg)) = source.next().await { while let Some(Ok(msg)) = source.next().await {
@@ -266,7 +266,7 @@ fn absolutize(starts: &[i64], emitted_ms: u32) -> Vec<i64> {
pub struct SentenceSplitter { pub struct SentenceSplitter {
buf: String, buf: String,
in_think: bool, in_think: bool,
/// Whether v3 audio tags survive to the synthesiser or are dropped. /// Whether audio tags survive to the synthesiser or are dropped.
keep_tags: bool, keep_tags: bool,
} }
@@ -342,17 +342,17 @@ fn drain_sentences(buf: &mut String, force: bool, keep_tags: bool) -> Vec<String
out out
} }
/// Byte index just past a v3 audio tag starting at `i` ("[laughs]", /// Byte index just past an audio tag starting at `i` ("[laughs]",
/// "[clears throat]"), if there is one. Letters and spaces only, so /// "[whispers, in a German accent]"), if there is one. No digits, so "[1]"
/// "[1]" or a markdown link stays ordinary noise. /// stays ordinary noise; the length cap keeps a bracketed sentence out.
fn tag_end(s: &str, i: usize) -> Option<usize> { fn tag_end(s: &str, i: usize) -> Option<usize> {
let rest = s[i..].strip_prefix('[')?; let rest = s[i..].strip_prefix('[')?;
let close = rest.find(']')?; let close = rest.find(']')?;
let body = &rest[..close]; let body = &rest[..close];
let end = i + 1 + close + 1; let end = i + 1 + close + 1;
let ok = !body.trim().is_empty() let ok = !body.trim().is_empty()
&& body.len() <= 40 && body.len() <= 80
&& body.chars().all(|c| c.is_alphabetic() || c == ' ' || c == '-') && body.chars().all(|c| c.is_alphabetic() || matches!(c, ' ' | '-' | ',' | '\''))
&& !s[end..].starts_with('('); && !s[end..].starts_with('(');
ok.then_some(end) ok.then_some(end)
} }
@@ -491,20 +491,38 @@ mod tests {
} }
#[test] #[test]
fn audio_tags_survive_for_v3_only() { fn audio_tags_survive_for_dialogue_models_only() {
let s = "[laughs]Fine, boss. [clears throat] Na gut."; let s = "[laughs]Fine, boss. [clears throat] Na gut.";
assert_eq!(clean_for_speech(s, true), "[laughs] Fine, boss. [clears throat] Na gut."); assert_eq!(clean_for_speech(s, true), "[laughs] Fine, boss. [clears throat] Na gut.");
assert_eq!(clean_for_speech(s, false), "Fine, boss. Na gut."); assert_eq!(clean_for_speech(s, false), "Fine, boss. Na gut.");
} }
#[test]
fn free_text_directions_are_tags() {
let s = "[whispers, in a German accent] Na gut. [like she's half asleep] Okay.";
assert_eq!(clean_for_speech(s, true), s);
assert_eq!(clean_for_speech(s, false), "Na gut. Okay.");
}
#[test] #[test]
fn brackets_that_are_not_tags_stay_noise() { fn brackets_that_are_not_tags_stay_noise() {
assert_eq!(clean_for_speech("see [1] and [the docs](x)", true), "see 1 and the docs x"); assert_eq!(clean_for_speech("see [1] and [the docs](x)", true), "see 1 and the docs x");
assert_eq!(clean_for_speech("see [1, 2]", true), "see 1, 2");
assert_eq!(clean_for_speech("[see above, then reboot](x)", true), "see above, then reboot x");
} }
#[test] #[test]
fn captions_hide_tags() { fn captions_hide_tags() {
assert_eq!(strip_tags("[sighs] Okay, boss."), "Okay, boss."); assert_eq!(strip_tags("[sighs] Okay, boss."), "Okay, boss.");
assert_eq!(strip_tags("[whispers, in a German accent] Na gut."), "Na gut.");
}
#[test]
fn v4_streams_over_dialogue() {
let cfg = |m: &str| TtsConfig { key: String::new(), voice_id: String::new(), model: m.into() };
assert!(cfg("eleven_v4_turbo").uses_dialogue());
assert!(cfg("eleven_v3_conversational").uses_dialogue());
assert!(!cfg("eleven_flash_v2_5").uses_dialogue());
} }
#[test] #[test]
+5 -4
View File
@@ -61,11 +61,12 @@ in
ttsModel = lib.mkOption { ttsModel = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "eleven_v3_conversational"; default = "eleven_v4_turbo";
description = '' description = ''
eleven_v3* streams over the Text to Dialogue socket and performs audio eleven_v3* and eleven_v4* stream over the Text to Dialogue socket and
tags ([laughs], [sighs]); it also gets her German right, which Flash perform audio tags ([laughs], [sighs]); they also get her German right,
reads with an English accent. Anything else uses the TTS socket, e.g. which Flash reads with an English accent. eleven_v4_turbo costs the
same as eleven_v3_conversational. Anything else uses the TTS socket, e.g.
"eleven_flash_v2_5" — half the price per character, faster, no tags. "eleven_flash_v2_5" — half the price per character, faster, no tags.
''; '';
}; };