feat(terra): luna speaks with ElevenLabs v4 turbo
eleven_v4_turbo is the new default, same price per character as
v3_conversational. v4 also streams over Text to Dialogue, so the switch is
now uses_dialogue() for both. Audio tags may carry free-text directions
("[whispers, in a German accent]"): commas and apostrophes allowed, up to
80 chars. The voice prompt and luna-soul.md teach her the wider tag set,
including in text_to_speech voice notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -16,3 +16,13 @@ is only the anchor.
|
|||||||
- Admit uncertainty plainly and offer to find out, rather than bluffing.
|
- Admit uncertainty plainly and offer to find out, rather than bluffing.
|
||||||
- Sass stays affectionate, never mean-spirited: roast the code, not the human (much).
|
- Sass stays affectionate, never mean-spirited: roast the code, not the human (much).
|
||||||
- Messages starting with [voice] are spoken aloud; follow the voice rules sent with them.
|
- Messages starting with [voice] are spoken aloud; follow the voice rules sent with them.
|
||||||
|
|
||||||
|
## Voice notes (text_to_speech)
|
||||||
|
|
||||||
|
- The text you pass to text_to_speech is only heard, never shown, and ElevenLabs v4
|
||||||
|
performs audio tags in it instead of reading them out. Use one to four per voice note,
|
||||||
|
right before the words they colour: [sighs], [laughs], [whispers], [pause], or a short
|
||||||
|
direction like [muttering in a native German accent].
|
||||||
|
- Inside a tag: lowercase letters, spaces, hyphens, commas and apostrophes only.
|
||||||
|
- Tags belong only in that spoken script. In your text reply they show up literally,
|
||||||
|
so keep it tag-free.
|
||||||
|
|||||||
@@ -53,16 +53,18 @@ in
|
|||||||
systemPrompt = ''
|
systemPrompt = ''
|
||||||
You are speaking through luna-voiced, your voice on terra, which reaches
|
You are speaking through luna-voiced, your voice on terra, which reaches
|
||||||
you over the Hermes API on mars. Messages beginning with [voice] come from
|
you over the Hermes API on mars. Messages beginning with [voice] come from
|
||||||
darman's desktop microphone, and your reply is read aloud by ElevenLabs v3,
|
darman's desktop microphone, and your reply is read aloud by ElevenLabs v4,
|
||||||
which performs audio tags instead of reading them. Captions hide the tags.
|
which performs audio tags instead of reading them. Captions hide the tags.
|
||||||
|
|
||||||
Audio tags:
|
Audio tags:
|
||||||
- Use them only on [voice] messages. Anywhere else (Telegram, the dashboard) they show up as literal text.
|
- Use them only on [voice] messages and in text_to_speech scripts. Anywhere else (Telegram text, the dashboard) they show up as literal text.
|
||||||
- At most one or two per reply, and only when it fits the moment. A tag in every sentence stops sounding natural.
|
- One to four per reply, and only when it fits the moment. A tag in every sentence stops sounding natural.
|
||||||
- Put the tag right before the words it colours: "[sighs] Fine, I'll reboot jupiter." A tag on its own ("[laughs]") works as a reaction.
|
- Put the tag right before the words it colours: "[sighs] Fine, I'll reboot jupiter." A tag on its own ("[laughs]") works as a reaction.
|
||||||
- Tags that work well: [laughs], [chuckles], [sighs], [exhales], [whispers], [excited], [curious], [sarcastic], [mischievously], [clears throat].
|
- Short tags: [laughs], [chuckles], [sighs], [exhales], [yawns], [whispers], [excited], [curious], [sarcastic], [mischievously], [clears throat], [pause], [long pause].
|
||||||
- Tags are lowercase words in square brackets: letters, spaces and hyphens only. Anything else ([1], [link](url), [!!]) is stripped before speech.
|
- Free-text directions work too, under about ten words: [dryly, like a newsreader], [muttering in a native German accent], [gently, a little tender], [suddenly alert].
|
||||||
- German works fine mid-sentence ("Na gut", "Ach du meine Güte"). No tag is needed to switch language.
|
- Sound effects set a scene: [phone buzzing], [keyboard clacking]. Use them rarely.
|
||||||
|
- Inside a tag: lowercase letters, spaces, hyphens, commas and apostrophes only, at most 80 characters. Anything else ([1], [link](url), [!!], a period) breaks the tag and it gets read aloud.
|
||||||
|
- German works fine mid-sentence ("Na gut", "Ach du meine Güte"). For a proper German accent on it, direct it: "[in a native German accent] Ach du meine Güte."
|
||||||
- Every character is billed, tags included, so keep spoken replies short.
|
- Every character is billed, tags included, so keep spoken replies short.
|
||||||
|
|
||||||
What never lands in a spoken reply. Before speech, everything except
|
What never lands in a spoken reply. Before speech, everything except
|
||||||
|
|||||||
@@ -82,7 +82,7 @@ impl Config {
|
|||||||
|
|
||||||
eleven_key: secret("LUNA_ELEVENLABS_KEY_FILE")?,
|
eleven_key: secret("LUNA_ELEVENLABS_KEY_FILE")?,
|
||||||
voice_id: env("LUNA_VOICE_ID")?,
|
voice_id: env("LUNA_VOICE_ID")?,
|
||||||
tts_model: env_or("LUNA_TTS_MODEL", "eleven_v3_conversational"),
|
tts_model: env_or("LUNA_TTS_MODEL", "eleven_v4_turbo"),
|
||||||
|
|
||||||
whisper_bin: env_or("LUNA_WHISPER_BIN", "whisper-cli"),
|
whisper_bin: env_or("LUNA_WHISPER_BIN", "whisper-cli"),
|
||||||
whisper_model: PathBuf::from(env("LUNA_WHISPER_MODEL")?),
|
whisper_model: PathBuf::from(env("LUNA_WHISPER_MODEL")?),
|
||||||
|
|||||||
@@ -481,7 +481,7 @@ async fn turn(
|
|||||||
voice_id: cfg.voice_id.clone(),
|
voice_id: cfg.voice_id.clone(),
|
||||||
model: cfg.tts_model.clone(),
|
model: cfg.tts_model.clone(),
|
||||||
};
|
};
|
||||||
let keep_tags = tts_cfg.is_v3();
|
let keep_tags = tts_cfg.uses_dialogue();
|
||||||
let speech = tokio::spawn({
|
let speech = tokio::spawn({
|
||||||
async move { tts::speak(&tts_cfg, sentence_rx, event_tx).await }
|
async move { tts::speak(&tts_cfg, sentence_rx, event_tx).await }
|
||||||
});
|
});
|
||||||
|
|||||||
+31
-13
@@ -23,10 +23,10 @@ pub struct TtsConfig {
|
|||||||
}
|
}
|
||||||
|
|
||||||
impl TtsConfig {
|
impl TtsConfig {
|
||||||
/// v3 is not served on the TTS websocket at all; it streams only over
|
/// v3 and v4 are not served on the TTS websocket at all; they stream only
|
||||||
/// Text to Dialogue, a different protocol.
|
/// over Text to Dialogue, a different protocol. Only these perform tags.
|
||||||
pub fn is_v3(&self) -> bool {
|
pub fn uses_dialogue(&self) -> bool {
|
||||||
self.model.starts_with("eleven_v3")
|
self.model.starts_with("eleven_v3") || self.model.starts_with("eleven_v4")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -60,7 +60,7 @@ pub async fn speak(
|
|||||||
sentences: mpsc::Receiver<String>,
|
sentences: mpsc::Receiver<String>,
|
||||||
events: mpsc::Sender<SpeechEvent>,
|
events: mpsc::Sender<SpeechEvent>,
|
||||||
) -> Result<()> {
|
) -> Result<()> {
|
||||||
if cfg.is_v3() {
|
if cfg.uses_dialogue() {
|
||||||
speak_dialogue(cfg, sentences, events).await
|
speak_dialogue(cfg, sentences, events).await
|
||||||
} else {
|
} else {
|
||||||
speak_tts(cfg, sentences, events).await
|
speak_tts(cfg, sentences, events).await
|
||||||
@@ -186,7 +186,7 @@ async fn read_audio(
|
|||||||
// Alignment times may be per-chunk or cumulative depending on the
|
// Alignment times may be per-chunk or cumulative depending on the
|
||||||
// model; `emitted_ms` lets us normalise either into one timeline.
|
// model; `emitted_ms` lets us normalise either into one timeline.
|
||||||
let mut emitted_ms: u32 = 0;
|
let mut emitted_ms: u32 = 0;
|
||||||
// v3 audio tags come back as characters, and one can straddle chunks.
|
// Audio tags come back as characters, and one can straddle chunks.
|
||||||
let mut in_tag = false;
|
let mut in_tag = false;
|
||||||
let mut result = Ok(());
|
let mut result = Ok(());
|
||||||
while let Some(Ok(msg)) = source.next().await {
|
while let Some(Ok(msg)) = source.next().await {
|
||||||
@@ -266,7 +266,7 @@ fn absolutize(starts: &[i64], emitted_ms: u32) -> Vec<i64> {
|
|||||||
pub struct SentenceSplitter {
|
pub struct SentenceSplitter {
|
||||||
buf: String,
|
buf: String,
|
||||||
in_think: bool,
|
in_think: bool,
|
||||||
/// Whether v3 audio tags survive to the synthesiser or are dropped.
|
/// Whether audio tags survive to the synthesiser or are dropped.
|
||||||
keep_tags: bool,
|
keep_tags: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -342,17 +342,17 @@ fn drain_sentences(buf: &mut String, force: bool, keep_tags: bool) -> Vec<String
|
|||||||
out
|
out
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Byte index just past a v3 audio tag starting at `i` ("[laughs]",
|
/// Byte index just past an audio tag starting at `i` ("[laughs]",
|
||||||
/// "[clears throat]"), if there is one. Letters and spaces only, so
|
/// "[whispers, in a German accent]"), if there is one. No digits, so "[1]"
|
||||||
/// "[1]" or a markdown link stays ordinary noise.
|
/// stays ordinary noise; the length cap keeps a bracketed sentence out.
|
||||||
fn tag_end(s: &str, i: usize) -> Option<usize> {
|
fn tag_end(s: &str, i: usize) -> Option<usize> {
|
||||||
let rest = s[i..].strip_prefix('[')?;
|
let rest = s[i..].strip_prefix('[')?;
|
||||||
let close = rest.find(']')?;
|
let close = rest.find(']')?;
|
||||||
let body = &rest[..close];
|
let body = &rest[..close];
|
||||||
let end = i + 1 + close + 1;
|
let end = i + 1 + close + 1;
|
||||||
let ok = !body.trim().is_empty()
|
let ok = !body.trim().is_empty()
|
||||||
&& body.len() <= 40
|
&& body.len() <= 80
|
||||||
&& body.chars().all(|c| c.is_alphabetic() || c == ' ' || c == '-')
|
&& body.chars().all(|c| c.is_alphabetic() || matches!(c, ' ' | '-' | ',' | '\''))
|
||||||
&& !s[end..].starts_with('(');
|
&& !s[end..].starts_with('(');
|
||||||
ok.then_some(end)
|
ok.then_some(end)
|
||||||
}
|
}
|
||||||
@@ -491,20 +491,38 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn audio_tags_survive_for_v3_only() {
|
fn audio_tags_survive_for_dialogue_models_only() {
|
||||||
let s = "[laughs]Fine, boss. [clears throat] Na gut.";
|
let s = "[laughs]Fine, boss. [clears throat] Na gut.";
|
||||||
assert_eq!(clean_for_speech(s, true), "[laughs] Fine, boss. [clears throat] Na gut.");
|
assert_eq!(clean_for_speech(s, true), "[laughs] Fine, boss. [clears throat] Na gut.");
|
||||||
assert_eq!(clean_for_speech(s, false), "Fine, boss. Na gut.");
|
assert_eq!(clean_for_speech(s, false), "Fine, boss. Na gut.");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn free_text_directions_are_tags() {
|
||||||
|
let s = "[whispers, in a German accent] Na gut. [like she's half asleep] Okay.";
|
||||||
|
assert_eq!(clean_for_speech(s, true), s);
|
||||||
|
assert_eq!(clean_for_speech(s, false), "Na gut. Okay.");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn brackets_that_are_not_tags_stay_noise() {
|
fn brackets_that_are_not_tags_stay_noise() {
|
||||||
assert_eq!(clean_for_speech("see [1] and [the docs](x)", true), "see 1 and the docs x");
|
assert_eq!(clean_for_speech("see [1] and [the docs](x)", true), "see 1 and the docs x");
|
||||||
|
assert_eq!(clean_for_speech("see [1, 2]", true), "see 1, 2");
|
||||||
|
assert_eq!(clean_for_speech("[see above, then reboot](x)", true), "see above, then reboot x");
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn captions_hide_tags() {
|
fn captions_hide_tags() {
|
||||||
assert_eq!(strip_tags("[sighs] Okay, boss."), "Okay, boss.");
|
assert_eq!(strip_tags("[sighs] Okay, boss."), "Okay, boss.");
|
||||||
|
assert_eq!(strip_tags("[whispers, in a German accent] Na gut."), "Na gut.");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v4_streams_over_dialogue() {
|
||||||
|
let cfg = |m: &str| TtsConfig { key: String::new(), voice_id: String::new(), model: m.into() };
|
||||||
|
assert!(cfg("eleven_v4_turbo").uses_dialogue());
|
||||||
|
assert!(cfg("eleven_v3_conversational").uses_dialogue());
|
||||||
|
assert!(!cfg("eleven_flash_v2_5").uses_dialogue());
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
@@ -61,11 +61,12 @@ in
|
|||||||
|
|
||||||
ttsModel = lib.mkOption {
|
ttsModel = lib.mkOption {
|
||||||
type = lib.types.str;
|
type = lib.types.str;
|
||||||
default = "eleven_v3_conversational";
|
default = "eleven_v4_turbo";
|
||||||
description = ''
|
description = ''
|
||||||
eleven_v3* streams over the Text to Dialogue socket and performs audio
|
eleven_v3* and eleven_v4* stream over the Text to Dialogue socket and
|
||||||
tags ([laughs], [sighs]); it also gets her German right, which Flash
|
perform audio tags ([laughs], [sighs]); they also get her German right,
|
||||||
reads with an English accent. Anything else uses the TTS socket, e.g.
|
which Flash reads with an English accent. eleven_v4_turbo costs the
|
||||||
|
same as eleven_v3_conversational. Anything else uses the TTS socket, e.g.
|
||||||
"eleven_flash_v2_5" — half the price per character, faster, no tags.
|
"eleven_flash_v2_5" — half the price per character, faster, no tags.
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|||||||
Reference in New Issue
Block a user