Introducing OCTAVE (Omni-Capable Text and Voice Engine) | Hume Blog | Hume AI

Octave: The first TTS powered by a language model - YouTube

Octave: The first TTS powered by a language model

We’re introducing OCTAVE (Omni-Capable Text and Voice Engine), a next-generation speech-language model that combines the capabilities of our EVI 2 speech-language model with those of systems like OpenAI’s Voice Engine, Elevenlab’s TTS Voice Design, and Google Deepmind’s NotebookLM.

From descriptive prompts or recordings as brief as 5s, OCTAVE generates not just voices, but personalities (language, accent, expressions, underlying disposition, etc.) that can talk to you. And it can generate multiple, interacting AI personalities and voices within a real-time response.

1. Generating not just voices, but personalities from prompts

OCTAVE can generate any voice and personality – and the accompanying language – from a prompt, emulating gender, age, accent, vocal register, emotional intonation, speaking styles related to vocations or roles ("gentle therapist," "wizard mentor"), and many other characteristics.

Example 1: “A male voice that is extremely gravelly, as if he was gargling hot asphalt.”

Example 2: “The speaker's voice is that of an extremely gentle and empathetic therapist voice, with thoughtful pauses between phrases. Warm, supportive tone that feels like a comforting embrace, speaking just above a whisper.”

Example 3: “The speaker is an English male AI assistant, mid-30s sound, impossibly smooth and sophisticated. Perfect BBC pronunciation with subtle electronic undertones. Slight reverb suggesting spacious room acoustics.”

Example 4: “The speaker is an engaging narrator with Welsh accent. Naturally builds suspense with pacing and tone. Like a favorite uncle telling bedtime stories but with professional voice acting training.”

Example 5: “The speaker is a New Zealand female, mid-40s, with a soothing but grounded therapeutic voice. Deliberately slow pace with extended pauses. Gentle prosody that naturally guides breathing, slight Kiwi accent adds warmth."

Example 6: “The speaker is a Brooklyn cab driver with a rapid-fire, nasal voice that barrels through conversations like rush hour traffic. The voice carries a thick New York accent with aggressively pronounced vowels and clipped endings."

Example 7: “The speaker is a wise wizard with a refined British accent that gives weight to every carefully chosen word. The voice carries scholarly authority and frequently employs archaic terms and formal phrasing.”

Example 8: “Authoritative, firm, commanding, direct, crisp, clear.”

2. Instant voice and personality adoption from recordings

OCTAVE can extract a clean representation of any speaker’s voice, accent, and personality from a noisy recording as brief as 5s, clone these voice qualities in a single shot, and generate clean dialog with the speaker’s voice, all in one step.

Example 1: Cloning Hume's Lauren Kim from a 5s clip

Example 2: Extending Ilya Sutskever's 2024 NeurIPS talk

3. Interacting with any voice or personality in real-time

Any voice and personality OCTAVE generates or adopts can be used for real-time interaction. For instance, we can append a recording of Hume CEO Alan Cowen to the last example and have OCTAVE respond with its imitation of Ilya's voice and personality:

OCTAVE’s understanding of the interplay between speaking style, expressions, and underlying disposition informs its generated language and voice during real-time interaction. This results in richer and more authentic communication than can be achieved with separate models handling transcription, language response, and speech generation.

4. Generating multiple, interacting characters

Because OCTAVE has full control over the acoustic properties of the voices it generates, it can generate dialog for multiple interacting speakers, switching among them at will. OCTAVE can clone and generate the same characters using a brief sample from NotebookLM as input.

Example 1: Replicating NotebookLM from a single recording

Example 2: Continuing an OpenAI Advanced Voice Mode Demo

Language performance vs. similar-sized LLM

Despite its diverse speech processing and generation capabilities, OCTAVE maintains comparable performance on language understanding tasks to a similar-sized frontier LLM. Note that all responses in this blog post were generated by OCTAVE 3B, demonstrating the capabilities of our smallest model.

OCTAVE's frontier language capabilities ensure that the same source of intelligence that determines its language maintains its personality, hears the voice of the user, and produces nuanced vocal responses. The result is a coherent persona that sounds like it understands what it's saying. The model's language capabilities also mean that it is well-suited to power AI systems that follow detailed instructions, use tools, or control an interface on its own.

Model availability

We are still working to improve OCTAVE, and given its range of new capabilities, we are taking a cautious approach to releasing it. We’ve begun giving trusted partners early access to a limited version of OCTAVE so that the model can be evaluated for safety and effectiveness in various application settings. We plan to roll out broader availability in the coming months.

OCTAVE promises to enable richer, more realistic, and more multifaceted AI experiences than EVI 2. For example, users and developers will be able to craft personas for AI agents, personalize them for individuals or even create them on the fly to answer a particular question, or enable real-time group conversations involving multiple users or AIs. We’re excited to hear what you’d like to see built with it.