Real World VoiceEQ Bench - Hume AI | Hume AI

Voice AI, evaluated across every dimension that matters

Four leaderboards, one framework: speech recognition, speech understanding, text-to-speech quality, and live voice agent performance, scored in the conditions that decide real-world deployment.

TTS Leaderboard

Text-to-speech, scored on what it produces

We score how well a model turns text into speech across the dimensions that decide real-world quality: expressive range, identity stability, pronunciation accuracy, and audio cleanliness. What matters is what the model produces across a full performance, not how clean one sentence sounds.

TTS - Acting / Role-fit

How convincingly the voice adopts a performative role, narrator, commentator, comedian, casual speaker, with style, timing, and register that match the context, not just correct reading.

All Open Proprietary

Ranked by Score · Higher is better

What we score

The factors that separate good synthesis from great

Most models can read one clean sentence. We score the parts that show up in real use.

Expressiveness

How reliably the voice delivers the intended emotion, from plainly stated feeling to subtext, sarcasm, and mid-line emotional shifts, including subtle or suppressed affect that shows only in the prosody.

Voice identity

Whether the voice stays recognizably the same speaker under stress: emotional extremes, whispered or fast or loud delivery, tiredness or illness, and shifts in register.

Correctness

Pronunciation accuracy on the hard cases that break production voice products: drug names, alphanumeric strings, dates, identifiers, and other structured data.

Long-form stability

Whether quality and intelligibility hold across extended passages and mid-clip emotional shifts, the kind of degradation short clips never reveal.

Methodology

How the scores are computed

Every model gets an absolute score from 1 to 5 on each factor it was evaluated on. Raters set the scale so that 3 is the field average and higher is better. A model is measured against the field, not against the other models it happens to sit next to.

What the data shows

No model leads on both expressiveness and precision.

There's a consistent tradeoff across the field: models that score highest on pronunciation accuracy (pharmaceutical names, alphanumeric strings) tend to underperform on expressive range, and vice versa.

Voice identity collapses under emotional stress.

Models that sound consistent at normal register often sound like a different voice when pushed to extremes: loud projection, emotional intensity, physical states. This dimension isn't tested anywhere else.

Long-form stability is the hardest dimension, and the least benchmarked.

Quality degrades significantly over extended passages in ways that short-clip evaluations never reveal. Models that sound great on one sentence often drift noticeably by paragraph three.

Acting and role-fit don't correlate with overall quality scores.

A model can be technically accurate and expressively flat. Role-fit - whether the voice convincingly adopts a narrator, commentator, or character - is a separate capability that the overall score obscures.

Download PDF

Download PDF

RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems

Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.