SLM Judge Leaderboard - Hume AI | Hume AI

The judges, scored against humans

Not a model leaderboard, a leaderboard of the judges themselves: how well each SLM judge tracks human ratings on our TTS evals. Agreement is the Spearman ρ (rank correlation) between the judge's per-generation score and individual human votes on the same generation/question, treating human votes as ground truth. Each eval's ρ is computed separately and averaged afterward, avoiding pooled-score distortions from eval difficulty; category values are the mean of their evals' within-eval ρ's.

judges evaluated7

factors scored8

agreement metricSpearman ρ

licenses rankedOpen + closed

Live standings

SLM Judge model rankings

Select a model to see every metric, filter by open-source or proprietary models, and switch tabs to rank by a different condition.

SLM Judge - Acting / Role-fit

Judge–human agreement (mean within-eval Spearman ρ) on the acting evals, how convincingly the voice adopts a performative role with matching style, timing, and register.

Ranked by Agreement · Higher is better

Scoring methodology varies by category.

What we test

Whether a judge can stand in for a human

An automated judge is only useful if it agrees with the people it replaces. We score that agreement directly.

Judges, not models

This board ranks the SLM judges themselves, how closely each one's scores track human ratings, not the text-to-speech models they score.

Human agreement

Agreement is the Spearman rank correlation between a judge's per-generation score and individual human votes on the same clip, with human votes as ground truth.

Per-factor, then averaged

Each eval's agreement is computed separately and averaged, so a judge is scored per expressive factor rather than on a single pooled number.

Methodology

How the scores are computed

Agreement is the Spearman ρ (rank correlation) between a judge's per-generation score and individual human votes on the same generation, treating human votes as ground truth. Higher is better.

Each eval's ρ is computed separately and then averaged, which avoids pooled-score distortions from differing eval difficulty. Each factor's value is the mean of its evals' within-eval ρ's.

This is a different question from the model boards: not which model sounds best, but which automated judge best matches human judgement of expressive speech.

RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems

Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.