Publications - Research Papers from Hume AI | Hume AI
Publications
July 2026
RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems
DA
+11
David Ayllon, Alice Baird, Jeffrey Brooks and 11 more
Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.
arXiv·May 2026
The 2026 ACII Dyadic Conversations (DaiKon) Workshop & Challenge
PT
+6
Panagiotis Tzirakis, Alice Baird, Jeffrey Brooks and 6 more
The 2026 ACII Dyadic Conversations (ACII-DaiKon) Workshop & Challenge introduces a benchmark for modeling interpersonal affect and social dynamics in dyadic conversations. Although conversational affect modeling has advanced rapidly, most benchmarks remain speaker-centric and underrepresent coupled, time-evolving processes between partners, including directional influence, conversational timing coordination, and rapport development. To address this gap, ACII-DaiKon presents three coordinated sub-challenges built on a shared dataset: (1) directional interpersonal influence prediction, (2) turn-taking prediction (next-speaker and time-to-next-speech), and (3) rapport trajectory prediction across full interactions.
The challenge is built on the Hume-DaiKon dataset, comprising 945 dyadic conversations (743.4 hours of audiovisual data) collected under naturalistic conditions across five languages. The benchmark supports multimodal modeling, temporal reasoning, and cross-context generalization through fixed train/validation/test splits, standardized metrics, and released baseline systems. Evaluation uses Concordance Correlation Coefficient (CCC), Pearson correlation, Macro-F1, and Mean Absolute Error (MAE) depending on the sub-challenge.
Baseline experiments establish initial reference performance, with best test results of 0.40 CCC and 0.50 Pearson for influence prediction, 0.66 Macro-F1 and 1.50~s MAE for turn-taking, and 0.68 CCC and 0.70 Pearson for rapport trajectory modeling. These results indicate that while current methods capture coarse dyadic patterns, robust modeling of directional dependence and long-horizon interpersonal dynamics remains challenging. The workshop provides a shared platform for rigorous comparison and cross-disciplinary discussion on data validity, evaluation protocols, and culturally aware modeling for dyadic interaction.
arXiv·February 2026
TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment (Under Review)
TD
SR
AG
+6
Trung Dang, Sharath Rao, Ananya Gupta and 6 more
Modern Text-to-Speech (TTS) systems increasingly leverage Large Language Model (LLM) architectures to achieve scalable, high-fidelity, zero-shot generation. However, these systems typically rely on fixed-frame-rate acoustic tokenization, resulting in speech sequences that are significantly longer than, and asynchronous with their corresponding text. Beyond computational inefficiency, this sequence length disparity often triggers hallucinations in TTS and amplifies the modality gap in spoken language modeling (SLM). In this paper, we propose a novel tokenization scheme that establishes one-to-one synchronization between continuous acoustic features and text tokens, enabling unified, single-stream modeling within an LLM. We demonstrate that these synchronous tokens maintain high-fidelity audio reconstruction and can be effectively modeled in a latent space by a large language model with a flow matching head. Moreover, the ability to seamlessly toggle speech modality within the context enables text-only guidance--a technique that blends logits from text-only and text-speech modes to flexibly bridge the gap toward text-only LLM intelligence. Experimental results indicate that our approach achieves performance competitive with state-of-the-art TTS and SLM systems while virtually eliminating content hallucinations and preserving linguistic integrity, all at a significantly reduced inference cost.
Frontiers in Psychology·May 2024
How emotion is experienced and expressed in multiple cultures: a large-scale experiment across North America, Europe, and Japan
GP
+13
Alan Cowen, Jeffrey Brooks, Gautam Prasad and 13 more
Core to understanding emotion are subjective experiences and their expression in facial behavior. Past studies have largely focused on six emotions and prototypical facial poses, reflecting limitations in scale and narrow assumptions about the variety of emotions and their patterns of expression.
iScience·February 2024
Deep learning reveals what facial expressions mean to people in different cultures
LK
MO
+10
Jeffrey Brooks, Lauren Kim, Michael Opara and 10 more
Cross-cultural studies of the meaning of facial expressions have largely focused on judgments of small sets of stereotypical images by small numbers of people. Here, we used large-scale data collection and machine learning to map what facial expressions convey in six countries.
Current Directions in Psychological Science·February 2023
Semantic Space Theory: Data-Driven Insights Into Basic Emotions
Dacher Keltner, Jeffrey Brooks, and Alan Cowen
Here we present semantic space theory and the data-driven methods it entails. Across the largest studies to date of emotion-related experience, expression, and physiology, we find that emotion is high dimensional, defined by blends of upward of 20 distinct kinds of emotions, and not reducible to low-dimensional structures and conceptual processes as assumed by constructivist accounts. Specific emotions are not separated by sharp boundaries, contrary to basic emotion theory, and include states that often blend. Emotion concepts such as “anger” are primary in the unfolding of emotional experience and emotion recognition, more so than core affect processes of valence and arousal. We conclude by outlining studies showing how these data-driven discoveries are a basis of machine-learning models that are serving larger-scale, more diverse studies of naturalistic emotional behavior.
ACM International Conference on Multimedia·October 2022
The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress
SA
LC
AK
+4
Shahin Amiriparian, Lukas Christ, Andreas König and 4 more
The 3rd Multimodal Sentiment Analysis Challenge (MuSe) focuses on multimodal affective computing.
International Conference on Affective Computing and Intelligent Interaction·July 2022
The ACII 2022 Affective Vocal Bursts Workshop & Competition: Understanding a critically understudied modality of emotional expression
PT
+5
Alice Baird, Panagiotis Tzirakis, Jeffrey Brooks and 5 more
The ACII Affective Vocal Bursts Workshop & Competition is focused on understanding multiple affective dimensions of vocal bursts: laughs, gasps, cries, screams, and many other non-linguistic vocalizations central to the expression of emotion and to human communication more generally.
PsyArXiv·May 2022
Facial movements have over twenty dimensions of perceived meaning that are only partially captured with traditional methods
KM
XF
+3
Alan Cowen, Kunalan Manokara, Xia Fang and 3 more
Central to science and technology are questions about how to measure facial expression. The current gold standard is the facial action coding system (FACS), which is often assumed to account for all facial muscle movements relevant to perceived emotion. However, the mapping from FACS codes to perceived emotion is not well understood.
International Conference on Machine Learning·May 2022
The ICML 2022 Expressive Vocalizations Workshop and Competition: Recognizing, generating, and personalizing vocal bursts
PT
GG
+7
Alice Baird, Panagiotis Tzirakis, Gauthier Gidel and 7 more
The ICML Expressive Vocalization (EXVO) Competition is focused on understanding and generating vocal bursts: laughs, gasps, cries, and other non-verbal vocalizations that are central to emotional expression and communication.
Emotion·April 2022
Intersectionality in emotion signaling and recognition: The influence of gender, ethnicity, and social class
MM
Maria Monroy, Alan Cowen, and Dacher Keltner
Emotional expressions are a language of social interaction. Guided by recent advances in the study of expression and intersectionality, the present investigation examined how gender, ethnicity, and social class influence the signaling and recognition of 34 states in dynamic full-body expressive behavior.
Cognition and Emotion·March 2022
How emotions, relationships, and culture constitute each other: Advances in social functionalist theory
DS
JL
+2
Dacher Keltner, Disa Sauter, Jessica L. Tracy and 2 more
Social Functionalist Theory (SFT) emerged 20 years ago to orient emotion science to the social nature of emotion. Here we expand upon SFT and make the case for how emotions, relationships, and culture constitute one another.
Nature Human Behaviour·January 2022
Deep learning reveals what vocal bursts express in different cultures
PT
+8
Jeffrey Brooks, Panagiotis Tzirakis, Alice Baird and 8 more
Human social life is rich with sighs, chuckles, shrieks and other emotional vocalizations, called ‘vocal bursts’. Nevertheless, the meaning of vocal bursts across cultures is only beginning to be understood. Here, we combined large-scale experimental data collection with deep learning to reveal the shared and culture-specific meanings of vocal bursts.
Nature·December 2020
Sixteen facial expressions occur in similar contexts worldwide
FS
+3
Alan Cowen, Dacher Keltner, Florian Schroff and 3 more
What emotions do the face and body express? Guided by new conceptual and quantitative approaches (Cowen, Elfenbein, Laukka, & Keltner, 2018; Cowen & Keltner, 2017, 2018), we explore the taxonomy of emotion recognized in facial-bodily expression.
Science Advances·August 2020
Universal facial expressions uncovered in art of the ancient Americas: A computational approach
Alan Cowen and Dacher Keltner
Central to the study of emotion is evidence concerning its universality, particularly the degree to which emotional expressions are similar across cultures. Here, we present an approach to studying the universality of emotional expression that rules out cultural contact and circumvents potential biases in survey-based methods: A computational analysis of apparent facial expressions portrayed in artwork created by members of cultures isolated from Western civilization.
Annual Meeting of the Association for Computational Linguistics·July 2020
GoEmotions: A dataset of fine-grained emotions
DD
DM
JK
+3
Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko and 3 more
Understanding emotion expressed in language has a wide range of applications, from building empathetic chatbots to detecting harmful online behavior. Advancement in this area can be improved using large-scale datasets with a fine-grained typology, adaptable to multiple downstream tasks.
iScience·May 2020
The neural representation of visually evoked emotion is high-dimensional, categorical, and distributed across transmodal brain regions
TH
+1
Tomoyasu Horikawa, Alan Cowen, Dacher Keltner and 1 more
Emotional vocalizations are central to human social life. Recent studies have documented that people recognize at least 13 emotions in brief vocalizations. This capacity emerges early in development, is preserved in some form across cultures, and informs how people respond emotionally to music.
Proceedings of the National Academy of Sciences·January 2020
What music makes us feel: At least 13 dimensions organize subjective experiences associated with music across different cultures
XF
DS
+1
Alan Cowen, Xia Fang, Disa Sauter and 1 more
Psychological Science in the Public Interest·September 2019
Mapping the passions: Toward a high-dimensional taxonomy of emotional experience and expression
DS
JL
+1
Alan Cowen, Disa Sauter, Jessica L. Tracy and 1 more
What would a comprehensive atlas of human emotions include? For 50 years, scientists have sought to map emotion-related experience, expression, physiology, and recognition in terms of the “basic six”—anger, disgust, fear, happiness, sadness, and surprise.
Current Directions in Psychological Science·July 2019
Emotional expression: Advances in basic emotion theory
DS
JT
+1
Dacher Keltner, Disa Sauter, Jessica Tracy and 1 more
In this article, we review recent developments in the study of emotional expression within a basic emotion framework. Dozens of new studies find that upwards of 20 emotions are signaled in multimodal and dynamic patterns of expressive behavior.
Emotion·June 2019
What the face displays: Mapping 28 emotions conveyed by naturalistic expression
Alan Cowen and Dacher Keltner
What emotions do the face and body express? Guided by new conceptual and quantitative approaches, we explore the taxonomy of emotion recognized in facial-bodily expression. Participants judged the emotions captured in 1,500 photographs of facial-bodily expression in terms of emotion categories, appraisals, free response, and ecological validity.
American Psychologist·April 2019
Mapping 24 emotions conveyed by brief human vocalization
HA
PL
+1
Alan Cowen, Hillary Anger Elfenbein, Petri Laukka and 1 more
Nature Human Behaviour·February 2019
The primacy of categories in the recognition of 12 emotions in speech prosody across two cultures
PL
HA
+2
Alan Cowen, Petri Laukka, Hillary Anger Elfenbein and 2 more
Proceedings of the National Academy of Sciences·September 2017
Self-report captures 27 distinct categories of emotion bridged by continuous gradients
Alan Cowen and Dacher Keltner
Claims about how reported emotional experiences are geometrically organized within a semantic space have shaped the study of emotion. Using statistical methods to analyze reports of emotional states elicited by 2,185 emotionally evocative short videos with richly varying situational content, we uncovered 27 varieties of reported emotional experience.