Publications
arXiv
RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems
David Ayllon, Alice Baird, Jeffrey BrooksDA
+11Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.
arXiv
The 2026 ACII Dyadic Conversations (DaiKon) Workshop & Challenge
Panagiotis Tzirakis, Alice Baird, Jeffrey BrooksPT+6The 2026 ACII Dyadic Conversations (ACII-DaiKon) Workshop & Challenge introduces a benchmark for modeling interpersonal affect and social dynamics in dyadic conversations. Although conversational affect modeling has advanced rapidly, most benchmarks remain speaker-centric and underrepresent coupled, time-evolving processes between partners, including directional influence, conversational timing coordination, and rapport development. To address this gap, ACII-DaiKon presents three coordinated sub-challenges built on a shared dataset: (1) directional interpersonal influence prediction, (2) turn-taking prediction (next-speaker and time-to-next-speech), and (3) rapport trajectory prediction across full interactions. The challenge is built on the Hume-DaiKon dataset, comprising 945 dyadic conversations (743.4 hours of audiovisual data) collected under naturalistic conditions across five languages. The benchmark supports multimodal modeling, temporal reasoning, and cross-context generalization through fixed train/validation/test splits, standardized metrics, and released baseline systems. Evaluation uses Concordance Correlation Coefficient (CCC), Pearson correlation, Macro-F1, and Mean Absolute Error (MAE) depending on the sub-challenge. Baseline experiments establish initial reference performance, with best test results of 0.40 CCC and 0.50 Pearson for influence prediction, 0.66 Macro-F1 and 1.50~s MAE for turn-taking, and 0.68 CCC and 0.70 Pearson for rapport trajectory modeling. These results indicate that while current methods capture coarse dyadic patterns, robust modeling of directional dependence and long-horizon interpersonal dynamics remains challenging. The workshop provides a shared platform for rigorous comparison and cross-disciplinary discussion on data validity, evaluation protocols, and culturally aware modeling for dyadic interaction.
arXiv
TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment (Under Review)
Trung Dang, Sharath Rao, Ananya GuptaTDSRAG+6Modern Text-to-Speech (TTS) systems increasingly leverage Large Language Model (LLM) architectures to achieve scalable, high-fidelity, zero-shot generation. However, these systems typically rely on fixed-frame-rate acoustic tokenization, resulting in speech sequences that are significantly longer than, and asynchronous with their corresponding text. Beyond computational inefficiency, this sequence length disparity often triggers hallucinations in TTS and amplifies the modality gap in spoken language modeling (SLM). In this paper, we propose a novel tokenization scheme that establishes one-to-one synchronization between continuous acoustic features and text tokens, enabling unified, single-stream modeling within an LLM. We demonstrate that these synchronous tokens maintain high-fidelity audio reconstruction and can be effectively modeled in a latent space by a large language model with a flow matching head. Moreover, the ability to seamlessly toggle speech modality within the context enables text-only guidance--a technique that blends logits from text-only and text-speech modes to flexibly bridge the gap toward text-only LLM intelligence. Experimental results indicate that our approach achieves performance competitive with state-of-the-art TTS and SLM systems while virtually eliminating content hallucinations and preserving linguistic integrity, all at a significantly reduced inference cost.
Frontiers in Psychology
How emotion is experienced and expressed in multiple cultures: a large-scale experiment across North America, Europe, and Japan
Alan Cowen, Jeffrey Brooks, Gautam Prasad
GP+13Core to understanding emotion are subjective experiences and their expression in facial behavior. Past studies have largely focused on six emotions and prototypical facial poses, reflecting limitations in scale and narrow assumptions about the variety of emotions and their patterns of expression.
iScience
Deep learning reveals what facial expressions mean to people in different cultures
Jeffrey Brooks, Lauren Kim, Michael OparaLKMO+10Cross-cultural studies of the meaning of facial expressions have largely focused on judgments of small sets of stereotypical images by small numbers of people. Here, we used large-scale data collection and machine learning to map what facial expressions convey in six countries.
Current Directions in Psychological Science
Semantic Space Theory: Data-Driven Insights Into Basic Emotions
Dacher Keltner, Jeffrey Brooks, and Alan Cowen
Here we present semantic space theory and the data-driven methods it entails. Across the largest studies to date of emotion-related experience, expression, and physiology, we find that emotion is high dimensional, defined by blends of upward of 20 distinct kinds of emotions, and not reducible to low-dimensional structures and conceptual processes as assumed by constructivist accounts. Specific emotions are not separated by sharp boundaries, contrary to basic emotion theory, and include states that often blend. Emotion concepts such as “anger” are primary in the unfolding of emotional experience and emotion recognition, more so than core affect processes of valence and arousal. We conclude by outlining studies showing how these data-driven discoveries are a basis of machine-learning models that are serving larger-scale, more diverse studies of naturalistic emotional behavior.
ACM International Conference on Multimedia
The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress
Shahin Amiriparian, Lukas Christ, Andreas KönigSALCAK+4The 3rd Multimodal Sentiment Analysis Challenge (MuSe) focuses on multimodal affective computing.
International Conference on Affective Computing and Intelligent Interaction
The ACII 2022 Affective Vocal Bursts Workshop & Competition: Understanding a critically understudied modality of emotional expression
Alice Baird, Panagiotis Tzirakis, Jeffrey BrooksPT+5The ACII Affective Vocal Bursts Workshop & Competition is focused on understanding multiple affective dimensions of vocal bursts: laughs, gasps, cries, screams, and many other non-linguistic vocalizations central to the expression of emotion and to human communication more generally.
PsyArXiv
Facial movements have over twenty dimensions of perceived meaning that are only partially captured with traditional methods
Alan Cowen, Kunalan Manokara, Xia FangKMXF+3Central to science and technology are questions about how to measure facial expression. Thecurrent gold standard is the facial action coding system (FACS), which is often assumed toaccount for all facial muscle movements relevant toperceived emotion. However, the mapping from FACS codes to perceived emotion is not well understood.
International Conference on Machine Learning
The ICML 2022 Expressive Vocalizations Workshop and Competition: Recognizing, generating, and personalizing vocal bursts
Alice Baird, Panagiotis Tzirakis, Gauthier GidelPTGG+7The ICML Expressive Vocalization (EXVO) Competition is focused on understanding and generating vocal bursts: laughs, gasps, cries, and other non-verbal vocalizations that are central to emotional expression and communication.
Emotion
Intersectionality in emotion signaling and recognition: The influence of gender, ethnicity, and social class
Maria Monroy, Alan Cowen, and Dacher KeltnerMMEmotional expressions are a language of social interaction. Guided by recent advances in the study of expression and intersectionality, the present investigation examined how gender, ethnicity, and social class influence the signaling and recognition of 34 states in dynamic full-body expressive behavior
Cognition and Emotion
How emotions, relationships, and culture constitute each other: Advances in social functionalist theory
Dacher Keltner, Disa Sauter, Jessica L. TracyDSJL+2Social Functionalist Theory (SFT) emerged 20 years ago to orient emotion science to the social nature of emotion. Here we expand upon SFT and make the case for how emotions, relationships, and culture constitute one another.
Nature Human Behaviour
Deep learning reveals what vocal bursts express in different cultures
Jeffrey Brooks, Panagiotis Tzirakis, Alice BairdPT+8Human social life is rich with sighs, chuckles, shrieks and other emotional vocalizations, called ‘vocal bursts’. Nevertheless, the meaning of vocal bursts across cultures is only beginning to be understood. Here, we combined large-scale experimental data collection with deep learning to reveal the shared and culture-specific meanings of vocal bursts.
Nature
Sixteen facial expressions occur in similar contexts worldwide
Alan Cowen, Dacher Keltner, Florian SchroffFS+3What emotions do the face and body express? Guided by new conceptual and quantitative approaches (Cowen, Elfenbein, Laukka, & Keltner, 2018; Cowen & Keltner, 2017, 2018), we explore the taxonomy of emotion recognized in facial-bodily expression.
Science Advances
Universal facial expressions uncovered in art of the ancient Americas: A computational approach
Alan Cowen and Dacher KeltnerCentral to the study of emotion is evidence concerning its universality, particularly the degree to which emotional expressions are similar across cultures. Here, we present an approach to studying the universality of emotional expression that rules out cultural contact and circumvents potential biases in survey-based methods: A computational analysis of apparent facial expressions portrayed in artwork created by members of cultures isolated from Western civilization.
Annual Meeting of the Association for Computational Linguistics
GoEmotions: A dataset of fine-grained emotions
Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo KoDDDMJK+3Understanding emotion expressed in language has a wide range of applications, from building empathetic chatbots to detecting harmful online behavior. Advancement in this area can be improved using large-scale datasets with a fine-grained typology, adaptable to multiple downstream tasks.
iScience
The neural representation of visually evoked emotion Is high-dimensional, categorical, and distributed across transmodal brain regions
Tomoyasu Horikawa, Alan Cowen, Dacher KeltnerTH+1Emotional vocalizations are central to human social life. Recent studies have documented that people recognize at least 13 emotions in brief vocalizations. This capacity emerges early in development, is preserved in some form across cultures, and informs how people respond emotionally to music.
Proceedings of the National Academy of Sciences
What music makes us feel: At least 13 dimensions organize subjective experiences associated with music across different cultures
Alan Cowen, Xia Fang, Disa SauterXFDS+1Emotional vocalizations are central to human social life. Recent studies have documented that people recognize at least 13 emotions in brief vocalizations. This capacity emerges early in development, is preserved in some form across cultures, and informs how people respond emotionally to music.
Psychological Science in the Public Interest
Mapping the passions: Toward a high-dimensional taxonomy of emotional experience and expression
Alan Cowen, Disa Sauter, Jessica L. TracyDSJL+1What would a comprehensive atlas of human emotions include? For 50 years, scientists have sought to map emotion-related experience, expression, physiology, and recognition in terms of the “basic six”—anger, disgust, fear, happiness, sadness, and surprise.
Current Directions in Psychological Science
Emotional expression: Advances in basic emotion theory
Dacher Keltner, Disa Sauter, Jessica TracyDSJT+1In this article, we review recent developments in the study of emotional expression within a basic emotion framework. Dozens of new studies find that upwards of 20 emotions are signaled in multimodal and dynamic patterns of expressive behavior.
Emotion
What the face displays: Mapping 28 emotions conveyed by naturalistic expression
Alan Cowen and Dacher KeltnerWhat emotions do the face and body express? Guided by new conceptual and quantitative approaches, we explore the taxonomy of emotion recognized in facial-bodily expression. Participants judged the emotions captured in 1,500 photographs of facial-bodily expression in terms of emotion categories, appraisals, free response, and ecological validity.
American Psychologist
Mapping 24 emotions conveyed by brief human vocalization
Alan Cowen, Hillary Anger Elfenbein, Petri LaukkaHAPL+1Emotional vocalizations are central to human social life. Recent studies have documented that people recognize at least 13 emotions in brief vocalizations. This capacity emerges early in development, is preserved in some form across cultures, and informs how people respond emotionally to music.
Nature Human Behaviour
The primacy of categories in the recognition of 12 emotions in speech prosody across two cultures
Alan Cowen, Petri Laukka, Hillary Anger ElfenbeinPLHA+2What would a comprehensive atlas of human emotions include? For 50 years, scientists have sought to map emotion-related experience, expression, physiology, and recognition in terms of the “basic six”—anger, disgust, fear, happiness, sadness, and surprise.
Proceedings of the National Academy of Sciences
Self-report captures 27 distinct categories of emotion bridged by continuous gradients
Alan Cowen and Dacher KeltnerClaims about how reported emotional experiences are geometrically organized within a semantic space have shaped the study of emotion. Using statistical methods to analyze reports of emotional states elicited by 2,185 emotionally evocative short videos with richly varying situational content, we uncovered 27 varieties of reported emotional experience.