EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language
Lists
Only you see this
Abstract
Unlike spoken languages where the use of prosodic features to convey emotion is well studied, indicators of emotion in sign language remain poorly understood, creating communication barriers in critical settings. Sign languages present unique challenges as facial expressions and hand movements simultaneously serve both grammatical and emotional functions. To address this gap, we introduce EmoSign, the first sign video dataset containing sentiment and emotion labels for 200 American Sign Language (ASL) videos. We also collect open-ended descriptions of emotion cues, such as specific expressions and signing speed, that lead to the identified emotions. Annotations were done by 3 Deaf ASL signers with professional interpretation experience. Alongside the annotations, we include benchmarks of baseline models for sentiment and emotion classification. Our benchmark results show that current multimodal models fail to integrate visual cues into emotional reasoning and exhibit bias towards positive emotions. This dataset not only addresses a critical gap in existing sign language research but also establishes a new benchmark for understanding model capabilities in multimodal emotion recognition for sign languages. This work can inspire new architectures that integrate fine-grained visual understanding with linguistic context awareness to distinguish e.g., syntactic versus affective functions of visual cues.
BibTeX
@misc{chua2025emosign,
author = {Phoebe Chua and Cathy Mengying Fang and Takehiko Ohkawa and Raja Kushalnagar and Paul Pu Liang and Suranga Chandima Nanayakkara and Patricia Maes},
title = {{EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language}},
year = {2025},
eprint = {2505.17090},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2505.17090},
}