TL;DR
This paper introduces Entheos, the first multimodal (video, audio, text) dataset for studying enthusiasm in speech, specifically from TED talks. They also present baseline models and analyze multimodal features, highlighting the importance of vocal cues and discourse relations for enthusiasm detection.
What Problem Does This Paper Address?
While enthusiasm is a crucial element in engaging communication, its automatic detection and generation by machines remain a significant challenge. Existing research in NLP and AI has largely focused on more common emotions like anger or happiness, leaving finer-grained expressions like enthusiasm relatively unexplored. This gap limits the development of more sophisticated and emotionally aware human-machine interactions, as well as the creation of more engaging virtual agents. The authors aim to bridge this gap by providing a dedicated multimodal dataset and initial models for studying enthusiasm.
Methods
The researchers developed a multimodal dataset, Entheos, by collecting TED talk speeches and annotating them for enthusiasm levels (monotonous, normal, enthusiastic). They extracted features from three modalities:
- Video: Facial Action Units (FAUs) from the Facial Action Coding System (FACS) were used to capture facial expressions.
- Audio: A comprehensive set of acoustic features was extracted using OpenSMILE, including eGEMAPS, IS09, IS10, and IS13 features, covering aspects like voice quality, pitch, energy, and spectral characteristics.
- Text: Word embeddings from the BERT-large-uncased model were used as low-level features. Additionally, they extracted high-level discourse relations using Rhetorical Structure Theory (RST) and Penn Discourse Treebank (PDTB) parsers.
A multimodal model was then constructed by concatenating these features. They also explored unimodal models and an ablation study to understand the contribution of each modality and feature type.
Datasets
The primary dataset introduced in this paper is Entheos.
- Entheos: This multimodal dataset consists of TED talk speeches, including aligned video frames, audio, and text transcripts. It was annotated at the sentence level with three enthusiasm labels: monotonous, normal, and enthusiastic. The dataset was derived from the TEDLIUM corpus and includes metadata such as perceived gender and viewer ratings.
Key Insights and Ideas
- Multimodality is Key: Enthusiasm is not solely conveyed through one channel. The paper demonstrates that combining features from text, audio, and video leads to significantly better performance in detecting enthusiasm compared to unimodal approaches.
- Vocal Cues Matter: Acoustic features, particularly pitch (F0) and loudness, are strong indicators of enthusiasm. Enthusiastic speech tends to be louder and have a higher pitch.
- Discourse Structure Aids Understanding: Discourse relations, especially those identified by RST and PDTB parsers, provide valuable context for understanding enthusiasm, even though the parsers were trained on different data.
- Facial Action Units (FAUs) Less Informative (in this study): While FAUs were extracted, their direct statistical measures did not significantly improve the model's performance. This suggests that simpler statistical aggregation of FAUs might not capture the nuances of facial expressions related to enthusiasm.
- Gender Dependency: The study found a statistically significant dependency between speaker gender and enthusiasm level, suggesting that gender might play a role in how enthusiasm is expressed.
Key Results
- Multimodal Performance: Combining acoustic features (eGEMAPS) with text (BERT) and discourse relations (RST + PDTB) achieved the highest F1-score of 0.83 in the binary classification (enthusiastic vs. non-enthusiastic) and a slight improvement of 0.04 in the three-class problem compared to unimodal models.
- Unimodal Best: In unimodal settings, BERT embeddings performed best for text, while eGEMAPS was the strongest acoustic feature set.
- Discourse Relation Impact: Discourse relations were found to significantly influence enthusiasm levels, with enthusiastic samples showing distinct patterns in relation types compared to monotonous ones.
- Acoustic Feature Significance: Statistical tests confirmed that both mean pitch (F0) and loudness, as well as their standard deviations, are significantly related to enthusiasm levels.
- Facial Action Unit Significance: Certain AUs, like AU 12 (lip corner puller), AU 15 (lip corner depressor), AU 17 (chin raiser), and AU 26 (jaw drop), showed statistically significant differences in their mean values between enthusiasm levels, particularly AU 26 and AU 17.
Limitations and Future Work
- TED Talk Specificity: The dataset is derived from TED talks, which are well-rehearsed and prepared. This may not fully represent spontaneous enthusiasm in daily conversations. Future work should involve annotating data from other domains.
- Facial Feature Limitations: The current approach using statistical aggregation of FAUs was not successful. Future research should explore more sophisticated methods for analyzing facial expressions and gestures.
- Discourse Parser Domain Mismatch: The discourse parsers used were trained on Wall Street Journal data, which differs from monologue speech. Evaluating and improving discourse parsers for spoken monologues is suggested.
- Weakly Supervised Approaches: Given the difficulty of annotating multimodal datasets, investigating weakly supervised methods for labeling multimodal data is a promising future direction.
- Fine-grained Analysis: Further research is needed to explore finer-grained signals and definitions of enthusiasm and their interconnections across modalities.
Summary updated