Introduction
Multimodal Emotion Recognition (MER) aims to decipher human emotional states by integrating diverse data streams, typically speech (audio) and text. While attention mechanisms have significantly advanced this field, they often struggle to pinpoint emotionally salient segments within long sequences without external guidance.
A common strategy involves leveraging "emotion lexicons"—databases that quantify the emotional intensity of specific words. However, traditional "hard" injection methods, which force the model to prioritize lexicon-defined words, often introduce bias because these lexicons are not tailored to specific training datasets. This paper addresses the challenge of incorporating external emotional knowledge flexibly, ensuring the model can benefit from these "hints" without being strictly bound by them.
Methodology
The researchers propose a Knowledge-aware Bayesian Co-attention framework. The architecture integrates state-of-the-art feature extractors with a novel probabilistic attention mechanism.
Pipeline Components
- Feature Extraction: The model utilizes BERT for text processing and wav2vec 2.0 for speech representation.
- Word-level Force Alignment: To bridge the gap between continuous audio signals and discrete text, "force alignment" is employed to segment audio into word-corresponding units.
- Knowledge Softening: To address lexicon sparsity (where many words lack emotional scores), a self-attention module "smears" emotional intensity across the sequence, providing contextual awareness.
- Bayesian Attention Module (BAM): This core component treats attention weights as random variables rather than fixed scalars. By using the NRC VAD lexicon (Valence, Arousal, Dominance) as a probabilistic "prior," the model generates a "posterior" distribution based on the observed data. This allows the model to weigh lexicon knowledge against the actual evidence in the training data, ignoring the lexicon if it contradicts the specific context.
Key Results
The proposed model achieves a new state-of-the-art on the IEMOCAP dataset, a standard benchmark for emotion recognition.
The model reached an Unweighted Accuracy (UA) of 77.0% and a Weighted Accuracy (WA) of 75.5%, outperforming previous transformer-based approaches. Ablation studies confirmed that the Bayesian "soft" injection is superior to "hard" injection; the latter actually decreased performance (from 74.9% to 74.6% UA in certain configurations), while the full BAM-integrated model provided the highest gains.
Discussion and Future Work
The study highlights that while text is the strongest individual modality (66.6% UA), the fusion with speech (T+W) is crucial for capturing the full emotional spectrum. The Bayesian framework acts as a "temperature" control, where high-intensity emotional words increase internal connectivity within the model.
Future research directions include adapting the model for non-English languages (reducing dependency on the English-centric NRC VAD lexicon) and improving robustness to noisy real-world audio where force alignment might be less accurate. Additionally, exploring more complex emotional states like sarcasm remains a significant frontier for multimodal systems.
Summary updated