Abstract
Multimodal affective computing aims to decipher human emotions by integrating signals from video, audio, and text. However, the "emotional semantic gap"—where different modalities provide conflicting emotional cues—often leads to "semantic noise" that degrades model performance. This paper presents the Multimodal Relevance Estimation (MRE) network, a framework designed to identify and prioritize the most relevant modalities for a given emotional state. By employing category-level contrastive learning and a novel relevance estimation module, MRE effectively bridges the gap between heterogeneous data types. Experimental results on the newly introduced SDME dataset and existing benchmarks demonstrate that MRE achieves state-of-the-art accuracy while operating up to 26 times faster than leading complex fusion models.
Introduction: The Emotional Semantic Gap
In real-world scenarios, human emotional expression is rarely uniform across all sensory channels. A person might maintain a neutral facial expression while their voice betrays deep anxiety, or they might frown out of concentration rather than anger. Traditional "direct fusion" models often struggle with these discrepancies, as misleading signals in one modality can "pollute" the joint representation. This phenomenon is known as the Emotional Semantic Gap.
The challenge lies in determining which modality is the most reliable indicator of the true emotional state at any given moment.
Methodology: The MRE Framework
The proposed Multimodal Relevance Estimation (MRE) framework addresses the semantic gap through a four-stage process designed for both accuracy and efficiency.
1. Feature Representation
The model utilizes standard feature extraction tools: BERT for linguistic context, OpenFace for facial action units and landmarks, and Librosa for acoustic properties.
2. Relevant Semantic Estimation
At the core of MRE is an attention-based module that calculates a relevance weight for each modality. If the text is neutral but the audio indicates high arousal, the model dynamically assigns a higher weight to the audio features. This process is guided by a "weak supervision" loss ($L_{rs}$), allowing the model to learn dominant modalities even in the absence of explicit unimodal labels.
3. Contrastive Semantic Estimation
To ensure that features across different modalities are semantically aligned, the authors implement category-level contrastive learning. Unlike instance-level contrastive learning, this approach pulls "Happy" features from video closer to "Happy" features in audio across the entire dataset, creating a more robust and unified emotional space.
4. Low-Rank Fusion
To maintain high inference speeds, MRE employs low-rank tensor fusion. This technique reduces the computational complexity of combining high-dimensional multimodal data by focusing on the most informative components of the joint representation.
Datasets and Evaluation
The model was evaluated on several benchmarks, including MOSI, MOSEI, and SIMS. Additionally, the authors introduced the Single-label Discrete Multimodal Emotion (SDME) dataset. SDME consists of 3,667 high-quality clips from TV dramas, providing clean, single-label discrete emotion annotations (e.g., "Angry," "Happy," "Anxious") along with unimodal annotations. This dataset serves as a rigorous testbed for evaluating how models handle modality deviations.
Key Results
- State-of-the-Art Performance: MRE outperformed existing models like MULT and MISA on the SIMS and SDME datasets. On SDME, it achieved an accuracy of 67.41% and an F1-Score of 69.60%, representing a significant improvement over previous benchmarks.
- Computational Efficiency: One of the most notable outcomes is the model's speed. On the MOSEI dataset, MRE completed processing in approximately 96 seconds, whereas the MULT model required over 2,500 seconds. This 26.5x speedup makes MRE highly suitable for real-time emotional analysis.
- Modality Consistency: Quantitative analysis revealed that visual cues were consistent with the true emotion 88.72% of the time (excluding neutral cases), while audio was consistent only 59.47% of the time, highlighting the necessity of the MRE's relevance-weighting mechanism.
Conclusion and Future Work
The MRE framework successfully mitigates the impact of semantic noise by prioritizing relevant modalities through attention and contrastive learning. While the current model focuses on static feature relevance, future iterations will incorporate temporal modeling to capture how emotions evolve over the duration of a clip. This advancement aims to further refine the extraction of multimodal relevant semantic information in complex human interactions.
Summary updated