Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation

Chuan Zhang, Daoxin Zhang, Ruixiu Zhang, Jiawei Li, Jianke ZhuarXiv 2023 · DOI · arXiv

Abstract

Multimodal affective computing aims to decipher human emotions by integrating signals from video, audio, and text. However, the "emotional semantic gap"—where different modalities provide conflicting emotional cues—often leads to "semantic noise" that degrades model performance. This paper presents the Multimodal Relevance Estimation (MRE) network, a framework designed to identify and prioritize the most relevant modalities for a given emotional state. By employing category-level contrastive learning and a novel relevance estimation module, MRE effectively bridges the gap between heterogeneous data types. Experimental results on the newly introduced SDME dataset and existing benchmarks demonstrate that MRE achieves state-of-the-art accuracy while operating up to 26 times faster than leading complex fusion models.

Introduction: The Emotional Semantic Gap

In real-world scenarios, human emotional expression is rarely uniform across all sensory channels. A person might maintain a neutral facial expression while their voice betrays deep anxiety, or they might frown out of concentration rather than anger. Traditional "direct fusion" models often struggle with these discrepancies, as misleading signals in one modality can "pollute" the joint representation. This phenomenon is known as the Emotional Semantic Gap.

Figure 1: Illustration of the Emotional Semantic Gap, where visual, audio, and textual cues may point to different emotions, requiring a relevance-based approach to identify the true multimodal emotion.

The challenge lies in determining which modality is the most reliable indicator of the true emotional state at any given moment.

Methodology: The MRE Framework

The proposed Multimodal Relevance Estimation (MRE) framework addresses the semantic gap through a four-stage process designed for both accuracy and efficiency.

Figure 2: The MRE framework architecture, featuring feature extraction, relevance estimation via attention, category-level contrastive learning, and low-rank fusion.

1. Feature Representation

The model utilizes standard feature extraction tools: BERT for linguistic context, OpenFace for facial action units and landmarks, and Librosa for acoustic properties.

2. Relevant Semantic Estimation

At the core of MRE is an attention-based module that calculates a relevance weight for each modality. If the text is neutral but the audio indicates high arousal, the model dynamically assigns a higher weight to the audio features. This process is guided by a "weak supervision" loss ($L_{rs}$), allowing the model to learn dominant modalities even in the absence of explicit unimodal labels.

3. Contrastive Semantic Estimation

To ensure that features across different modalities are semantically aligned, the authors implement category-level contrastive learning. Unlike instance-level contrastive learning, this approach pulls "Happy" features from video closer to "Happy" features in audio across the entire dataset, creating a more robust and unified emotional space.

4. Low-Rank Fusion

To maintain high inference speeds, MRE employs low-rank tensor fusion. This technique reduces the computational complexity of combining high-dimensional multimodal data by focusing on the most informative components of the joint representation.

Datasets and Evaluation

The model was evaluated on several benchmarks, including MOSI, MOSEI, and SIMS. Additionally, the authors introduced the Single-label Discrete Multimodal Emotion (SDME) dataset. SDME consists of 3,667 high-quality clips from TV dramas, providing clean, single-label discrete emotion annotations (e.g., "Angry," "Happy," "Anxious") along with unimodal annotations. This dataset serves as a rigorous testbed for evaluating how models handle modality deviations.

Key Results

  • State-of-the-Art Performance: MRE outperformed existing models like MULT and MISA on the SIMS and SDME datasets. On SDME, it achieved an accuracy of 67.41% and an F1-Score of 69.60%, representing a significant improvement over previous benchmarks.
  • Computational Efficiency: One of the most notable outcomes is the model's speed. On the MOSEI dataset, MRE completed processing in approximately 96 seconds, whereas the MULT model required over 2,500 seconds. This 26.5x speedup makes MRE highly suitable for real-time emotional analysis.
  • Modality Consistency: Quantitative analysis revealed that visual cues were consistent with the true emotion 88.72% of the time (excluding neutral cases), while audio was consistent only 59.47% of the time, highlighting the necessity of the MRE's relevance-weighting mechanism.

Conclusion and Future Work

The MRE framework successfully mitigates the impact of semantic noise by prioritizing relevant modalities through attention and contrastive learning. While the current model focuses on static feature relevance, future iterations will incorporate temporal modeling to capture how emotions evolve over the duration of a clip. This advancement aims to further refine the extraction of multimodal relevant semantic information in complex human interactions.

Summary updated

Abstract

Human beings have rich ways of emotional expressions, including facial action, voice, and natural languages. Due to the diversity and complexity of different individuals, the emotions expressed by various modalities may be semantically irrelevant. Directly fusing information from different modalities may inevitably make the model subject to the noise from semantically irrelevant modalities. To tackle this problem, we propose a multimodal relevance estimation network to capture the relevant semantics among modalities in multimodal emotions. Specifically, we take advantage of an attention mechanism to reflect the semantic relevance weights of each modality. Moreover, we propose a relevant semantic estimation loss to weakly supervise the semantics of each modality. Furthermore, we make use of contrastive learning to optimize the similarity of category-level modality-relevant semantics across different modalities in feature space, thereby bridging the semantic gap between heterogeneous modalities. In order to better reflect the emotional state in the real interactive scenarios and perform the semantic relevance analysis, we collect a single-label discrete multimodal emotion dataset named SDME, which enables researchers to conduct multimodal semantic relevance research with large category bias. Experiments on continuous and discrete emotion datasets show that our model can effectively capture the relevant semantics, especially for the large deviations in modal semantics. The code and SDME dataset will be publicly available.
BibTeX
@misc{zhang2023bridging,
  author = {Chuan Zhang and Daoxin Zhang and Ruixiu Zhang and Jiawei Li and Jianke Zhu},
  title = {{Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation}},
  year = {2023},
  eprint = {2302.01555},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.01555},
}
Figure