MultiEMO: An Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations

Tao Shi, Shao-Lun HuangACL 2023 · OpenReview

Introduction

Emotion Recognition in Conversations (ERC) is a critical task in affective computing, aimed at identifying the emotional state of speakers for every utterance within a dialogue. While humans naturally integrate facial expressions, vocal tones, and linguistic context to perceive emotion, automated systems face significant challenges. Most existing models struggle with three primary hurdles: inadequate multimodal fusion, extreme class imbalance in real-world datasets, and the semantic similarity between certain emotion categories (e.g., "Anger" vs. "Disgust").

Class distribution in MELD and IEMOCAP datasets, highlighting the extreme class imbalance in MELD.

As shown in the data distribution above, datasets like MELD are heavily skewed toward "Neutral" states, leaving minority classes like "Fear" and "Disgust" underrepresented. MultiEMO is proposed as a comprehensive framework to address these issues by bridging textual, audio, and visual cues through specialized extraction and fusion techniques.

The MultiEMO Framework

The MultiEMO architecture follows a four-stage pipeline designed to maximize the utility of multimodal data while mitigating the effects of noise and class imbalance.

The MultiEMO architecture overview, illustrating unimodal feature extraction, multimodal fusion via MultiAttn, and emotion classification.

VisExtNet: Focused Visual Extraction

Traditional models often process entire video frames, which include irrelevant background scenery that can act as noise. MultiEMO introduces VisExtNet, which utilizes a Multi-task Cascaded Convolutional Network (MTCNN) to isolate faces within a frame. These isolated faces are then processed through a pre-trained VGGFace2-ResNet101 to extract emotion-rich facial features, ensuring the model focuses on facial geometry rather than environmental context.

MultiAttn: Multimodal Fusion

At the core of the framework is the MultiAttn module, which employs stacked bidirectional multi-head cross-attention layers. Unlike "naive fusion" methods that simply concatenate feature vectors, MultiAttn treats one modality (e.g., text) as a "Query" and others (audio/visual) as "Keys" and "Values." This mechanism allows the model to learn complex, non-linear correlations, effectively using one sense to resolve ambiguity in another—much like human perception.

Specialized Loss Functions

To tackle the "long tail" problem of rare emotions and the difficulty of distinguishing similar emotional states, MultiEMO employs two specialized loss functions:

  • Sample-Weighted Focal Contrastive (SWFC) Loss: This loss weights minority classes more heavily, focuses on "hard-to-classify" samples, and uses contrastive learning to push different emotion clusters further apart in the feature space.
  • Soft-HGR Loss: This term maximizes the mathematical correlation between fused representations, ensuring that the different modalities work in harmony.

Experimental Results

MultiEMO was evaluated on the IEMOCAP and MELD benchmarks, outperforming state-of-the-art (SOTA) models like UniMSE and MVN. On IEMOCAP, it achieved a Weighted-F1 of 72.84%, and on MELD, it reached 66.74%.

The most significant breakthroughs occurred in the minority classes of the MELD dataset, which are notoriously difficult to classify. Performance for "Fear" jumped from a previous best of ~11.7% to 29.67%, while "Disgust" improved from ~21.8% to 36.75%. Ablation studies confirmed that both the MultiAttn fusion module and the SWFC loss are critical to these performance gains.

Conclusion and Future Work

MultiEMO demonstrates that focusing on facial features and employing sophisticated cross-modal attention can significantly enhance ERC performance, particularly for rare and easily confused emotions. Future work will focus on improving speaker identification in crowded scenes and exploring low-resource learning techniques to further close the performance gap between majority and minority emotion classes.

Summary updated

BibTeX
@inproceedings{shi2023multiemo,
  author = {Tao Shi and Shao-Lun Huang},
  title = {{MultiEMO: An Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=j9H0otIN8U},
}
Figure