Introduction
Emotion Recognition in Conversations (ERC) is a critical task in affective computing, aimed at identifying the emotional state of speakers for every utterance within a dialogue. While humans naturally integrate facial expressions, vocal tones, and linguistic context to perceive emotion, automated systems face significant challenges. Most existing models struggle with three primary hurdles: inadequate multimodal fusion, extreme class imbalance in real-world datasets, and the semantic similarity between certain emotion categories (e.g., "Anger" vs. "Disgust").
As shown in the data distribution above, datasets like MELD are heavily skewed toward "Neutral" states, leaving minority classes like "Fear" and "Disgust" underrepresented. MultiEMO is proposed as a comprehensive framework to address these issues by bridging textual, audio, and visual cues through specialized extraction and fusion techniques.
The MultiEMO Framework
The MultiEMO architecture follows a four-stage pipeline designed to maximize the utility of multimodal data while mitigating the effects of noise and class imbalance.
VisExtNet: Focused Visual Extraction
Traditional models often process entire video frames, which include irrelevant background scenery that can act as noise. MultiEMO introduces VisExtNet, which utilizes a Multi-task Cascaded Convolutional Network (MTCNN) to isolate faces within a frame. These isolated faces are then processed through a pre-trained VGGFace2-ResNet101 to extract emotion-rich facial features, ensuring the model focuses on facial geometry rather than environmental context.
MultiAttn: Multimodal Fusion
At the core of the framework is the MultiAttn module, which employs stacked bidirectional multi-head cross-attention layers. Unlike "naive fusion" methods that simply concatenate feature vectors, MultiAttn treats one modality (e.g., text) as a "Query" and others (audio/visual) as "Keys" and "Values." This mechanism allows the model to learn complex, non-linear correlations, effectively using one sense to resolve ambiguity in another—much like human perception.
Specialized Loss Functions
To tackle the "long tail" problem of rare emotions and the difficulty of distinguishing similar emotional states, MultiEMO employs two specialized loss functions:
- Sample-Weighted Focal Contrastive (SWFC) Loss: This loss weights minority classes more heavily, focuses on "hard-to-classify" samples, and uses contrastive learning to push different emotion clusters further apart in the feature space.
- Soft-HGR Loss: This term maximizes the mathematical correlation between fused representations, ensuring that the different modalities work in harmony.
Experimental Results
MultiEMO was evaluated on the IEMOCAP and MELD benchmarks, outperforming state-of-the-art (SOTA) models like UniMSE and MVN. On IEMOCAP, it achieved a Weighted-F1 of 72.84%, and on MELD, it reached 66.74%.
The most significant breakthroughs occurred in the minority classes of the MELD dataset, which are notoriously difficult to classify. Performance for "Fear" jumped from a previous best of ~11.7% to 29.67%, while "Disgust" improved from ~21.8% to 36.75%. Ablation studies confirmed that both the MultiAttn fusion module and the SWFC loss are critical to these performance gains.
Conclusion and Future Work
MultiEMO demonstrates that focusing on facial features and employing sophisticated cross-modal attention can significantly enhance ERC performance, particularly for rare and easily confused emotions. Future work will focus on improving speaker identification in crowded scenes and exploring low-resource learning techniques to further close the performance gap between majority and minority emotion classes.
Summary updated