Abstract
Dialogue Emotion Recognition (ERC) is a critical task for building empathetic AI, yet it faces significant challenges in handling multimodal data and detecting sudden changes in emotional states. We present the Cross-modal Fusion Network with Emotion-Shift Awareness (CFN-ESA), a framework designed to overcome the "modality equality fallacy" and "contextual inertia." By prioritizing textual information as the primary emotional anchor and introducing an auxiliary emotion-shift detection task, CFN-ESA achieves state-of-the-art results on the MELD and IEMOCAP benchmarks.
Introduction: The Challenges of ERC
Emotion recognition in conversation is inherently difficult because human emotions are both context-dependent and multi-faceted. Most existing research in multimodal ERC suffers from two major limitations:
- The Modality Equality Fallacy: Many models treat text, video, and audio as equally important. In practice, transcribed text usually carries the core emotional message, while audio and visual cues serve as vital supplements to detect nuances like sarcasm or suppressed anger.
- The Contextual Inertia Problem: Models often over-rely on the assumption that emotional states are consistent over time. This leads to failures during "emotion shifts," where a speaker's state changes abruptly (e.g., from neutral to anger).
The CFN-ESA Framework
To address these gaps, the authors propose CFN-ESA, which integrates three specialized modules into a unified pipeline to process visual (DenseNet), textual (RoBERTa), and acoustic (OpenSmile) modalities.
1. RUME (Recurrence based Uni-Modality Encoder)
This module extracts contextual cues within each individual modality. To bridge the "heterogeneity gap"—the mathematical difference between pixels, audio waves, and text—RUME utilizes shared parameters across all three modalities. This forces the model to project disparate data types into a more compatible feature space before fusion.
2. ACME (Attention based Cross-Modality Encoder)
Rather than simple concatenation, ACME employs a text-centric fusion strategy. It treats textual features as the "query" and uses visual and acoustic data as "keys" and "values" within a multi-head attention mechanism. This allows the model to selectively use non-verbal cues to refine the textual representation, preventing noise in audio or video from degrading performance.
3. LESM (Label based Emotion-Shift Module)
LESM introduces an auxiliary task during training: predicting whether an emotion shift occurred between consecutive utterances. By learning to identify these transitions, the model becomes more sensitive to sudden changes in a speaker's state, effectively overcoming contextual inertia.
Experimental Results
The model was evaluated on two benchmark datasets: MELD (multiparty, noisy, class-imbalanced) and IEMOCAP (dyadic, high-quality).
- MELD: CFN-ESA achieved a Weighted F1 (W-F1) score of 66.70%, outperforming the previous state-of-the-art (MetaDrop) and significantly exceeding text-only baselines. It showed particular strength in recognizing minority classes like "disgust" and "fear."
- IEMOCAP: The model reached a W-F1 of 71.04%, representing a nearly 5% improvement over unimodal baselines.
Ablation studies highlighted that removing the LESM (emotion-shift) module led to a consistent drop in performance, confirming that explicit shift awareness is a critical component for accurate ERC.
Key Insights and Future Directions
The success of CFN-ESA underscores several key principles for multimodal AI:
- Text is the Anchor: While multimodal data is superior, text remains the most reliable source of emotional information. The text-centric fusion in ACME prevents the model from being distracted by noise in other channels.
- Shared Parameters for Alignment: Using shared weights in early encoding stages helps align diverse modalities into a shared feature space.
- Shift Awareness: There is a direct correlation between a model's ability to spot when an emotion changes and its ability to identify what the new emotion is.
Future work will focus on improving robustness against visual noise (e.g., poor lighting or obscured faces) and testing the model on "in-the-wild" datasets to ensure generalizability beyond scripted or lab-controlled environments.
Summary updated