Introduction
Emotion Recognition in Conversations (ERC) is a complex task because human emotion is rarely conveyed through a single channel; it involves a combination of words (text), tone (audio), and facial expressions (visual). Current research faces significant hurdles in effective multimodal fusion, often resulting in noise redundancy or "over-smoothing" in graph-based models where distinct features are lost.
AIMDiT (Modality Augmentation and Interaction Network via Dimension Transformation) introduces a novel "spatial" approach to this problem. By transforming 1D modality vectors into 2D tensors, the model utilizes Inception-style convolutions to simultaneously capture patterns within a single modality (temporal) and across different modalities (cross-modal), setting a new performance standard on the MELD benchmark.
Methodology
The AIMDiT architecture consists of two primary stages designed to enhance and then fuse multimodal features: the Modality Augmentation Network (MAN) and the Modality Interaction Network (MIN).
Modality Augmentation Network (MAN)
The core innovation is Dimension Transformation. 1D feature vectors from text, audio, and visual encoders are stacked into a 2D tensor. In this 2D space, one axis represents time (intra-modal information), and the other represents the different modalities (inter-modal information). By treating the fused data like an image, the model applies 2D Inception Blocks. These blocks use multi-scale kernels to capture emotional nuances at various temporal scales and across modality boundaries simultaneously.
Modality Interaction Network (MIN)
Once features are augmented, the MIN handles the final fusion. It employs a Cross-modal Transformer (CMT), where one modality acts as a guide to reconstruct another, ensuring the model learns complementary signals. This is paired with a Self-modal Transformer (SMT) to maintain the integrity of individual modality features. The results are averaged to create a robust final representation for emotion classification.
Experimental Evaluation
The model was evaluated on the MELD (Multimodal Emotion Lines Dataset), which includes 13,708 utterances from the TV series Friends across seven emotion categories (Anger, Disgust, Fear, Joy, Neutral, Sadness, Surprise). The authors utilized RoBERTa for text, Mel-spectrograms for audio, and EfficientNet for visual feature extraction.
Key Results and Analysis
AIMDiT achieved new state-of-the-art (SOTA) results on the MELD dataset, notably improving accuracy and F1-scores over previous benchmarks.
Performance Metrics
- Accuracy (Acc-7): 64.83% (a 2.34% improvement over the previous SOTA, MM-DFN).
- Weighted F1-Score: 62.33% (a 2.87% improvement).
- Category Gains: The model showed particularly strong performance in identifying "Neutral" (80.24%) and "Surprise" (56.97%).
Ablation and Depth Studies
Ablation studies revealed that removing the MAN module caused the largest performance drop (down to 61.72% accuracy), proving that the 2D dimension transformation is the primary driver of the model's success. Additionally, researchers found that performance peaked at 2 MAN layers; beyond this, redundant information began to accumulate, degrading accuracy.
Conclusion and Future Directions
AIMDiT demonstrates the power of treating multimodal fusion as a computer vision problem through spatial transformation. While highly effective, the authors noted a performance gap in audio-visual combinations compared to text-based pairings, likely due to high distribution differences. Future work will focus on better aligning sound and sight modalities and addressing data imbalances for rare emotional classes like "Disgust" and "Fear."
Summary updated