AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations

Sheng Wu, Jiaxing Liu, Longbiao Wang, Dongxiao He, Xiaobao Wang, Jianwu DangarXiv 2024 · DOI · arXiv

Introduction

Emotion Recognition in Conversations (ERC) is a complex task because human emotion is rarely conveyed through a single channel; it involves a combination of words (text), tone (audio), and facial expressions (visual). Current research faces significant hurdles in effective multimodal fusion, often resulting in noise redundancy or "over-smoothing" in graph-based models where distinct features are lost.

AIMDiT (Modality Augmentation and Interaction Network via Dimension Transformation) introduces a novel "spatial" approach to this problem. By transforming 1D modality vectors into 2D tensors, the model utilizes Inception-style convolutions to simultaneously capture patterns within a single modality (temporal) and across different modalities (cross-modal), setting a new performance standard on the MELD benchmark.

Methodology

The AIMDiT architecture consists of two primary stages designed to enhance and then fuse multimodal features: the Modality Augmentation Network (MAN) and the Modality Interaction Network (MIN).

Modality Augmentation Network (MAN)

The core innovation is Dimension Transformation. 1D feature vectors from text, audio, and visual encoders are stacked into a 2D tensor. In this 2D space, one axis represents time (intra-modal information), and the other represents the different modalities (inter-modal information). By treating the fused data like an image, the model applies 2D Inception Blocks. These blocks use multi-scale kernels to capture emotional nuances at various temporal scales and across modality boundaries simultaneously.

Modality Interaction Network (MIN)

Once features are augmented, the MIN handles the final fusion. It employs a Cross-modal Transformer (CMT), where one modality acts as a guide to reconstruct another, ensuring the model learns complementary signals. This is paired with a Self-modal Transformer (SMT) to maintain the integrity of individual modality features. The results are averaged to create a robust final representation for emotion classification.

Overview of the AIMDiT architecture, illustrating the 1D-to-2D dimension transformation and the subsequent interaction modules.

Experimental Evaluation

The model was evaluated on the MELD (Multimodal Emotion Lines Dataset), which includes 13,708 utterances from the TV series Friends across seven emotion categories (Anger, Disgust, Fear, Joy, Neutral, Sadness, Surprise). The authors utilized RoBERTa for text, Mel-spectrograms for audio, and EfficientNet for visual feature extraction.

Key Results and Analysis

AIMDiT achieved new state-of-the-art (SOTA) results on the MELD dataset, notably improving accuracy and F1-scores over previous benchmarks.

Performance comparison of AIMDiT against state-of-the-art methods on the MELD dataset.

Performance Metrics

  • Accuracy (Acc-7): 64.83% (a 2.34% improvement over the previous SOTA, MM-DFN).
  • Weighted F1-Score: 62.33% (a 2.87% improvement).
  • Category Gains: The model showed particularly strong performance in identifying "Neutral" (80.24%) and "Surprise" (56.97%).

Ablation and Depth Studies

Ablation studies revealed that removing the MAN module caused the largest performance drop (down to 61.72% accuracy), proving that the 2D dimension transformation is the primary driver of the model's success. Additionally, researchers found that performance peaked at 2 MAN layers; beyond this, redundant information began to accumulate, degrading accuracy.

Conclusion and Future Directions

AIMDiT demonstrates the power of treating multimodal fusion as a computer vision problem through spatial transformation. While highly effective, the authors noted a performance gap in audio-visual combinations compared to text-based pairings, likely due to high distribution differences. Future work will focus on better aligning sound and sight modalities and addressing data imbalances for rare emotional classes like "Disgust" and "Fear."

Summary updated

Abstract

Emotion Recognition in Conversations (ERC) is a popular task in natural language processing, which aims to recognize the emotional state of the speaker in conversations. While current research primarily emphasizes contextual modeling, there exists a dearth of investigation into effective multimodal fusion methods. We propose a novel framework called AIMDiT to solve the problem of multimodal fusion of deep features. Specifically, we design a Modality Augmentation Network which performs rich representation learning through dimension transformation of different modalities and parameter-efficient inception block. On the other hand, the Modality Interaction Network performs interaction fusion of extracted inter-modal features and intra-modal features. Experiments conducted using our AIMDiT framework on the public benchmark dataset MELD reveal 2.34% and 2.87% improvements in terms of the Acc-7 and w-F1 metrics compared to the state-of-the-art (SOTA) models.
BibTeX
@misc{wu2024aimdit,
  author = {Sheng Wu and Jiaxing Liu and Longbiao Wang and Dongxiao He and Xiaobao Wang and Jianwu Dang},
  title = {{AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations}},
  year = {2024},
  eprint = {2407.00743},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.00743},
}
Figure