Amanda: Adaptively Modality-Balanced Domain Adaptation for Multimodal Emotion Recognition

Xinxin Zhang, Jun Sun, Simin Hong, Taihao LiACL 2024 · DOI

Introduction

Recognizing human emotion requires integrating multiple modalities: text, audio, and visual expressions. While deep learning has advanced this field, the scarcity of labeled data remains a bottleneck. Unsupervised Domain Adaptation (UDA) offers a solution by transferring knowledge from a labeled source domain to an unlabeled target domain. However, standard UDA often fails in multimodal settings due to "modality disparity," where differences between domains vary across modalities. This leads to the "lazy modality" problem, where the model over-relies on easily aligned modalities while neglecting others.

The Amanda Framework

Amanda (Adaptively modality-and-data-balanced domain adaptation) addresses these challenges through two primary innovations: Information Bottleneck (IB) representations and an adaptive modality balancing mechanism.

Information Bottleneck Representation

To prevent dominant modalities from overshadowing weaker ones, Amanda learns independent representations for each modality. Guided by IB theory, the model filters out domain-specific noise (e.g., lighting variations in video) while preserving information relevant to emotion labels. This ensures that each modality provides a clean, independent signal for adaptation.

Adaptive Modality Balancing

The core of Amanda is its dynamic weighting system. Instead of uniform alignment, it uses a surrogate loss function (such as $p$-norm or log-exp) to prioritize modalities with larger domain gaps. By adaptively increasing the weight of the most "difficult" modality during training, Amanda ensures a balanced alignment across all input streams.

The Amanda framework architecture, illustrating the feature encoders, independent representations, and the adaptive alignment process across source and target domains.

Experimental Setup and Datasets

The framework was evaluated on four major emotion recognition datasets: IEMOCAP (lab-based), MSP-IMPROV (acted), CMU-MOSEI (YouTube), and MELD (TV show snippets). These datasets provide a rigorous test for domain adaptation, spanning from controlled environments to noisy, real-world settings.

Results and Discussion

Amanda consistently outperforms state-of-the-art baselines like DANN, CDAN, and DALN. In the MELD $\rightarrow$ IEMOCAP task, Amanda achieved an F1 score of 58.77, significantly higher than DANN's 52.43.

Ablation studies highlight the importance of the adaptive weighting mechanism. As shown in the performance analysis, using a fixed weight ($p=1$ in the $p$-norm surrogate) leads to suboptimal results compared to higher $p$ values or the log-exp function, which more aggressively balance the modalities.

F1 scores for the MOSEI $\rightarrow$ IEMOCAP task across different weights and surrogate functions, demonstrating the superiority of adaptive balancing over fixed-weight baselines.

Conclusion

Amanda effectively mitigates the "lazy modality" problem in multimodal domain adaptation. By combining IB-based feature extraction with an adaptive weighting strategy, it achieves superior performance in emotion recognition across diverse datasets. Future work will explore the framework's applicability to other multimodal tasks and its robustness to missing modalities.

Summary updated

BibTeX
@inproceedings{zhang2024amanda,
  author = {Xinxin Zhang and Jun Sun and Simin Hong and Taihao Li},
  title = {{Amanda: Adaptively Modality-Balanced Domain Adaptation for Multimodal Emotion Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.859},
}
Figure