Introduction
Recognizing human emotion requires integrating multiple modalities: text, audio, and visual expressions. While deep learning has advanced this field, the scarcity of labeled data remains a bottleneck. Unsupervised Domain Adaptation (UDA) offers a solution by transferring knowledge from a labeled source domain to an unlabeled target domain. However, standard UDA often fails in multimodal settings due to "modality disparity," where differences between domains vary across modalities. This leads to the "lazy modality" problem, where the model over-relies on easily aligned modalities while neglecting others.
The Amanda Framework
Amanda (Adaptively modality-and-data-balanced domain adaptation) addresses these challenges through two primary innovations: Information Bottleneck (IB) representations and an adaptive modality balancing mechanism.
Information Bottleneck Representation
To prevent dominant modalities from overshadowing weaker ones, Amanda learns independent representations for each modality. Guided by IB theory, the model filters out domain-specific noise (e.g., lighting variations in video) while preserving information relevant to emotion labels. This ensures that each modality provides a clean, independent signal for adaptation.
Adaptive Modality Balancing
The core of Amanda is its dynamic weighting system. Instead of uniform alignment, it uses a surrogate loss function (such as $p$-norm or log-exp) to prioritize modalities with larger domain gaps. By adaptively increasing the weight of the most "difficult" modality during training, Amanda ensures a balanced alignment across all input streams.
Experimental Setup and Datasets
The framework was evaluated on four major emotion recognition datasets: IEMOCAP (lab-based), MSP-IMPROV (acted), CMU-MOSEI (YouTube), and MELD (TV show snippets). These datasets provide a rigorous test for domain adaptation, spanning from controlled environments to noisy, real-world settings.
Results and Discussion
Amanda consistently outperforms state-of-the-art baselines like DANN, CDAN, and DALN. In the MELD $\rightarrow$ IEMOCAP task, Amanda achieved an F1 score of 58.77, significantly higher than DANN's 52.43.
Ablation studies highlight the importance of the adaptive weighting mechanism. As shown in the performance analysis, using a fixed weight ($p=1$ in the $p$-norm surrogate) leads to suboptimal results compared to higher $p$ values or the log-exp function, which more aggressively balance the modalities.
Conclusion
Amanda effectively mitigates the "lazy modality" problem in multimodal domain adaptation. By combining IB-based feature extraction with an adaptive weighting strategy, it achieves superior performance in emotion recognition across diverse datasets. Future work will explore the framework's applicability to other multimodal tasks and its robustness to missing modalities.
Summary updated