Abstract
There remain two critical challenges that hinder the development of Emotion Recognition in Conversation (ERC). Firstly, there is a lack of exploration into mining deeper insights from the data itself for conversational emotion tasks. Secondly, the systems exhibit vulnerability to random modality feature missing, which is a common occurrence in realistic settings. Focusing on these two key challenges, we propose a novel framework for incomplete multimodal learning in ERC, called "Inverted Teacher-studEnt seArCH Network (ITEACH-Net)." ITEACH-Net comprises two novel components: the Emotion Context Changing Encoder (ECCE) and the Inverted Teacher-Student (ITS) framework. Specifically, leveraging the tendency for emotional states to exhibit local stability within conversational contexts, ECCE captures these patterns and further perceives their evolution over time. Recognizing the varying challenges of handling incomplete versus complete data, ITS employs a teacher-student framework to decouple the respective computations. Subsequently, through Neural Architecture Search, the student model develops enhanced computational capabilities for handling incomplete data compared to the teacher model. Experiments on three benchmark ERC datasets demonstrate that ITEACH-Net outperforms existing methods in incomplete multimodal ERC.
Introduction
Emotion Recognition in Conversation (ERC) is a complex task requiring the integration of text, audio, and visual modalities to identify a speaker's emotional state. Despite significant progress, current models struggle with two primary issues: the dynamic nature of emotional context and the prevalence of missing data in real-world scenarios. Emotions are not static; they typically exhibit "local stability" (staying the same for a short duration) before undergoing "global evolution" as the conversation progresses. Furthermore, hardware glitches or environmental factors often lead to incomplete data streams. ITEACH-Net addresses these by explicitly modeling emotional shifts and introducing a robust "inverted" learning framework.
Methodology
Emotion Context Changing Encoder (ECCE)
To capture the nuances of conversational dynamics, the ECCE module operates in two stages. First, it identifies local context within a sliding window, acknowledging that emotional states are generally stable over short intervals. Second, it computes global context changes across the entire dialogue. This allows the model to perceive how the "emotional atmosphere" evolves, providing a more coherent representation than standard self-attention mechanisms.
Inverted Teacher-Student (ITS) Framework
Traditional knowledge distillation typically involves a large "teacher" model guiding a smaller "student." ITEACH-Net inverts this logic to handle data incompleteness.
In the ITS framework, the Teacher Model is trained on complete data using a relatively straightforward architecture. The Student Model, however, is trained on incomplete (masked) data. The core insight is that processing incomplete data is a more difficult task—requiring the model to simultaneously impute missing information and classify emotions. Consequently, the student is designed with a more complex architecture and higher parameter count than the teacher to effectively bridge the performance gap.
Neural Architecture Search (NAS)
To optimize the student model's increased capacity, the authors employ Neural Architecture Search. The student architecture includes a "Router" that dynamically selects the most effective computational path—choosing between different "Token Mixers" like Attention or MLPs—based on the specific input and which modalities are missing.
Experiments and Results
Datasets and Evaluation
The framework was evaluated on three major benchmarks: IEMOCAP, CMU-MOSI, and CMU-MOSEI. A key contribution is the Unified Model Evaluation (UME) method. Instead of training separate models for different levels of data loss, the researchers trained a single model and tested its performance across a spectrum of missing rates (from 0% to 70%), simulating real-world signal fluctuations.
Key Findings
- Superior Robustness: ITEACH-Net consistently outperformed state-of-the-art models like GCNet and MMIN. On the IEMOCAP 4-class task with a 70% missing rate, it achieved a Weighted Average F1 (WAF) of 71.73%.
- Capacity Matters: Ablation studies confirmed that the complex, NAS-enhanced student model suffered the smallest performance degradation as data became increasingly sparse.
- Stable Representations: Visual analysis via t-SNE showed that the student model maintains well-clustered internal representations even under high missing rates, successfully emulating the clean knowledge of the teacher.
Conclusion
ITEACH-Net introduces a paradigm shift in incomplete multimodal learning by proposing that harder tasks (incomplete data) require more complex models (the inverted student). By combining this framework with the Emotion Context Changing Encoder, the model achieves new levels of robustness for real-world ERC applications. Future work will explore real-time streaming scenarios and the application of the ITS framework to other multimodal domains like video captioning.
Summary updated