Abstract
There remain two critical challenges that hinder the development of Emotion Recognition in Conversation (ERC). Firstly, there is a lack of exploration into mining deeper insights from the data itself for conversational emotion tasks. Secondly, the systems exhibit vulnerability to random modality feature missing, which is a common occurrence in realistic settings. Focusing on these two key challenges, we propose a novel framework for incomplete multimodal learning in ERC, called "Inverted Teacher-studEnt seArCH Network (ITEACH-Net)." ITEACH-Net comprises two novel components: the Emotion Context Changing Encoder (ECCE) and the Inverted Teacher-Student (ITS) framework. Specifically, leveraging the tendency for emotional states to exhibit local stability within conversational contexts, ECCE captures these patterns and further perceives their evolution over time. Recognizing the varying challenges of handling incomplete versus complete data, ITS employs a teacher-student framework to decouple the respective computations. Subsequently, through Neural Architecture Search, the student model develops enhanced computational capabilities for handling incomplete data compared to the teacher model. During testing, we design a novel evaluation method, testing the model's performance under different missing rate conditions without altering the model weights. We conduct experiments on three benchmark ERC datasets, and the results demonstrate that our ITEACH-Net outperforms existing methods in incomplete multimodal ERC. Codes will be made available.
Introduction
Emotion Recognition in Conversation (ERC) is a critical subset of multimodal machine learning that aims to track the emotional state of speakers throughout a dialogue. While current models perform well in controlled environments, they face two major hurdles in the real world: contextual dynamics and the "missing modality" problem. Conversations exhibit "emotional inertia" where states stay stable before shifting, a pattern many models fail to explicitly mine. Furthermore, real-world data is often incomplete due to background noise, sensor failure, or transcription errors.
To address these, we introduce ITEACH-Net (Inverted Teacher-studEnt seArCH Network), a framework designed to be robust against significant data loss by matching computational capacity to task difficulty.
Methodology
The ITEACH-Net framework is built upon three primary innovations: the Emotion Context Changing Encoder (ECCE), the Inverted Teacher-Student (ITS) framework, and Neural Architecture Search (NAS).
Inverted Teacher-Student (ITS) Framework
In a typical distillation setup, a large "teacher" model trains a smaller "student." However, we argue that handling incomplete data is a harder task than handling complete data. Therefore, we invert the roles: a simpler teacher model (trained on perfect data) guides a more complex student model (trained on incomplete data). This decoupling allows the student to focus on the challenging task of inferring missing information.
Emotion Context Changing Encoder (ECCE)
This module mimics human perception by first capturing "local stability"—the fact that emotions usually don't flip instantaneously—and then perceiving how these states evolve over the entire conversation. By explicitly building local context into the ECCE, the model avoids jittery predictions and understands the broader emotional trajectory.
Internal Architecture and NAS
To ensure the student has the necessary encoding capacity to handle missing information, we use Neural Architecture Search (NAS) to automatically design its internal architecture. This allows the student to be more computationally intensive than the teacher, providing the capacity needed to reconstruct missing modalities.
As shown in Figure 3, the Token Mixer utilizes a "Mix Map" and "Emotion Context Changing" mechanism to refine features based on conversational context. The Encoder Layer employs cross-attention (using queries, keys, and values) to fuse information across text, audio, and visual streams.
Experimental Results
We evaluated ITEACH-Net on three gold-standard benchmarks: IEMOCAP, CMU-MOSI, and CMU-MOSEI.
- Robustness to Data Loss: On the IEMOCAP (six-class) task, when 70% of the data was missing, ITEACH-Net achieved a 54.01% Weighted Average F1 (WAF), significantly outperforming state-of-the-art methods like GCNet and MMIN.
- Unified Model Evaluation (UME): Instead of training separate models for different missingness levels, we proposed a unified evaluation where a single model is tested across a spectrum of data loss (0% to 70%). ITEACH-Net maintained high performance across the entire range without weight adjustments.
- Feature Stability: Visualization via t-SNE showed that ITEACH-Net maintains distinct clusters for different emotions even as the data missing rate increases, whereas other models see their clusters collapse into uncertainty.
Conclusion
ITEACH-Net demonstrates that inverting the traditional teacher-student paradigm is a powerful strategy for robust multimodal learning. By acknowledging that "imagining" missing data requires more parameters than processing present data, the framework provides a logical path for handling real-world data imperfections. Future work will explore real-time constraints for edge deployment and the application of this logic to other dialogue-based tasks.
Summary updated