ITEACH-Net: Inverted Teacher-studEnt seArCH Network for Emotion Recognition in Conversation

Haiyang Sun, Zheng Lian, Licai Sun, Bin Liu, Jianhua Tao, Liu, Bin, Tao, Jianhua

Abstract

There remain two critical challenges that hinder the development of Emotion Recognition in Conversation (ERC). Firstly, there is a lack of exploration into mining deeper insights from the data itself for conversational emotion tasks. Secondly, the systems exhibit vulnerability to random modality feature missing, which is a common occurrence in realistic settings. Focusing on these two key challenges, we propose a novel framework for incomplete multimodal learning in ERC, called "Inverted Teacher-studEnt seArCH Network (ITEACH-Net)." ITEACH-Net comprises two novel components: the Emotion Context Changing Encoder (ECCE) and the Inverted Teacher-Student (ITS) framework. Specifically, leveraging the tendency for emotional states to exhibit local stability within conversational contexts, ECCE captures these patterns and further perceives their evolution over time. Recognizing the varying challenges of handling incomplete versus complete data, ITS employs a teacher-student framework to decouple the respective computations. Subsequently, through Neural Architecture Search, the student model develops enhanced computational capabilities for handling incomplete data compared to the teacher model. Experiments on three benchmark ERC datasets demonstrate that ITEACH-Net outperforms existing methods in incomplete multimodal ERC.

Introduction

Emotion Recognition in Conversation (ERC) is a complex task requiring the integration of text, audio, and visual modalities to identify a speaker's emotional state. Despite significant progress, current models struggle with two primary issues: the dynamic nature of emotional context and the prevalence of missing data in real-world scenarios. Emotions are not static; they typically exhibit "local stability" (staying the same for a short duration) before undergoing "global evolution" as the conversation progresses. Furthermore, hardware glitches or environmental factors often lead to incomplete data streams. ITEACH-Net addresses these by explicitly modeling emotional shifts and introducing a robust "inverted" learning framework.

Methodology

Emotion Context Changing Encoder (ECCE)

To capture the nuances of conversational dynamics, the ECCE module operates in two stages. First, it identifies local context within a sliding window, acknowledging that emotional states are generally stable over short intervals. Second, it computes global context changes across the entire dialogue. This allows the model to perceive how the "emotional atmosphere" evolves, providing a more coherent representation than standard self-attention mechanisms.

Inverted Teacher-Student (ITS) Framework

Traditional knowledge distillation typically involves a large "teacher" model guiding a smaller "student." ITEACH-Net inverts this logic to handle data incompleteness.

The ITEACH-Net framework featuring the Inverted Teacher-Student (ITS) architecture and Emotion Context Changing Encoder (ECCE).

In the ITS framework, the Teacher Model is trained on complete data using a relatively straightforward architecture. The Student Model, however, is trained on incomplete (masked) data. The core insight is that processing incomplete data is a more difficult task—requiring the model to simultaneously impute missing information and classify emotions. Consequently, the student is designed with a more complex architecture and higher parameter count than the teacher to effectively bridge the performance gap.

Neural Architecture Search (NAS)

To optimize the student model's increased capacity, the authors employ Neural Architecture Search. The student architecture includes a "Router" that dynamically selects the most effective computational path—choosing between different "Token Mixers" like Attention or MLPs—based on the specific input and which modalities are missing.

Detailed architecture of the Encoder Layer, ITEACH Encoder, and Token Mixer used within the student model to adapt to varying data quality.

Experiments and Results

Datasets and Evaluation

The framework was evaluated on three major benchmarks: IEMOCAP, CMU-MOSI, and CMU-MOSEI. A key contribution is the Unified Model Evaluation (UME) method. Instead of training separate models for different levels of data loss, the researchers trained a single model and tested its performance across a spectrum of missing rates (from 0% to 70%), simulating real-world signal fluctuations.

Key Findings

  • Superior Robustness: ITEACH-Net consistently outperformed state-of-the-art models like GCNet and MMIN. On the IEMOCAP 4-class task with a 70% missing rate, it achieved a Weighted Average F1 (WAF) of 71.73%.
  • Capacity Matters: Ablation studies confirmed that the complex, NAS-enhanced student model suffered the smallest performance degradation as data became increasingly sparse.
  • Stable Representations: Visual analysis via t-SNE showed that the student model maintains well-clustered internal representations even under high missing rates, successfully emulating the clean knowledge of the teacher.

Conclusion

ITEACH-Net introduces a paradigm shift in incomplete multimodal learning by proposing that harder tasks (incomplete data) require more complex models (the inverted student). By combining this framework with the Emotion Context Changing Encoder, the model achieves new levels of robustness for real-world ERC applications. Future work will explore real-time streaming scenarios and the application of the ITS framework to other multimodal domains like video captioning.

Summary updated

Abstract

There remain two critical challenges that hinder the development of ERC. Firstly, there is a lack of exploration into mining deeper insights from the data itself for conversational emotion tasks. Secondly, the systems exhibit vulnerability to random modality feature missing, which is a common occurrence in realistic settings. Focusing on these two key challenges, we propose a novel framework for incomplete multimodal learning in ERC, called "Inverted Teacher-studEnt seArCH Network (ITEACH-Net)." ITEACH-Net comprises two novel components: the Emotion Context Changing Encoder (ECCE) and the Inverted Teacher-Student (ITS) framework. Specifically, leveraging the tendency for emotional states to exhibit local stability within conversational contexts, ECCE captures these patterns and further perceives their evolution over time. Recognizing the varying challenges of handling incomplete versus complete data, ITS employs a teacher-student framework to decouple the respective computations. Subsequently, through Neural Architecture Search, the student model develops enhanced computational capabilities for handling incomplete data compared to the teacher model. During testing, we design a novel evaluation method, testing the model's performance under different missing rate conditions without altering the model weights. We conduct experiments on three benchmark ERC datasets, and the results demonstrate that our ITEACH-Net outperforms existing methods in incomplete multimodal ERC. We believe ITEACH-Net can inspire relevant research on the intrinsic nature of emotions within conversation scenarios and pave a more robust route for incomplete learning techniques. Codes will be made available.
BibTeX
@misc{sun2023iteach,
  author = {Haiyang Sun and Zheng Lian and Licai Sun and Bin Liu and Jianhua Tao and Liu, Bin and Tao, Jianhua},
  title = {{ITEACH-Net: Inverted Teacher-studEnt seArCH Network for Emotion Recognition in Conversation}},
  year = {2023},
  eprint = {2312.15583},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.15583},
}
Figure