MATCH: Modality-Calibrated Hypergraph Fusion Network for Conversational Emotion Recognition

Jiandong Shi, Ming Li, Lu Bai, Feilong Cao, Ke Lu, Jiye Liang

Introduction

Multimodal Emotion Recognition in Conversations (MERC) is a critical task in artificial intelligence, aimed at identifying the emotional states of speakers by integrating textual, acoustic, and visual information. Unlike static emotion recognition, MERC must account for the dynamic nature of dialogue, where emotional states are influenced by past interactions and the complex relationships between participants.

Example of a multimodal conversation from the MELD dataset, illustrating the complexity of emotional cues across text and audio.

Conversations are inherently non-linear. A speaker's mood might shift based on a specific word, a change in pitch, or a facial expression from a peer. Existing models often struggle with two primary issues: insufficient calibration of raw multimodal data and the inability to model high-order multivariate dependencies effectively. To address these challenges, the MATCH framework introduces a novel approach to modality calibration and hypergraph-based semantic learning.

The MATCH Framework

The MATCH (Modality-Calibrated Hypergraph Fusion Network) architecture is designed to untangle conversational complexities through a multi-stage process: encoding, calibration, and fusion.

The complete MATCH architecture, featuring Utterance Encoding, Conversational Entity Calibration, and Emotion-aligned Hypergraph Fusion.

Conversational Entity Calibration

A significant bottleneck in MERC is "modality interference," where noise in one data stream (e.g., poor audio quality) obscures signals in another. MATCH addresses this by refining representations of both the utterance and the speaker. Using a contrastive learning strategy involving Gaussian noise and re-normalization, the model learns robust, distinct features for each speaker across different modalities, ensuring that the underlying emotional signal is preserved despite environmental noise.

Hypergraph Semantic Learning and Line Graph Expansion

Traditional graph models use nodes and edges to represent relationships between two points. However, conversations involve group dynamics that are better represented by hypergraphs, where "hyperedges" can connect multiple nodes simultaneously. MATCH groups utterances by context or modality to capture these high-order relationships.

To further enhance the semantic depth of these connections, the authors employ a "line graph" expansion. By treating the relationships (edges) as nodes themselves, the model can perceive "emotional pathways"—the flow of emotion through a conversation. This is coupled with an emotion-alignment strategy that uses textual features as a reliable "anchor" to calibrate visual and acoustic distributions via KL divergence.

Experimental Evaluation

The researchers evaluated MATCH on two prominent benchmarks:

  • IEMOCAP: A dyadic dataset featuring scripted and improvised performances.
  • MELD: A multi-party dataset derived from the TV show Friends, known for its realistic and noisy conversational environment.

Key Results

MATCH demonstrated superior performance compared to ten baseline models, including those utilizing external common-sense knowledge bases. On IEMOCAP, the model achieved an Accuracy of 72.64% and a Weighted-F1 of 72.55%. On the more challenging MELD dataset, it reached an Accuracy of 68.16% and a Weighted-F1 of 67.04%.

Discussion and Insights

The success of MATCH highlights several key principles for future MERC research:

  1. Calibration as a Universal Booster: Plugging the Entity Calibration module into existing models like DiaRNN or M3Net resulted in consistent performance gains, suggesting that data calibration is a fundamental requirement for multimodal fusion.
  2. Textual Anchoring: By aligning audio and visual features with textual distributions, the model effectively mitigates the impact of noisy non-verbal cues.
  3. Structural Sensitivity: While deeper hypergraph structures capture more context, the authors noted a performance drop after four layers due to "over-smoothing." This indicates a delicate balance between context integration and feature distinctiveness.

Conclusion

MATCH offers a robust solution for conversational emotion recognition by rethinking how we calibrate multimodal data and model complex speaker interactions. By leveraging hypergraph structures and fine-grained entity calibration, the framework achieves state-of-the-art results without the need for external knowledge, paving the way for more nuanced and context-aware affective computing.

Summary updated

Abstract

Multimodal emotion recognition aims to identify emotions by integrating multimodal features derived from spoken utterances. However, existing work often neglects the calibration of conversational entities, focusing mainly on extracting potential intra- or cross-modal information. This leads to the underutilization of utterance information that is essential for accurately characterizing emotion. Additionally, the lack of effective modeling of conversational patterns limits the ability to capture emotional pathways across contexts, modalities and speakers, impacting the overall emotional understanding. In this study, we propose the modality-calibrated hypergraph fusion network (MATCH), which leverages multimodal fusion and hypergraph learning techniques to address these challenges. In particular, we introduce an entity calibration strategy that refines the representations of conversational entities both at the modality and context levels, allowing for deeper insights into emotion-related cues. Furthermore, we present an emotion-aligned hypergraph fusion method that incorporates a line graph to explore conversational patterns, facilitating flexible knowledge transfer across modalities through hyperedge-level and graph-level alignments. Experiments demonstrate that MATCH outperforms state-of-the-art approaches on two benchmark datasets.
BibTeX
@inproceedings{shi2025match,
  author = {Jiandong Shi and Ming Li and Lu Bai and Feilong Cao and Ke Lu and Jiye Liang},
  title = {{MATCH: Modality-Calibrated Hypergraph Fusion Network for Conversational Emotion Recognition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/686},
}
Figure