GSDNet: Revisiting Incomplete Multimodality-Diffusion Emotion Recognition from the Perspective of Graph Spectrum

Yuntao Shou, Jun Yao, Tao Meng, Wei Ai, Cen Chen, Keqin LiIJCAI 2025 · DOI

Abstract

Multimodal Emotion Recognition in Conversations (MERC) often faces the challenge of incomplete data due to sensor failure or environmental noise. While existing methods use Graph Neural Networks (GNNs) and Diffusion Models (DMs) to recover missing information, adding noise directly to the graph's spatial structure often leads to "graph collapse." This paper introduces GSDNet (Graph Spectral Diffusion Network), which shifts the diffusion process to the spectral domain. By diffusing the eigenvalues of the conversation graph rather than its adjacency matrix, GSDNet preserves the underlying topological structure while effectively reconstructing missing features. Experimental results on CMU-MOSI and CMU-MOSEI demonstrate that GSDNet achieves state-of-the-art performance and remarkable robustness, even with up to 70% data loss.

Introduction

In an ideal setting, multimodal emotion recognition models leverage perfectly synchronized text, audio, and video. However, real-world data is frequently "incomplete"—microphones fail, speakers move out of frame, or background noise corrupts transcripts. Current state-of-the-art methods attempt to solve this using GNNs to model speaker relationships and DMs to "hallucinate" missing data.

Combining these approaches is difficult because standard graph diffusion models add noise directly to the adjacency matrix. This often destroys the local structure and connectivity before the model can learn to repair it, leading to a rapid drop in the signal-to-noise ratio (SNR). GSDNet addresses this by revisiting graph diffusion from a spectral perspective, ensuring that the global topological information remains intact throughout the recovery process.

Methodology

The core innovation of GSDNet is the shift from spatial to spectral diffusion. The framework consists of three primary stages:

  1. Node Construction: Utterances from different modalities are aligned into a common feature space using 1D convolutional layers and positional embeddings.
  2. Graph Spectral Diffusion: Instead of adding noise to the connections between utterances, the model performs an eigendecomposition of the adjacency matrix. It then applies a Score-based Generative Model (SGM) to the eigenvalues and the node features.
  3. Forward Process: Gradually adds Gaussian noise to the eigenvalues and features.
  4. Reverse Process: A scoring network learns to "denoise" these spectral components, reconstructing the missing modality's distribution based on the remaining observed modalities.
  5. Fusion and Classification: Once the missing data is recovered, a standard Graph Convolutional Network (GCN) fuses the modalities to predict the final emotion label.
The GSDNet architecture, illustrating the transition from node construction to spectral diffusion and final classification.

By strictly limiting noise to the spectral space (eigenvalues) and not the eigenvectors (the "shape" of the connections), the global topological information is preserved. This maintains a higher SNR during training, allowing the scoring network to capture more nuanced gradients of the original data distribution.

Experimental Results

GSDNet was evaluated on two gold-standard benchmarks: CMU-MOSI and CMU-MOSEI. The researchers utilized RoBERTa-Large for text, DenseNet for video, and wav2vec for acoustic features.

Performance and Robustness

GSDNet achieved state-of-the-art (SOTA) performance across nearly all tested scenarios. On the CMU-MOSI dataset, GSDNet achieved an average binary accuracy (Acc2) of 77.3%, outperforming previous benchmarks like IMDer (75.5%).

Crucially, the model is remarkably resilient to data loss. Even when 70% of the data was missing, GSDNet maintained an Acc2 of 77.6% on MOSI, while competing models like GCNet dropped significantly to 65.7%.

Qualitative Analysis

Qualitative t-SNE visualizations demonstrate the model's ability to match the original data distribution. As shown in the comparison below, the data points recovered by GSDNet cluster almost identically to the ground-truth data, whereas other methods produce more scattered or overlapping distributions.

t-SNE visualizations comparing the recovery quality of GSDNet against other baseline methods and ground truth.

Conclusion and Future Work

GSDNet demonstrates that the spectrum of a graph contains its essential structural DNA, making it a superior domain for diffusion-based data recovery in MERC. By preserving topology, the model achieves high contextual accuracy and robustness to extreme data loss.

Future work may address the computational complexity of eigendecomposition ($O(n^3)$), which could be a bottleneck for massive graphs. Additionally, exploring different architectures for the reconstruction modules and optimizing for real-time inference latency remain important areas for further research in live applications and robotics.

Summary updated

BibTeX
@inproceedings{shou2025gsdnet,
  author = {Yuntao Shou and Jun Yao and Tao Meng and Wei Ai and Cen Chen and Keqin Li},
  title = {{GSDNet: Revisiting Incomplete Multimodality-Diffusion Emotion Recognition from the Perspective of Graph Spectrum}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/688},
}
Figure