Abstract
Multimodal Emotion Recognition in Conversations (MERC) often faces the challenge of incomplete data due to sensor failure or environmental noise. While existing methods use Graph Neural Networks (GNNs) and Diffusion Models (DMs) to recover missing information, adding noise directly to the graph's spatial structure often leads to "graph collapse." This paper introduces GSDNet (Graph Spectral Diffusion Network), which shifts the diffusion process to the spectral domain. By diffusing the eigenvalues of the conversation graph rather than its adjacency matrix, GSDNet preserves the underlying topological structure while effectively reconstructing missing features. Experimental results on CMU-MOSI and CMU-MOSEI demonstrate that GSDNet achieves state-of-the-art performance and remarkable robustness, even with up to 70% data loss.
Introduction
In an ideal setting, multimodal emotion recognition models leverage perfectly synchronized text, audio, and video. However, real-world data is frequently "incomplete"—microphones fail, speakers move out of frame, or background noise corrupts transcripts. Current state-of-the-art methods attempt to solve this using GNNs to model speaker relationships and DMs to "hallucinate" missing data.
Combining these approaches is difficult because standard graph diffusion models add noise directly to the adjacency matrix. This often destroys the local structure and connectivity before the model can learn to repair it, leading to a rapid drop in the signal-to-noise ratio (SNR). GSDNet addresses this by revisiting graph diffusion from a spectral perspective, ensuring that the global topological information remains intact throughout the recovery process.
Methodology
The core innovation of GSDNet is the shift from spatial to spectral diffusion. The framework consists of three primary stages:
- Node Construction: Utterances from different modalities are aligned into a common feature space using 1D convolutional layers and positional embeddings.
- Graph Spectral Diffusion: Instead of adding noise to the connections between utterances, the model performs an eigendecomposition of the adjacency matrix. It then applies a Score-based Generative Model (SGM) to the eigenvalues and the node features.
- Forward Process: Gradually adds Gaussian noise to the eigenvalues and features.
- Reverse Process: A scoring network learns to "denoise" these spectral components, reconstructing the missing modality's distribution based on the remaining observed modalities.
- Fusion and Classification: Once the missing data is recovered, a standard Graph Convolutional Network (GCN) fuses the modalities to predict the final emotion label.
By strictly limiting noise to the spectral space (eigenvalues) and not the eigenvectors (the "shape" of the connections), the global topological information is preserved. This maintains a higher SNR during training, allowing the scoring network to capture more nuanced gradients of the original data distribution.
Experimental Results
GSDNet was evaluated on two gold-standard benchmarks: CMU-MOSI and CMU-MOSEI. The researchers utilized RoBERTa-Large for text, DenseNet for video, and wav2vec for acoustic features.
Performance and Robustness
GSDNet achieved state-of-the-art (SOTA) performance across nearly all tested scenarios. On the CMU-MOSI dataset, GSDNet achieved an average binary accuracy (Acc2) of 77.3%, outperforming previous benchmarks like IMDer (75.5%).
Crucially, the model is remarkably resilient to data loss. Even when 70% of the data was missing, GSDNet maintained an Acc2 of 77.6% on MOSI, while competing models like GCNet dropped significantly to 65.7%.
Qualitative Analysis
Qualitative t-SNE visualizations demonstrate the model's ability to match the original data distribution. As shown in the comparison below, the data points recovered by GSDNet cluster almost identically to the ground-truth data, whereas other methods produce more scattered or overlapping distributions.
Conclusion and Future Work
GSDNet demonstrates that the spectrum of a graph contains its essential structural DNA, making it a superior domain for diffusion-based data recovery in MERC. By preserving topology, the model achieves high contextual accuracy and robustness to extreme data loss.
Future work may address the computational complexity of eigendecomposition ($O(n^3)$), which could be a bottleneck for massive graphs. Additionally, exploring different architectures for the reconstruction modules and optimizing for real-time inference latency remain important areas for further research in live applications and robotics.
Summary updated