Abstract: Multimodal emotion recognition for video has gained considerable attention in recent years, in which three modalities (i.e., textual, visual and acoustic) are involved. Due to the diverse levels of informational content related to emotion, three modalities typically possess varying degrees of contribution to emotion recognition. More seriously, there might be inconsistencies between the emotion of individual modality and the video. The challenges mentioned above are caused by the inherent uncertainty of emotion. Inspired by the recent advances of quantum theory in modeling uncertainty, we make an initial attempt to design a quantum-inspired adaptive-priority-learning model (QAP) to address the challenges. Specifically, the quantum state is introduced to model modal features, which allows each modality to retain all emotional tendencies until the final classification. Additionally, we design Q-attention to orderly integrate three modalities, and then QAP learns modal priority adaptively so that modalities can provide different amounts of information based on priority. Experimental results on the IEMOCAP and MOSEI datasets show that QAP establishes new state-of-the-art results.
Introduction
Multimodal Emotion Recognition (MER) aims to identify a speaker’s emotional state by analyzing textual, visual, and acoustic data. However, current models face significant hurdles due to the inherent uncertainty of human emotion. Specifically, different modalities often contribute varying degrees of information depending on the context. Furthermore, modalities can be inconsistent; for instance, a person might use "happy" words sarcastically while their tone of voice or facial expression indicates anger. Traditional AI models often struggle with these contradictions because they treat all modalities equally or use fixed feature vectors that assume an emotion is "decided" before the fusion process begins.
Methods: The QAP Model
To address these challenges, the authors introduce QAP, a quantum-inspired adaptive-priority-learning model. Inspired by quantum theory's ability to model uncertainty, QAP represents emotions as "uncertain" density matrices rather than fixed vectors.
Quantum-Inspired Representation
Instead of standard feature vectors, QAP represents each modality as a complex-valued density matrix. This allows the model to retain all potential emotional tendencies in a "superposition" until the final classification. The model maps "end-to-end" features (from pre-trained models like ALBERT and VGG) to the amplitude of the complex values, while "hand-crafted" features (like OpenFace or sentiment dictionaries) are encoded into the phase.
Adaptive Priority Learning and Q-Attention
The model employs a novel Q-attention mechanism designed specifically for density matrices, using a "trace inner product" to calculate cross-modal attention. Crucially, QAP does not use a fixed fusion order. Through Adaptive Priority Learning, the model evaluates the input and adaptively chooses the best fusion sequence (e.g., prioritizing text then video, or text then audio) based on which modality is most informative for that specific sample. Finally, a quantum measurement operator "collapses" the fused uncertain state into a concrete probability distribution across emotion labels.
Experimental Results
The model was evaluated on two gold-standard benchmarks: IEMOCAP (7,380 utterances) and CMU-MOSEI (20,477 utterances). QAP established new state-of-the-art (SOTA) results on both datasets, outperforming previous models like MulT and MESM.
- IEMOCAP: Achieved an average accuracy of 86.8% and an F1-score of 60.8%, surpassing the previous best of 84.5% accuracy.
- CMU-MOSEI: Achieved a weighted accuracy of 68.9% and an F1-score of 49.9%.
Ablation studies confirmed that removing the quantum components (switching back to pure state vectors) led to a drop in performance, validating the effectiveness of the density matrix approach.
Discussion and Future Work
QAP demonstrates that treating modalities as uncertain states prevents noisy or contradictory signals from immediately derailing predictions. By leveraging the complex plane to combine raw neural features with expert-engineered domain knowledge, the model achieves superior fusion.
However, quantum-inspired math introduces computational overhead. Future research may explore "sparse density matrices" to reduce memory usage and training time. Additionally, while the current model tests primary fusion orders, exploring all possible permutations of the three modalities could further enhance performance.
Summary updated