Knowledge-Aware Bayesian Co-Attention for Multimodal Emotion Recognition

Zihan Zhao, Yu Wang, Yanfeng Wang

TL;DR

Researchers have developed a new multimodal emotion recognition model that combines text and speech by "injecting" external emotional knowledge through a Bayesian framework. By using an emotion lexicon as a probabilistic prior rather than a rigid rule, the model learns to focus on emotionally charged words and sounds more effectively, achieving a new state-of-the-art accuracy of 77.0% on the IEMOCAP dataset.

What Problem Does This Paper Address?

Multimodal Emotion Recognition (MER) aims to understand human feelings by analyzing multiple inputs—usually what someone says (text) and how they say it (audio). While "attention mechanisms" in AI are great at finding patterns, they often struggle to identify which specific parts of a long sentence or audio clip are actually relevant to emotion without guidance.

Previous attempts to help models by using "emotion lexicons" (dictionaries that rate how "angry" or "happy" specific words are) often used a "hard" injection method—essentially forcing the model to look at those words. However, because these lexicons aren't built specifically for the dataset the model is training on, this rigid approach can introduce bias and actually degrade performance. The challenge is: How do we give the model the "hint" of external knowledge without forcing it to follow that hint blindly?

Methods

The authors propose a Knowledge-aware Bayesian Co-attention model. Here is the breakdown of their pipeline:

  1. Feature Extraction: They use heavy-hitters for feature extraction: BERT for text and wav2vec 2.0 for speech.
  2. Word-level Force Alignment: Since lexicons are word-based but speech is a continuous signal, they use "force alignment" to chop the audio into segments that correspond exactly to the words in the text.
  3. Softening Knowledge: Not every emotional word is in the lexicon. To fix this "sparsity," they use a self-attention module to "smear" or soften the emotional intensity of known words across the entire sequence, helping the model understand context.
  4. The Bayesian Attention Module (BAM): This is the core innovation. Instead of treating attention weights as fixed numbers, they treat them as random variables.
  5. The Prior: They use the NRC VAD lexicon (which scores words on Valence, Arousal, and Dominance) to create a "prior" distribution (a starting guess of what's important).
  6. The Posterior: The model then looks at the actual data and calculates a "posterior" distribution.
  7. The Benefit: This Bayesian approach allows the model to incorporate the lexicon's wisdom while remaining flexible enough to ignore it if the specific data suggests the lexicon is wrong in a particular context.

Datasets

  • IEMOCAP: The gold standard for emotion recognition research. It contains videos of actors in scripted and improvised sessions. The authors focused on four categories: anger, happiness, sadness, and neutral.
  • NRC VAD Lexicon: A massive dictionary of over 20,000 English words rated for emotional qualities.

Key Insights and Ideas

  • "Hard" vs. "Soft" Injection: The paper proves that simply adding lexicon scores to an attention map (hard injection) actually decreases performance (from 74.9% to 74.6% UA). The Bayesian "soft" approach is what unlocks the performance gain.
  • The Temperature Analogy: The authors describe the emotional intensity from the lexicon as acting like "temperature." High-intensity words (like "ecstatic") increase the "heat," strengthening the connections between emotional words in the model's internal map.
  • Efficiency of Alignment: By aligning speech to word-level segments, the model achieves results comparable to frame-level analysis but with a much smaller data footprint, making the fusion process more efficient.

Key Results

The proposed model sets a new benchmark for the IEMOCAP dataset:

  • Unweighted Accuracy (UA): 77.0%, outperforming the previous state-of-the-art (76.3%) and other recent transformer-based models.
  • Weighted Accuracy (WA): 75.5%.
  • Modality Comparison: Text remains the strongest single modality (66.6% UA), but the fusion of text and speech (T+W) provides a massive jump to ~75-77%, proving that "how" we speak is a vital complement to "what" we say.

Limitations and Future Work

  • Lexicon Dependency: The model relies on the NRC VAD lexicon, which is English-centric. Adapting this to other languages would require similarly robust emotional dictionaries.
  • Alignment Quality: The method depends on "force alignment," which requires accurate transcripts. In real-world scenarios with noisy audio or poor speech-to-text, the performance might dip.
  • Emotion Complexity: The study limits itself to four broad emotion categories. Future work could explore more nuanced "secondary" emotions or complex states like sarcasm, where the conflict between text and tone is the primary signal.

Summary updated

Abstract

Multimodal emotion recognition is a challenging research area that aims to fuse different modalities to predict human emotion. However, most existing models that are based on attention mechanisms have difficulty in learning emotionally relevant parts on their own. To solve this problem, we propose to incorporate external emotion-related knowledge in the co-attention based fusion of pre-trained models. To effectively incorporate this knowledge, we enhance the co-attention model with a Bayesian attention module (BAM) where a prior distribution is estimated using the emotion-related knowledge. Experimental results on the IEMOCAP dataset show that the proposed approach can outperform several state-of-the-art approaches by at least 0.7% unweighted accuracy (UA).
BibTeX
@inproceedings{zhao2023knowledge,
  author = {Zihan Zhao and Yu Wang and Yanfeng Wang},
  title = {{Knowledge-Aware Bayesian Co-Attention for Multimodal Emotion Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=AYwc4Ip4Lx},
}
Figure