Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition

Pengfei Liu, Kun Li, Helen Meng

Introduction

Recognizing human emotion is a complex task because emotions are expressed through multiple channels: linguistic content (what we say) and prosody/acoustics (how we say it). While multimodal models exist, they often struggle with temporal alignment—linking specific speech frames to corresponding words—and expressive fusion of these disparate signals. Many models rely on crude fusion methods like simple concatenation, which fail to capture the nuanced interplay between modalities.

The Gated Bidirectional Alignment Network (GBAN) addresses these challenges by explicitly aligning speech frames with text tokens using bidirectional attention and employing a novel Group Gated Fusion (GGF) layer to dynamically weight modality contributions.

The GBAN Architecture

The GBAN architecture consists of three primary stages designed to extract, align, and fuse multimodal features effectively.

The GBAN architecture featuring CNN-LSTM encoders, a Bidirectional Alignment Network, and Group Gated Fusion.
  1. CNN-LSTM Encoders: Separate pipelines process speech and text. CNN layers extract local features (e.g., phonemes or word shapes), while bidirectional LSTMs capture the global temporal context of the entire utterance.
  2. Bidirectional Alignment Network: This core component performs mutual attention. It generates a speech representation weighted to match the text sequence (Speech-to-Text) and a text representation weighted to match the acoustic sequence (Text-to-Speech). This ensures the model captures complementary information from both directions.
  3. Group Gated Fusion (GGF): Instead of simple concatenation, the GGF layer groups "aligned" representations and "raw" LSTM hidden states separately. Sigmoid-based gates learn to dynamically weight the contribution of each group and modality. For instance, if the text is more emotionally salient than the audio for a specific sentence, the gate prioritizes text features.

Experimental Setup

The model was evaluated on the IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset, which contains approximately 12 hours of audiovisual dialogue. The study focused on four emotional categories: happy, angry, sad, and neutral. Evaluation followed a 5-fold cross-validation setup in a speaker-independent manner, training on four sessions and testing on the fifth.

Results and Key Insights

The GBAN model demonstrated superior performance over several strong baselines and previous state-of-the-art (SOTA) models.

Comparison of GBAN against baseline and state-of-the-art models on the IEMOCAP dataset.

Performance Metrics

As shown in the results table, GBAN achieved a 72.39% Weighted Accuracy (WA) and 70.08% Unweighted Accuracy (UA), outperforming the previous SOTA multimodal approach (BiAtt-GMU). Ablation studies confirmed that attention-aligned representations ($a_s, a_t$) significantly outperform standard last-hidden-state representations ($h_s, h_t$), validating the importance of explicit alignment.

Key Findings

  • Directionality: "Speech-aligned text" (using audio to weight words) proved more effective than "text-aligned speech." This is likely because speech sequences are significantly longer than text sequences, making it easier for the model to attend to key words than to find specific acoustic cues in a long audio stream.
  • Text Dominance: Weight analysis revealed that the text modality consistently received higher weights than speech. This suggests that for the IEMOCAP dataset, linguistic cues are often more discriminative than acoustic ones.
  • Interpretability: The GGF layer provides a "window" into the model's logic. By examining gate values, researchers can determine which modality the model prioritized for a specific prediction, enhancing the transparency of the emotion recognition process.

Conclusion

GBAN advances multimodal emotion recognition by combining bidirectional alignment with a sophisticated gating mechanism. While the model achieves SOTA results, future work could explore its performance on larger datasets and its robustness when using Automated Speech Recognition (ASR) instead of human transcripts. Additionally, integrating visual data, such as facial expressions, into the bidirectional alignment framework remains a promising direction for further research.

Summary updated

BibTeX
@inproceedings{liu2020group,
  author = {Pengfei Liu and Kun Li and Helen Meng},
  title = {{Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition}},
  booktitle = {Interspeech},
  year = {2020},
  doi = {10.21437/interspeech.2020-2067},
}
Figure