Abstract
Recognizing human emotion from speech is a complex task due to its inherent subjectivity and the multi-modal nature of human expression. This paper presents a novel two-stage "Modality Attention Flow" (MAF) framework designed to effectively fuse acoustic and textual data. By simultaneously modeling intra-modality expressions and inter-modality alignments through stacked attention blocks, the proposed system achieves state-of-the-art performance on major benchmarks, demonstrating the power of deep modality interaction modeling.
Introduction
Speech Emotion Recognition (SER) is a critical component of human-computer interaction. However, effectively integrating multiple modalities—such as vocal tone (acoustic) and spoken words (textual)—remains a challenge. Traditional fusion strategies often fall short: late fusion captures intra-modality relationships well but misses subtle cross-modal alignments, while early fusion often fails to account for the distinct structural differences between data types. This work addresses these limitations by introducing a unified architecture that models both intra- and inter-modality interactions simultaneously.
Methodology
The core of the proposed system is the Modality Attention Flow (MAF) framework, which processes inputs through a multi-stage pipeline.
Architecture Components
- Bidirectional Recurrent Encoders: The system utilizes Bidirectional Long Short-Term Memory (BLSTM) networks to encode 128-dimensional log-mel spectrograms for audio and 300-dimensional GloVe word embeddings for text.
- Elementary Attention Flow (EAF): In the first stage, the model calculates basic attention masks, giving equal priority to "word-to-word" (intra) and "frame-to-word" (inter) relations.
- Advanced Attention Flow (AAF): The second stage models higher-order interactions. A key innovation here is the use of acoustic features to "condition" textual attention, allowing the model to weight the importance of specific word pairs based on vocal cues.
- Stacking: The MAF blocks are designed with consistent input and output shapes, allowing them to be stacked (e.g., MAF2, MAF4) to capture increasingly complex interactions.
Experimental Setup
The framework was evaluated on two distinct benchmarks:
- IEMOCAP: A standard English dataset (~12 hours) featuring four emotion labels (happy/excited, sad, angry, neutral).
- MERC2020: A large-scale Korean dataset (~78 hours) with seven balanced emotional labels, providing a rigorous test for scalability and robustness.
Results and Discussion
The MAF framework demonstrated significant performance gains over baseline models and existing state-of-the-art methods.
Key Findings
- Modality Synergy: Bimodal models consistently outperformed unimodal versions, confirming that textual cues significantly complement acoustic information in SER.
- Depth vs. Data Trade-off: As shown in the results, increasing the number of MAF blocks (from MAF2 to MAF4) improved accuracy on the large MERC2020 dataset, reaching 53.26%. This notably outperformed the official competition baseline of 52.6%, which utilized an additional visual modality. However, on the smaller IEMOCAP dataset, the MAF2 configuration performed best (72.86% UAR), while MAF4 showed signs of overfitting.
- Effective Fusion: The AAF stage's ability to use speech to condition textual attention mimics human perception, such as interpreting sarcasm by aligning tone with word choice.
Conclusion
The two-stage MAF framework provides a robust method for fusing acoustic and textual modalities in SER. While the system achieves state-of-the-art results, the researchers identified that model complexity must be balanced against dataset size to avoid overfitting. Future work will explore incorporating visual modalities (facial expressions) and more sophisticated encoder structures like Transformers to further enhance performance and generalizability across diverse datasets.
Summary updated