Abstract
While having numerous real-world applications, speech emotion recognition is still a technically challenging problem. How to effectively leverage the inherent multiple modalities in speech data (e.g., audio and text) is key to accurate classification. Existing studies normally choose to fuse multimodal features at the utterance level and largely neglect the dynamic interplay of features from different modalities at a fine-granular level over time. In this paper, we explicitly model dynamic interactions between audio and text at the word level via interaction units between two long short-term memory networks representing audio and text. We also devise a hierarchical representation of audio information from the frame, phoneme and word levels, which largely improves the expressiveness of resulting audio features. We finally propose WISE, a novel word-level interaction-based multimodal fusion framework for speech emotion recognition, to accommodate the aforementioned components. We evaluate WISE on the public benchmark IEMOCAP corpus and demonstrate that it outperforms state-of-the-art methods.
Introduction
Speech Emotion Recognition (SER) is a cornerstone of human-computer interaction, yet it remains difficult because human emotions are subtle and evolve over the course of an utterance. Most existing multimodal SER systems rely on utterance-level fusion, processing the entire audio signal and text transcript separately before combining them. This approach suffers from a granularity gap, ignoring the fine-grained timing of emotional cues, and is sensitive to acoustic noise in frame-level audio processing.
To address these challenges, we introduce WISE (Word-level Interaction-based multimodal fusion for Speech Emotion Recognition). WISE moves beyond simple sentence-level fusion by modeling the dynamic interplay between audio and text at the word level, combined with a hierarchical audio representation that filters noise and enhances feature expressiveness.
Methodology
The WISE framework incorporates three primary innovations to improve multimodal fusion:
Word-Level Alignment and Interaction
Instead of treating an utterance as a single block, WISE aligns audio segments to their corresponding words using timestamps. The system employs two Long Short-Term Memory (LSTM) networks—one for audio and one for text—that communicate at every word step through an interaction unit. This allows the hidden states of both modalities to influence each other as the sentence unfolds, capturing how a word's meaning might change based on its acoustic delivery.
Hierarchical Audio Representation
To mitigate the impact of silence and background noise, WISE builds a hierarchy of audio features:
- Frames: Extraction of raw MFCC features.
- Phonemes: A Gated Recurrent Unit (GRU) aggregates frame-level features into phoneme-level representations.
- Words: A second GRU aggregates phonemes into a single word-level audio vector.
Attention-Based Fusion
Recognizing that not all words carry equal emotional weight, an attention layer is applied to the fused word-level representations. This layer learns to assign higher importance to emotionally salient word-audio pairs before final classification via a SoftMax layer.
Experimental Setup
The model was evaluated on the IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset, a standard benchmark in the field. The dataset consists of 5,531 utterances from 10 actors, labeled with four emotion categories: angry, happy, neutral, and sad. Performance was measured using 5-fold cross-validation.
Results and Discussion
WISE demonstrates superior performance compared to existing state-of-the-art (SOTA) multimodal models. As shown in the comparison table below, WISE achieved a Weighted Accuracy (WA) of 75.9% and an Unweighted Accuracy (UA) of 76.4%.
Key Insights
- Phonemes as a Bridge: Moving directly from raw audio frames to words is often too noisy. The introduction of the phoneme level effectively filters out non-speech acoustic information, leading to significant accuracy gains in ablation studies.
- Mutual Influence: The interaction matrix allows modalities to "talk" to each other, mapping features into shared spaces to capture nuanced emotional expressions.
- Temporal Evolution: By modeling word-level sequences, WISE captures how emotions develop over the course of an utterance rather than relying on a static snapshot.
Conclusion
WISE provides a robust framework for speech emotion recognition by leveraging fine-grained word-level interactions and hierarchical audio features. While the current model relies on ground-truth transcripts, future work will focus on integrating Automatic Speech Recognition (ASR) systems and improving robustness to transcription errors.
Summary updated