Abstract
In Speech Emotion Recognition (SER), emotional characteristics often appear in diverse forms of energy patterns in spectrograms. Typical attention neural network classifiers of SER are usually optimized on a fixed attention granularity. In this paper, we apply multiscale area attention in a deep convolutional neural network to attend emotional characteristics with varied granularities and therefore the classifier can benefit from an ensemble of attentions with different scales. To deal with data sparsity, we conduct data augmentation with vocal tract length perturbation (VTLP) to improve the generalization capability of the classifier. Experiments are carried out on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. We achieved 79.34% weighted accuracy (WA) and 77.54% unweighted accuracy (UA), which, to the best of our knowledge, is the state of the art on this dataset.
Introduction
Recognizing emotion in speech is a complex task because emotional "signatures" lack a uniform scale. In a spectrogram—a visual representation of audio frequencies over time—different emotions manifest as distinct energy patterns. For instance, anger might appear as a sharp, high-energy spike, whereas sadness often presents as subtle, long-term energy shifts.
Most existing Speech Emotion Recognition (SER) models rely on fixed-granularity attention mechanisms. These models observe data through a single-sized "lens," which can be problematic: a lens that is too small misses broader context, while one that is too large may blur critical fine-grained details. Additionally, SER research is frequently hampered by data sparsity, as high-quality labeled emotional speech datasets are difficult to produce.
Methodology
Multiscale Area Attention
To address the limitations of fixed-granularity attention, we propose a Multiscale Area Attention mechanism. Instead of calculating attention for individual points in the spectrogram, this method evaluates "areas" of varying dimensions (e.g., 1x2, 2x1, 2x2 blocks).
By defining the "key" of an area as its mean and the "value" as its sum, the model can attend to multiple granularities concurrently. This allows the network to dynamically prioritize the most appropriate scale for a specific emotional cue, effectively creating an ensemble of attentions.
Data Augmentation: Vocal Tract Length Perturbation (VTLP)
To combat data scarcity and improve generalization, we employ Vocal Tract Length Perturbation (VTLP). This technique warps the frequency axis of the audio to simulate variations in vocal tract lengths, effectively synthesizing new speakers from the existing training set. In our experiments, we generated seven replicas of the original training data, significantly expanding the model's exposure to acoustic variability.
Model Architecture
The underlying architecture is a 5-layer Convolutional Neural Network (CNN) designed to extract textures from both the time and frequency axes.
As shown in the architecture diagram, the Log Mel spectrogram is processed through parallel convolutional layers (Conv1A and Conv1B) before being concatenated. The subsequent layers (Conv2 through Conv5) further refine the features before they are passed to the Attention Layer (Area Attention), a Fully Connected Layer, and finally the Classification output.
Experimental Results
Dataset and Setup
We evaluated our approach on the IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset, a standard benchmark in the field. The dataset contains approximately 12 hours of emotional speech from 10 actors. Following standard protocols, we focused on four core emotions—neutral, excitement, sadness, and anger—using improvised data to ensure natural emotional expression.
Performance Benchmarks
The proposed model achieved state-of-the-art results on the IEMOCAP dataset:
- Weighted Accuracy (WA): 79.34%
- Unweighted Accuracy (UA): 77.54%
Our model outperformed previous leading methods, such as Head Fusion Net and various Multitask Learning approaches.
Key Insights
- Impact of Granularity: Experimental results indicated that a maximum area size of 3x3 or 4x4 provides the best performance. Exceeding this size likely introduces excessive noise into the attention mechanism, degrading accuracy.
- Augmentation Benefits: The addition of VTLP data augmentation provided a consistent absolute accuracy boost of approximately 0.5%. Interestingly, as the model architecture became more robust through multiscale attention, the relative gain from augmentation decreased slightly, suggesting the architecture itself handles some inherent data variability.
- Temporal Context: Visualization of the attention weights suggests that area attention effectively captures wider context along the time axis, which is crucial for identifying long-term emotional states.
Conclusion
By implementing multiscale area attention and VTLP data augmentation, we have established a new performance benchmark for Speech Emotion Recognition on the IEMOCAP dataset. Our findings highlight the importance of multi-granular analysis in capturing the diverse "shapes" of human emotion in speech. Future work will explore more efficient computational methods for area search and the application of these techniques to a broader range of nuanced emotional states.
Summary updated