Introduction
Deep learning models in Affective Computing, particularly Speech Emotion Recognition (SER), face a persistent challenge: overfitting. These models often possess millions of parameters but are trained on relatively small datasets, leading to poor generalization across new speakers or environments. While standard tools like Dropout and Batch Normalization are helpful, they are often insufficient for complex audio tasks. This paper proposes leveraging domain knowledge—specifically that emotional information is not distributed uniformly across the frequency spectrum—to create more effective regularization strategies.
Methodology: Shake-Shake Regularization
The authors build upon a 3-branch Residual Network (ResNet) architecture and integrate Shake-Shake regularization. In this setup, the outputs of parallel branches are combined using random coefficients.
As shown in the figure above, during the forward pass, a random coefficient $\alpha$ weights the branches. Crucially, during the backward pass (gradient descent), a different set of random coefficients $\beta$ is used. This process creates "model-based adversarial noise" that forces the network to learn more robust features.
Sub-band Shaking
The core novelty of this research is Sub-band Shaking. Instead of applying the same random "shake" to the entire spectral-temporal feature map, the authors propose dividing the frequency spectrum into sub-bands.
By shaking the upper and lower sub-bands independently with different random weights (the "Both" configuration), the model can treat affective cues in different frequency bands with varying levels of stochasticity. This breaks the structure of adversarial noise more effectively than standard "Full" shaking.
Experimental Setup and Results
The study was conducted using several well-known emotion corpora, including RAVDESS, eNTERFACE’05, EMOVO, and SAVEE, later expanding to include IEMOCAP and Emo-DB for deeper architecture testing. The task was a 4-class classification: joy, anger, sadness, and fear.
In the most rigorous tests, the "Both" (Sub-band) configuration achieved a 64.97% Unweighted Accuracy (UA), a statistically significant improvement over both the baseline ResNet (61.34%) and the standard "Full" Shake-Shake approach (62.99%).
Key Insights
- Generalization: Shake-Shake models maintain a much smaller gap between training and validation performance. The "Both" model provided the best balance, with a generalization gap of only 1.79% compared to the baseline's 7.48%.
- Training Patience: Regularized models require longer training times (more "patience") to reach peak performance. However, once converged, they are far more stable and less prone to error spikes on new data.
- Architectural Synergy: This work successfully bridges "multi-stream" speech processing (analyzing different feature space regions) with "multi-branch" deep learning architectures.
Conclusion
By adapting Shake-Shake regularization to account for the spectral characteristics of speech, the authors demonstrate a significant reduction in overfitting for SER tasks. Future work may explore more granular sub-band divisions or integration with temporal modeling layers like Transformers.
Summary updated