Abstract
Shake-Shake regularization stochastically blends parallel neural network branches to improve generalization. This paper investigates why this technique often clashes with Batch Normalization (BN) in Speech Emotion Recognition (SER). We demonstrate that placing a BN layer immediately before the "shaking" operation is essential for maintaining a symmetric feature space. We introduce "sub-band shaking" and a specific BN initialization strategy to significantly boost accuracy and reduce overfitting in SER.
Introduction
Deep learning models for Speech Emotion Recognition (SER) often struggle with limited data, leading to severe overfitting and poor generalization across different speakers. While regularization techniques like Shake-Shake—which randomly scales the contribution of multiple residual branches—have shown promise in image tasks, their behavior in audio and their interaction with Batch Normalization (BN) have remained unclear. This research addresses two main challenges: architectural disharmony between BN and shaking, and the persistent generalization gap in SER.
Architectural Refinement: Normalization Before Shaking
The authors propose a refined ResNeXt architecture tailored for spectrogram analysis. A key finding is that a BN layer must be placed directly before the shaking layer. This ensures that features from different classes remain dispersed and symmetrically distributed around the origin before being perturbed.
Beyond placement, the authors introduce sub-band shaking. Instead of applying a uniform shaking strength across the entire spectral-temporal feature map, they shake different frequency sub-bands independently. This accounts for the non-uniform distribution of emotional information across frequencies and breaks the structure of potential adversarial noise.
Learning Dynamics and Initialization
Drawing a mathematical parallel between BN-residual blocks and BN-Recurrent Neural Networks (RNNs), the study suggests initializing the BN scaling parameter ($\gamma$) to a small value (e.g., 0.05) rather than the standard 1.0. This slows down initial convergence but significantly improves the model's ultimate ability to generalize to unseen speakers. This approach aligns with Vicinal Risk Minimization (VRM), where the goal is to "fill" the feature space between classes (zero margin) to reduce prediction uncertainty, rather than maximizing margins.
Experimental Results and Feature Visualization
The approach was evaluated on an aggregated dataset of 6,803 utterances from six public corpora (e.g., IEMOCAP, RAVDESS). The proposed PreActBN layout with sub-band shaking achieved an Unweighted Accuracy (UA) of 66.19%, a substantial improvement over the 61.34% baseline.
Visualizations confirm the "Symmetry" hypothesis: without BN before shaking, class representations become "tilted" and overlap. BN acts as a stabilizer that keeps class centers well-separated and symmetric, allowing the "shaking" to explore the space between them effectively. Furthermore, the $\gamma$ initialization trick reduced the generalization gap from over 8% to 3.4%.
Conclusion
This work highlights the critical importance of architectural symmetry in regularized deep networks. By ensuring normalization precedes stochastic perturbations, models can learn more robust, margin-less representations that generalize better in data-constrained tasks like Speech Emotion Recognition. Future research will extend these principles to multi-modal fusion and more granular spectral divisions.
Summary updated