Normalization Before Shaking Toward Learning Symmetrically Distributed Representation Without Margin in Speech Emotion Recognition

Che-Wei Huang, Shrikanth S. NarayananarXiv 2018 · arXiv

Abstract

Shake-Shake regularization stochastically blends parallel neural network branches to improve generalization. This paper investigates why this technique often clashes with Batch Normalization (BN) in Speech Emotion Recognition (SER). We demonstrate that placing a BN layer immediately before the "shaking" operation is essential for maintaining a symmetric feature space. We introduce "sub-band shaking" and a specific BN initialization strategy to significantly boost accuracy and reduce overfitting in SER.

Introduction

Deep learning models for Speech Emotion Recognition (SER) often struggle with limited data, leading to severe overfitting and poor generalization across different speakers. While regularization techniques like Shake-Shake—which randomly scales the contribution of multiple residual branches—have shown promise in image tasks, their behavior in audio and their interaction with Batch Normalization (BN) have remained unclear. This research addresses two main challenges: architectural disharmony between BN and shaking, and the persistent generalization gap in SER.

Architectural Refinement: Normalization Before Shaking

The authors propose a refined ResNeXt architecture tailored for spectrogram analysis. A key finding is that a BN layer must be placed directly before the shaking layer. This ensures that features from different classes remain dispersed and symmetrically distributed around the origin before being perturbed.

Comparison of different Batch Normalization (BN) placements. The proposed 'pre-activation + BN' (d) places BN immediately before the shaking operation.

Beyond placement, the authors introduce sub-band shaking. Instead of applying a uniform shaking strength across the entire spectral-temporal feature map, they shake different frequency sub-bands independently. This accounts for the non-uniform distribution of emotional information across frequencies and breaks the structure of potential adversarial noise.

Learning Dynamics and Initialization

Drawing a mathematical parallel between BN-residual blocks and BN-Recurrent Neural Networks (RNNs), the study suggests initializing the BN scaling parameter ($\gamma$) to a small value (e.g., 0.05) rather than the standard 1.0. This slows down initial convergence but significantly improves the model's ultimate ability to generalize to unseen speakers. This approach aligns with Vicinal Risk Minimization (VRM), where the goal is to "fill" the feature space between classes (zero margin) to reduce prediction uncertainty, rather than maximizing margins.

Experimental Results and Feature Visualization

The approach was evaluated on an aggregated dataset of 6,803 utterances from six public corpora (e.g., IEMOCAP, RAVDESS). The proposed PreActBN layout with sub-band shaking achieved an Unweighted Accuracy (UA) of 66.19%, a substantial improvement over the 61.34% baseline.

Visualizing the learned feature space. Proper BN placement (PreActBN) ensures symmetric and well-dispersed class representations compared to standard layouts.

Visualizations confirm the "Symmetry" hypothesis: without BN before shaking, class representations become "tilted" and overlap. BN acts as a stabilizer that keeps class centers well-separated and symmetric, allowing the "shaking" to explore the space between them effectively. Furthermore, the $\gamma$ initialization trick reduced the generalization gap from over 8% to 3.4%.

Conclusion

This work highlights the critical importance of architectural symmetry in regularized deep networks. By ensuring normalization precedes stochastic perturbations, models can learn more robust, margin-less representations that generalize better in data-constrained tasks like Speech Emotion Recognition. Future research will extend these principles to multi-modal fusion and more granular spectral divisions.

Summary updated

BibTeX
@misc{huang2018normalization,
  author = {Che-Wei Huang and Shrikanth S. Narayanan},
  title = {{Normalization Before Shaking Toward Learning Symmetrically Distributed Representation Without Margin in Speech Emotion Recognition}},
  year = {2018},
  eprint = {1808.00876},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.00876},
}
Figure