MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognition

Yu Pan, Yuguang Yang, Yuheng Huang, Jixun Yao, Jingjing Yin, Yanni Hu, Heng Lu, Lei Ma, Jianjun ZhaoarXiv 2023 · arXiv

TL;DR

MSAC-SERNet is a novel, unified framework for speaker-independent Speech Emotion Recognition (SER) that explicitly disentangles overlapping speech attributes (like gender, language, and speaker identity) to learn cleaner emotional representations. Furthermore, it pioneers the study of model reliability under semantic data shifts by introducing rODIN, a reverse out-of-distribution (OOD) detection method tailored specifically for audio signals.

What Problem Does This Paper Address?

Speech Emotion Recognition (SER) is notoriously difficult because a single audio clip contains a messy cocktail of overlapping acoustic information: the speaker's unique voiceprint, their gender, their language/dialect, and finally, their emotional state. Traditional deep learning models often struggle to separate these attributes, leading to poor generalization when tested on new speakers or entirely unseen datasets (cross-corpus scenarios).

Moreover, when deployed in the real world (e.g., healthcare or security), these models will inevitably encounter unfamiliar emotions or acoustic environments. Despite this, the SER field has largely ignored reliability—the ability of a model to flag out-of-distribution (OOD) data rather than confidently making a wrong prediction. This paper tackles both challenges: improving cross-corpus generalization and establishing a benchmark for reliable OOD detection in speech.

Methods

The authors introduce MSAC-SERNet, which consists of two core components: a custom CNN architecture and a novel multi-attribute learning paradigm.

  1. The SER Model Architecture: The model processes FilterBank features (FBanks) through a multi-scale shallow feature extractor using parallel convolutional layers with varying kernel sizes ($11\times1$, $3\times3$, and $9\times1$) to capture both fine-grained temporal and spatial information. This is followed by deep convolutional blocks and a statistical pooling layer that captures the mean and standard deviation of the temporal features. To maximize the distance between different emotion classes, the network is trained using Additive Margin Softmax (AM-Softmax) loss instead of standard cross-entropy.

  2. Multiple Speech Attribute Control (MSAC): Instead of treating all non-emotion attributes the same, MSAC categorizes them based on their relationship to emotion:

  3. Emotion-Agnostic Attributes (Speaker, Language): These share little acoustic overlap with emotion. The framework uses a Gradient Reversal Layer (GRL) during backpropagation to actively strip this information away, forcing the model to be speaker- and language-invariant.

  4. Emotion-Correlated Attributes (Gender): Because male and female voices naturally differ in pitch and energy—features critical to emotion—completely removing gender hurts performance. MSAC uses a multi-task learning strategy to preserve this useful context.
Overall architecture of the proposed unified SER workflow MSAC-SERNet
  1. rODIN (Reverse ODIN): Popular OOD detection methods from computer vision and NLP often apply a positive perturbation to the input to boost in-distribution confidence. The authors discovered this actually hurts audio models. They propose rODIN, which applies a reverse perturbation (similar to generating adversarial examples) to successfully separate in-distribution and out-of-distribution speech samples.

Datasets

The framework was rigorously evaluated across six public speech emotion corpora:

  • IEMOCAP: Used for single-corpus evaluation (4 classes for training/testing, 3 classes held out as OOD data).
  • EMO-DB: A German database used for single-corpus and cross-corpus evaluation.
  • EMOVO (Italian), CASIA (Chinese), and RAVDESS (English): Combined with EMO-DB and IEMOCAP to evaluate cross-corpus and cross-language performance.
  • TESS: Kept entirely unseen during training to serve as the ultimate out-of-distribution generalization test.

Key Insights and Ideas

  • Not all background attributes are enemies: A common trend in domain adaptation is to eliminate speaker and gender bias simultaneously using adversarial loss. This paper proves that treating gender as a cooperative attribute (via multi-task learning) while adversarially suppressing speaker identity yields far superior emotional representations.
  • Audio OOD detection requires flipping the script: Standard computer vision OOD techniques like ODIN fail in speech processing. While adding positive noise helps images look more "typical" to a vision model, audio models require a reverse perturbation (rODIN) to effectively widen the confidence gap between known and unknown data distributions.

Key Results

  • Single-Corpus Superiority: On the challenging IEMOCAP dataset, MSAC-SERNet achieved a state-of-the-art Weighted Average Recall (WAR) of 72.97% and Unweighted Average Recall (UAR) of 71.76%. On EMO-DB, it reached a WAR of 93.21%.
  • Cross-Corpus Generalization: When evaluated on the entirely unseen TESS dataset, the base model's accuracy jumped from 54.25% to 70.00% once the MSAC learning paradigm was applied.
  • Reliability & OOD Detection: Combining the MSAC framework with the proposed rODIN detector yielded the best reliability metrics, reducing the False Positive Rate (FPR95) on IEMOCAP down to 68.29% (a 27.4% absolute reduction compared to the baseline) and achieving an AUROC of 73.23%.

Limitations and Future Work

While MSAC-SERNet marks a major step forward for reliable speech processing, the authors note that speech contains an even broader array of attributes not yet covered by this framework, such as age, health status, and background noise profiles. Future work will focus on scaling the MSAC paradigm to control for these additional variables and designing OOD detection algorithms natively optimized for the unique mathematical properties of audio signals.

Summary updated

Abstract

Despite notable progress, speech emotion recognition (SER) remains challenging due to the intricate and ambiguous nature of speech emotion, particularly in wild world. While current studies primarily focus on recognition and generalization abilities, our research pioneers an investigation into the reliability of SER methods in the presence of semantic data shifts and explores how to exert fine-grained control over various attributes inherent in speech signals to enhance speech emotion modeling. In this paper, we first introduce MSAC-SERNet, a novel unified SER framework capable of simultaneously handling both single-corpus and cross-corpus SER. Specifically, concentrating exclusively on the speech emotion attribute, a novel CNN-based SER model is presented to extract discriminative emotional representations, guided by additive margin softmax loss. Considering information overlap between various speech attributes, we propose a novel learning paradigm based on correlations of different speech attributes, termed Multiple Speech Attribute Control (MSAC), which empowers the proposed SER model to simultaneously capture fine-grained emotion-related features while mitigating the negative impact of emotion-agnostic representations. Furthermore, we make a first attempt to examine the reliability of the MSAC-SERNet framework using out-of-distribution detection methods. Experiments on both single-corpus and cross-corpus SER scenarios indicate that MSAC-SERNet not only consistently outperforms the baseline in all aspects, but achieves superior performance compared to state-of-the-art SER approaches.
BibTeX
@misc{pan2023msac,
  author = {Yu Pan and Yuguang Yang and Yuheng Huang and Jixun Yao and Jingjing Yin and Yanni Hu and Heng Lu and Lei Ma and Jianjun Zhao},
  title = {{MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognition}},
  year = {2023},
  eprint = {2308.04025},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.04025},
}
Figure