Data Augmentation with Unsupervised Speaking Style Transfer for Speech Emotion Recognition

Leyuan Qu, Wei Wang, Taihao Li, Cornelius Weber, Stefan Wermter, Fuji Ren

Abstract

Speech Emotion Recognition (SER) is frequently hindered by data scarcity and class imbalance. This paper introduces EmoAug, a novel unsupervised speaking style transfer model designed to generate diverse, high-quality emotional speech samples. By decoupling semantic content from paralinguistic style using self-supervised representations, EmoAug enables the creation of synthetic training data that preserves linguistic integrity while varying prosodic attributes. Experimental results on the IEMOCAP benchmark demonstrate that EmoAug-augmented models achieve a state-of-the-art Unweighted Accuracy (UA) of 73.75%, significantly outperforming existing supervised and self-supervised methods.

Introduction

Effective Speech Emotion Recognition (SER) is essential for natural human-computer interaction. However, the field faces a chronic shortage of high-quality, labeled emotional data. Most existing datasets are small and imbalanced, often dominated by "Neutral" expressions. Standard data augmentation techniques, such as pitch shifting or noise injection, frequently fail because they do not capture the complex nuances of human prosody—the rhythm, stress, and intonation that define emotional expression—and can even inadvertently alter the perceived emotion.

To address these challenges, we propose a system capable of "re-styling" existing utterances. The goal is to change the emotional intensity or rhythm of a sentence without requiring text transcriptions or paired source-to-target training data, thereby multiplying the available training samples for minority emotion classes.

Methodology

The EmoAug architecture is designed to separate a speech signal into two distinct, independent streams:

  1. Semantic Encoder: This component captures linguistic content without relying on text. It utilizes HuBERT representations quantized into 200 discrete cluster labels via k-means. By removing repeated labels, the model filters out tempo information, isolating the core semantic units.
  2. Paralinguistic Encoder: Based on the ECAPA-TDNN architecture, this encoder captures the "style" of the speech, including emotion and speaker identity.
  3. Attention-based Decoder: A Tacotron2-based decoder reconstructs the speech as a mel-spectrogram by integrating the semantic units with the paralinguistic style.
  4. Pitch Discriminator: A discriminator is employed during fine-tuning to ensure generated pitch contours remain realistic, which is critical for the naturalness of the synthesized audio.
EmoAug architecture overview showing the separation of semantic and paralinguistic streams.

Style transfer is achieved by combining the semantic content of one utterance with the paralinguistic style of another from the same speaker and emotion category. This process creates novel versions of emotional speech with varied prosodic attributes.

Experimental Setup

The model was pre-trained on the LRS3-TED dataset, which contains over 400 hours of diverse speaking styles. Evaluation was conducted on IEMOCAP, the standard benchmark for SER, focusing on four primary emotions: happy, sad, angry, and neutral.

Results and Discussion

EmoAug demonstrates a significant improvement in SER performance. A HuBERT Large model trained with EmoAug-augmented data achieved an Unweighted Accuracy (UA) of 73.75%, surpassing previous state-of-the-art (SOTA) methods like WavLM (70.62%) and CycleGAN-based augmentation (72.02%).

Impact of Augmentation on Feature Representation

A key insight of this work is that the diversity provided by EmoAug helps the model generalize better. As shown in t-SNE visualizations, the augmented model develops much clearer and more distinct clusters for different emotions compared to models trained only on raw data. This suggests that the style transfer effectively fills the gaps in the original data distribution.

t-SNE visualization of emotion clusters for (a) original audio and (b) four-times augmented audio.

Furthermore, ablation studies revealed a direct correlation between the number of augmentation iterations and model accuracy, confirming that the synthetic samples provide genuine linguistic and prosodic variety rather than mere noise.

Conclusion

EmoAug provides a robust framework for unsupervised data augmentation in SER. By leveraging self-supervised speech representations and a pitch-aware discriminator, it generates high-quality samples that mitigate the effects of data scarcity. Future work will focus on improving synthesized audio quality further and exploring cross-speaker style transfer to enhance dataset diversity even more broadly.

Summary updated

BibTeX
@inproceedings{qu2024data,
  author = {Leyuan Qu and Wei Wang and Taihao Li and Cornelius Weber and Stefan Wermter and Fuji Ren},
  title = {{Data Augmentation with Unsupervised Speaking Style Transfer for Speech Emotion Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446186},
}
Figure