Introduction
Speech Emotion Recognition (SER) models frequently encounter the challenge of speaker overfitting. Rather than identifying universal acoustic indicators of emotion, these models often learn the specific vocal idiosyncrasies of the training subjects. Consequently, performance typically degrades when the model encounters a new speaker in an out-of-distribution setting.
Adapting to a new speaker is fundamentally a few-shot learning problem. While standard fine-tuning often fails with limited data, this research explores "pre-finetuning"—an intermediate multi-task learning phase between self-supervised pre-training and final downstream adaptation—to bridge this gap and improve generalization.
Methodology
The researchers implemented a three-stage pipeline designed to maximize the model's ability to extract emotion-relevant features from minimal data:
- Pre-training: Utilizing a standard Wav2Vec 2.0 model pre-trained on large-scale unlabeled audio to establish general speech representations.
- Pre-finetuning: The model undergoes multi-task learning across four diverse emotional speech datasets. To accommodate varying label spaces, a unique classification head is assigned to each dataset, and a scaled loss function is applied to balance the learning process across tasks.
- Downstream Fine-tuning: The model is finally adapted to a specific target speaker using a very small number of examples ($k$ shots, ranging from 2 to 64).
Datasets and Experimental Setup
The study utilized four major corpora for the pre-finetuning phase: MSP-IMPROV (dyadic interactions), MSP-PODCAST (naturalistic speech), Mandarin Affective Speech (Mandarin AS), and IEMOCAP (scripted/improvised acting). Evaluation was conducted on the Emotional Speech Dataset (ESD), which includes both English and Mandarin speakers, facilitating an analysis of cross-lingual transfer.
Key Results and Insights
Massive Performance Boosts
The most significant impact of pre-finetuning was observed in extreme low-data regimes. At $k=2$ (only two examples per emotion), pre-finetuned models achieved Macro F1 scores exceeding 0.600, whereas baseline models performed near random chance. As training data increased, the performance gap narrowed, though pre-finetuned models maintained a consistent lead.
The "Critical Point" of Diversity
A key finding was the existence of a "critical point" in dataset diversity. Pre-finetuning on a single additional dataset occasionally hindered performance compared to the baseline. However, combining two or more datasets led to consistent improvements. This suggests that the model requires a diverse set of intermediate tasks to learn generalized "emotion" features rather than simply overfitting to a new intermediate dataset's biases.
Cross-Lingual Synergy
The experiments revealed that pre-finetuning on Mandarin data improved emotion recognition in English speakers and vice versa. This indicates that the model successfully learns high-level acoustic features of emotion that are invariant across different languages.
Conclusion and Future Work
Pre-finetuning effectively mitigates speaker overfitting in few-shot SER, particularly when using diverse intermediate datasets like MSP-PODCAST. While the current study focused on Wav2Vec 2.0 Base, future research could investigate the scaling laws of this approach with larger models and a wider variety of speech-related tasks, such as speaker identification or clinical speech analysis.
Summary updated