After: Active Learning Based Fine-Tuning Framework for Speech Emotion Recognition

Dongyuan Li, Yusong Wang, Kotaro Funakoshi, Manabu Okumura

Introduction

Speech Emotion Recognition (SER) is a critical component for achieving natural human-machine interaction. While large-scale pre-trained models like wav2vec 2.0 have revolutionized Automatic Speech Recognition (ASR), applying them to SER remains challenging. ASR models are optimized to extract linguistic content, whereas SER requires the capture of subtle acoustic nuances and paralinguistic features. This discrepancy creates an "information gap" that limits the direct effectiveness of ASR-centric models for emotion tasks.

To address these challenges, researchers from the Tokyo Institute of Technology have introduced AFTER (Active Learning Based Fine-Tuning Framework). This framework optimizes the fine-tuning process by bridging the domain gap through unsupervised task adaptation and maximizing labeling efficiency via a novel active learning strategy.

The Information Gap and Data Bottleneck

Most current SER systems fine-tune ASR-pre-trained models on small, labeled emotion datasets. This approach faces three primary hurdles:

  1. Objective Mismatch: ASR models prioritize word recognition, while SER requires tone and pitch analysis.
  2. Overfitting: Fine-tuning massive models on small datasets often leads to poor generalization, especially in noisy, real-world environments.
  3. Labeling Costs: Annotating emotional speech is resource-intensive and requires expert judgment, making large-scale labeled datasets difficult to acquire.

The AFTER Framework

The AFTER framework employs a two-stage approach to overcome these limitations: Task Adaptation Pre-training (TAPT) and Active Learning (AL) with Clustering-based initialization.

Task Adaptation Pre-training (TAPT)

Before supervised fine-tuning, the model undergoes an unsupervised "warm-up" phase on the target dataset. By using the original wav2vec 2.0 self-supervised loss—predicting masked audio segments—the model adapts its internal representations to the specific acoustic characteristics of the emotion dataset without requiring labels.

Active Learning with Clustering

Instead of training on the entire dataset, AFTER iteratively selects the most informative samples for labeling. Unlike traditional active learning that starts with a random subset, AFTER uses K-means clustering to select an initial diverse set of samples that represent the entire data distribution. Following this, an Entropy-based selection strategy identifies samples where the model is most uncertain, ensuring that the most valuable data points are prioritized for annotation.

Overview of the AFTER framework, illustrating Task Adaptation Pre-training (TAPT), fine-tuning on Wav2vec 2.0, and the Active Learning loop.

Experimental Setup and Datasets

The framework was evaluated on two primary benchmarks:

  • IEMOCAP: A standard benchmark for SER research.
  • Merged Dataset: A heterogeneous collection of six datasets (IEMOCAP, EMODB, SHEMO, RAVDESS, EMov-DB, and CREMA-D) totaling over 21,000 samples. This merged set simulates real-world conditions with varying languages, noise levels, and recording environments.

Results and Discussion

The AFTER framework demonstrated superior performance compared to several state-of-the-art (SOTA) baselines, including GLAM and Pseudo-TAPT.

Performance Gains

On the Merged Dataset, AFTER achieved a significant 8.45% boost in Unweighted Accuracy (UA) over the primary baseline. The results indicate that the combination of TAPT and strategic sample selection allows the model to generalize better than models trained on the full, potentially noisy, dataset.

Comparison of AFTER against state-of-the-art methods on IEMOCAP and Merged datasets, showing significant improvements in Unweighted Accuracy (UA) and Weighted Accuracy (WA).

Efficiency and Speed

One of the most impactful findings is the framework's efficiency. AFTER achieved its peak performance using only 20% of the labeled samples. This reduction in required data translated to a 79% reduction in total fine-tuning time, making the framework highly viable for rapid deployment in real-world applications where labeling resources are limited.

Conclusion

The AFTER framework effectively bridges the gap between general speech pre-training and specialized emotion recognition. By prioritizing data diversity through clustering and focusing on high-uncertainty samples, it achieves state-of-the-art results with a fraction of the data and time required by traditional methods. Future work will focus on further improving domain adaptation to handle the transition from controlled laboratory recordings to unpredictable, noisy environments.

Summary updated

BibTeX
@inproceedings{li2023after,
  author = {Dongyuan Li and Yusong Wang and Kotaro Funakoshi and Manabu Okumura},
  title = {{After: Active Learning Based Fine-Tuning Framework for Speech Emotion Recognition}},
  booktitle = {ASRU},
  year = {2023},
  doi = {10.1109/asru57964.2023.10389652},
}
Figure