Abstract
This study investigates the effectiveness of finetuning Transformer-based acoustic models for gender-specific speaker adaptation. Using a baseline model pretrained on the LibriSpeech-960 dataset, we conducted experiments with gender-specific finetuning and domain adaptation on accented speech (L2 Arctic). While gender-specific adaptation yielded modest Word Error Rate (WER) reductions—approximately 5% for male and 3% for female subsets—the most significant improvements came from general domain adaptation to accented speech. We also found that incorporating x-vector voice embeddings provided no significant accuracy gains.
Introduction
Automatic Speech Recognition (ASR) systems must handle the high variability of human speech, influenced by factors such as age, accent, and gender. Despite being trained on massive datasets, general-purpose models often exhibit performance disparities across demographic groups. This research explores "gender domain adaptation"—specializing a high-performing general model for either male or female voices—to determine if targeted finetuning can outperform a universal model.
Methodology
The researchers utilized a Transformer-based acoustic model with a Connectionist Temporal Classification (CTC) loss architecture, implemented via the ESPnet framework. The experimental pipeline involved:
- Pre-training: A baseline model was trained on the full 960-hour LibriSpeech corpus.
- Gender-Specific Finetuning: Pre-trained checkpoints were further trained on gender-segregated subsets of LibriSpeech (approx. 496h male, 465h female).
- Optimization: Adadelta was found to be more effective than the standard Noam optimizer for the adaptation phase.
- Speaker Embeddings: The team tested the integration of 512-dimensional x-vectors to provide explicit speaker identity information.
Results and Discussion
The Impact of Gender Adaptation
Finetuning the general model on gender-specific subsets resulted in modest relative reductions in WER. For female speakers, the adaptation improved performance across several test sets, as shown in the table below.
However, models trained exclusively on gender-specific subsets from scratch performed significantly worse than the 960-hour baseline. This confirms that the volume of data in the general model provides a crucial foundation of phonetic and linguistic knowledge that smaller, specific datasets cannot replicate on their own.
Domain Adaptation: The L2 Arctic Case
The most dramatic performance gains were observed when adapting the general LibriSpeech model to the L2 Arctic dataset, which consists of non-native accented speech. Accented speech is traditionally challenging for ASR, and the baseline model struggled significantly.
Finetuning on the full L2 Arctic dataset reduced the WER from 29.5% to 19.1%. Interestingly, further gender-specific adaptation on top of this domain-adapted model provided only marginal additional gains (0.1% to 2%).
Speaker Embeddings and Overfitting
The inclusion of x-vectors did not yield significant improvements. The researchers hypothesize that the Transformer's self-attention mechanism already captures sufficient speaker-relevant acoustic features. Furthermore, the high level of optimization in the baseline model made it difficult to "bias" the weights toward a specific gender without risking overfitting or losing general performance.
Conclusion
While gender-specific finetuning offers incremental improvements, the research highlights that data volume remains a dominant factor in ASR performance. Domain adaptation (e.g., for accents) provides much larger gains than demographic specialization. Future work may explore more granular acoustic features, such as pitch, or meta-learning techniques to facilitate more efficient speaker adaptation.
Summary updated