Gender domain adaptation for automatic speech recognition task

Artem Sokolov, Andrey V. SavchenkoarXiv 2020 · OpenReview

Abstract

This study investigates the effectiveness of finetuning Transformer-based acoustic models for gender-specific speaker adaptation. Using a baseline model pretrained on the LibriSpeech-960 dataset, we conducted experiments with gender-specific finetuning and domain adaptation on accented speech (L2 Arctic). While gender-specific adaptation yielded modest Word Error Rate (WER) reductions—approximately 5% for male and 3% for female subsets—the most significant improvements came from general domain adaptation to accented speech. We also found that incorporating x-vector voice embeddings provided no significant accuracy gains.

Introduction

Automatic Speech Recognition (ASR) systems must handle the high variability of human speech, influenced by factors such as age, accent, and gender. Despite being trained on massive datasets, general-purpose models often exhibit performance disparities across demographic groups. This research explores "gender domain adaptation"—specializing a high-performing general model for either male or female voices—to determine if targeted finetuning can outperform a universal model.

Methodology

The researchers utilized a Transformer-based acoustic model with a Connectionist Temporal Classification (CTC) loss architecture, implemented via the ESPnet framework. The experimental pipeline involved:

  1. Pre-training: A baseline model was trained on the full 960-hour LibriSpeech corpus.
  2. Gender-Specific Finetuning: Pre-trained checkpoints were further trained on gender-segregated subsets of LibriSpeech (approx. 496h male, 465h female).
  3. Optimization: Adadelta was found to be more effective than the standard Noam optimizer for the adaptation phase.
  4. Speaker Embeddings: The team tested the integration of 512-dimensional x-vectors to provide explicit speaker identity information.

Results and Discussion

The Impact of Gender Adaptation

Finetuning the general model on gender-specific subsets resulted in modest relative reductions in WER. For female speakers, the adaptation improved performance across several test sets, as shown in the table below.

WER comparison for female-adapted model vs baseline.

However, models trained exclusively on gender-specific subsets from scratch performed significantly worse than the 960-hour baseline. This confirms that the volume of data in the general model provides a crucial foundation of phonetic and linguistic knowledge that smaller, specific datasets cannot replicate on their own.

Domain Adaptation: The L2 Arctic Case

The most dramatic performance gains were observed when adapting the general LibriSpeech model to the L2 Arctic dataset, which consists of non-native accented speech. Accented speech is traditionally challenging for ASR, and the baseline model struggled significantly.

WER reduction after finetuning on the L2 Arctic dataset.

Finetuning on the full L2 Arctic dataset reduced the WER from 29.5% to 19.1%. Interestingly, further gender-specific adaptation on top of this domain-adapted model provided only marginal additional gains (0.1% to 2%).

Speaker Embeddings and Overfitting

The inclusion of x-vectors did not yield significant improvements. The researchers hypothesize that the Transformer's self-attention mechanism already captures sufficient speaker-relevant acoustic features. Furthermore, the high level of optimization in the baseline model made it difficult to "bias" the weights toward a specific gender without risking overfitting or losing general performance.

Conclusion

While gender-specific finetuning offers incremental improvements, the research highlights that data volume remains a dominant factor in ASR performance. Domain adaptation (e.g., for accents) provides much larger gains than demographic specialization. Future work may explore more granular acoustic features, such as pitch, or meta-learning techniques to facilitate more efficient speaker adaptation.

Summary updated

Abstract

This paper is focused on the finetuning of acoustic models for speaker adaptation goals on a given gender. We pretrained the Transformer baseline model on Librispeech-960 and conduct experiments with finetuning on the gender-specific test subsets and. In general, we do not obtain essential WER reduction by finetuning techniques by this approach. We achieved up to ~5% lower word error rate on the male subset and 3% on the female subset if the layers in the encoder and decoder are not frozen, but the tuning is started from the last checkpoints. Moreover, we adapted our base model on the full L2 Arctic dataset of accented speech and fine-tuned it for particular speakers and male and female genders separately. The models trained on the gender subsets obtained 1-2% higher accuracy when compared to the model tuned on the whole L2 Arctic dataset. Finally, we tested the concatenation of the pretrained x-vector voice embeddings and embeddings from a conventional encoder, but its gain in accuracy is not significant.
BibTeX
@misc{sokolov2020gender,
  author = {Artem Sokolov and Andrey V. Savchenko},
  title = {{Gender domain adaptation for automatic speech recognition task}},
  year = {2020},
  eprint = {2010.04224},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.04224},
}
Figure