Magic Dust for Cross-Lingual Adaptation of Monolingual Wav2vec-2.0

Sameer Khurana, Antoine Laurent, James R. Glass

Abstract

This paper presents a framework for adapting monolingual English wav2vec 2.0 models to diverse target languages using limited resources. By employing Dropout Uncertainty-Driven Self-Training (DUST), we demonstrate that monolingual models can achieve performance comparable to massive multilingual models like XLSR-53. Our results show an average Word Error Rate Recovery (WERR) of 86.3% across eight languages, with significant gains in out-of-domain scenarios where multilingual models often struggle.

Introduction

Building Automatic Speech Recognition (ASR) systems for low-resource languages is a persistent challenge. While self-supervised learning (SSL) has reduced the need for labeled data, models like XLSR-53 still require extensive multilingual pre-training and can suffer from domain gaps. This study investigates the efficiency of adapting a robustly pre-trained English model to new languages using a combination of limited labeled data (10 hours) and moderate unlabeled data (100 hours).

Methodology: The DUST Framework

The adaptation process follows a sequential transfer learning approach. After initial pre-training on high-resource English data, the model is fine-tuned on 10 hours of target-language audio. To further improve performance, we introduce Dropout Uncertainty-Driven Self-Training (DUST).

In the DUST phase, the fine-tuned model acts as a teacher, generating pseudo-labels for unlabeled target-language audio. To ensure label quality, we use uncertainty filtering: the model performs multiple stochastic passes with dropout enabled. Data points with high prediction variance (uncertainty) are discarded, while high-confidence samples are used to train a student model. This process is repeated iteratively to refine the model's accuracy.

Experimental Setup

We evaluated the framework using English source data from Libri-Light (LL60k) and target data from the Multilingual LibriSpeech (MLS) corpus, covering seven Indo-European languages. Additionally, we tested on the Multi-Genre Multi-Dialectal Broadcast News (MGB) Arabic dataset to assess performance in out-of-domain conditions.

Key Results

The experiments confirm that English-only wav2vec 2.0 models are excellent few-shot learners. As shown in Table 1, the robustly pre-trained model (w2v_rob) achieves an average WERR of 86.3% relative to the XLSR topline.

Table 1: WER/CER performance across target languages for various pre-trained models.

The application of DUST significantly bridges the remaining performance gap. For the French (MLS/fr) dataset, the WER was reduced from 38.3% to 22.3% over five iterations, nearly matching the 19.7% WER of the XLSR topline.

Table 2: Iterative improvement of WER and WERR using DUST on the French (MLS/fr) dataset.

Furthermore, in the out-of-domain Arabic task, the adapted model reached a 21.2% WER, substantially outperforming the 37.9% WER achieved by the fine-tuned XLSR model. This highlights the robustness of domain-specific adaptation over general multilingual pre-training.

Conclusion

Our findings suggest that monolingual models, when combined with iterative self-training techniques like DUST, offer a powerful and efficient alternative to massive multilingual models. This approach is particularly valuable for adapting ASR systems to new domains and languages where large-scale unlabeled data may be scarce.

Summary updated

Abstract

We propose a simple and effective cross-lingual transfer learning method to adapt monolingual wav2vec-2.0 models for Automatic Speech Recognition (ASR) in resource-scarce languages. We show that a monolingual wav2vec-2.0 is a good few-shot ASR learner in several languages. We improve its performance further via several iterations of Dropout Uncertainty-Driven Self-Training (DUST) by using a moderate-sized unlabeled speech dataset in the target language. A key finding of this work is that the adapted monolingual wav2vec-2.0 achieves similar performance as the topline multilingual XLSR model, which is trained on fifty-three languages, on the target language ASR task.
BibTeX
@inproceedings{khurana2021magic,
  author = {Sameer Khurana and Antoine Laurent and James R. Glass},
  title = {{Magic Dust for Cross-Lingual Adaptation of Monolingual Wav2vec-2.0}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp43922.2022.9746276},
}
Figure