Abstract
This paper presents a framework for adapting monolingual English wav2vec 2.0 models to diverse target languages using limited resources. By employing Dropout Uncertainty-Driven Self-Training (DUST), we demonstrate that monolingual models can achieve performance comparable to massive multilingual models like XLSR-53. Our results show an average Word Error Rate Recovery (WERR) of 86.3% across eight languages, with significant gains in out-of-domain scenarios where multilingual models often struggle.
Introduction
Building Automatic Speech Recognition (ASR) systems for low-resource languages is a persistent challenge. While self-supervised learning (SSL) has reduced the need for labeled data, models like XLSR-53 still require extensive multilingual pre-training and can suffer from domain gaps. This study investigates the efficiency of adapting a robustly pre-trained English model to new languages using a combination of limited labeled data (10 hours) and moderate unlabeled data (100 hours).
Methodology: The DUST Framework
The adaptation process follows a sequential transfer learning approach. After initial pre-training on high-resource English data, the model is fine-tuned on 10 hours of target-language audio. To further improve performance, we introduce Dropout Uncertainty-Driven Self-Training (DUST).
In the DUST phase, the fine-tuned model acts as a teacher, generating pseudo-labels for unlabeled target-language audio. To ensure label quality, we use uncertainty filtering: the model performs multiple stochastic passes with dropout enabled. Data points with high prediction variance (uncertainty) are discarded, while high-confidence samples are used to train a student model. This process is repeated iteratively to refine the model's accuracy.
Experimental Setup
We evaluated the framework using English source data from Libri-Light (LL60k) and target data from the Multilingual LibriSpeech (MLS) corpus, covering seven Indo-European languages. Additionally, we tested on the Multi-Genre Multi-Dialectal Broadcast News (MGB) Arabic dataset to assess performance in out-of-domain conditions.
Key Results
The experiments confirm that English-only wav2vec 2.0 models are excellent few-shot learners. As shown in Table 1, the robustly pre-trained model (w2v_rob) achieves an average WERR of 86.3% relative to the XLSR topline.
The application of DUST significantly bridges the remaining performance gap. For the French (MLS/fr) dataset, the WER was reduced from 38.3% to 22.3% over five iterations, nearly matching the 19.7% WER of the XLSR topline.
Furthermore, in the out-of-domain Arabic task, the adapted model reached a 21.2% WER, substantially outperforming the 37.9% WER achieved by the fine-tuned XLSR model. This highlights the robustness of domain-specific adaptation over general multilingual pre-training.
Conclusion
Our findings suggest that monolingual models, when combined with iterative self-training techniques like DUST, offer a powerful and efficient alternative to massive multilingual models. This approach is particularly valuable for adapting ASR systems to new domains and languages where large-scale unlabeled data may be scarce.
Summary updated