Abstract
Pseudo-labeling is the most adopted method for pre-training automatic speech recognition (ASR) models. However, its performance suffers from the supervised teacher model's degrading quality in low-resource setups and under domain transfer. Inspired by the successes of contrastive representation learning for computer vision and speech applications, and more recently for supervised learning of visual objects, we propose Contrastive Semi-supervised Learning (CSL). CSL eschews directly predicting teacher-generated pseudo-labels in favor of utilizing them to select positive and negative examples. In the challenging task of transcribing public social media videos, using CSL reduces the WER by 8% compared to the standard Cross-Entropy pseudo-labeling (CE-PL) when 10hr of supervised data is used to annotate 75,000hr of videos. The WER reduction jumps to 19% under the ultra low-resource condition of using 1hr labels for teacher supervision. CSL generalizes much better in out-of-domain conditions, showing up to 17% WER reduction compared to the best CE-PL pre-trained model.
Introduction
The current state-of-the-art for training Automatic Speech Recognition (ASR) models with limited labeled data relies heavily on pseudo-labeling (CE-PL). This teacher-student paradigm involves training a teacher model on a small labeled dataset to transcribe a large unlabeled corpus. The resulting "pseudo-labels" are then used to train a student model via cross-entropy loss.
However, this approach faces a significant bottleneck: the quality of the student is strictly bounded by the teacher. In low-resource scenarios or during domain transfer, teacher models produce noisy and inaccurate transcripts. Standard cross-entropy training forces the student to mimic these errors, leading to error propagation. We propose Contrastive Semi-supervised Learning (CSL) to decouple the student's learning process from the absolute accuracy of the teacher's labels.
Methodology
CSL shifts the training objective from absolute classification to relative similarity. Instead of predicting a specific phoneme or word for a given audio frame, the model learns to organize its representation space based on the pseudo-labels.
Contrastive Loss and Soft Constraints
Using the InfoNCE loss, CSL encourages the model to produce similar representations for audio segments sharing the same pseudo-label (positive pairs) while pushing apart representations with different labels (negative pairs). This "soft constraint" allows the model to learn the underlying acoustic structure of speech even when the labels themselves are imperfect.
Label-Aware Batching (LAB) and STM
To address the long-tail distribution of speech units, we introduced Label-Aware Batching (LAB). LAB ensures that rare labels are adequately represented within each training batch, providing sufficient positive pairs for the contrastive loss. Additionally, Short Time Masking (STM) is employed as a data augmentation strategy, forcing the model to rely on broader context rather than local features.
Experimental Results
We evaluated CSL using 75,000 hours of unlabeled social media video audio in British English and Italian. We simulated low-resource conditions using 10-hour and 1-hour labeled subsets for teacher training.
10-Hour Low-Resource Performance
In the 10-hour setup, CSL consistently outperformed standard CE-PL. As shown in Table 1, the optimized CSL configuration (C9) achieved a Word Error Rate (WER) of 29.4 in British English, representing an 8% relative reduction over the best CE-PL baseline (B3, 32.0 WER). Similar gains were observed in Italian, where CSL reached a WER of 16.0.
Ultra-Low-Resource and Iterative Learning
The advantages of CSL are even more pronounced in the ultra-low-resource 1-hour setting. Here, CSL reduced the WER from 53.1 (CE-PL) to 42.8, a 19% improvement. Furthermore, CSL benefits significantly from iterative training. By using a CSL-trained model to generate a second generation of labels, the WER was further reduced to 32.3. This iterative process closed approximately 80% of the performance gap between the 1-hour baseline and a 650-hour fully supervised model.
Generalization and Domain Transfer
Beyond raw WER improvements, CSL demonstrates superior generalization. In out-of-domain tests involving dictation and long-form conversations, CSL showed up to a 17% WER reduction compared to CE-PL. By focusing on normalized representations and hard negatives, CSL learns more universal speech features that are less sensitive to the specific characteristics of the training domain.
Conclusion
Contrastive Semi-supervised Learning provides a robust alternative to traditional pseudo-labeling for ASR, particularly when labeled data is scarce. While CSL introduces a modest increase in computational overhead (approximately 15% in training time) and requires larger batch sizes, its ability to learn from noisy labels and generalize across domains makes it a powerful tool for scaling ASR to thousands of languages. Future work will explore the integration of CSL with self-supervised pre-training methods like wav2vec 2.0.
Summary updated