Contrastive Semi-Supervised Learning for ASR

Alex Xiao, Christian Fuegen, Abdelrahman Mohamed

TL;DR

Researchers at Facebook AI have developed Contrastive Semi-supervised Learning (CSL), a new pre-training method for Automatic Speech Recognition (ASR) that makes models significantly more robust when labeled data is scarce. By using a contrastive loss instead of traditional cross-entropy to process pseudo-labels, CSL reduces Word Error Rate (WER) by up to 19% in ultra-low-resource settings and generalizes much better to unfamiliar domains.

What Problem Does This Paper Address?

The gold standard for training ASR models with limited data is pseudo-labeling. In this "teacher-student" setup, a small amount of labeled data is used to train a "teacher" model, which then transcribes a massive amount of unlabeled audio. The "student" model is then trained to predict these teacher-generated transcripts (pseudo-labels).

However, there is a major bottleneck: if the teacher model is weak—which happens in low-resource languages or when moving to a new domain—it produces noisy, incorrect labels. The student model then ends up "copying" the teacher's mistakes, leading to poor performance. The authors wanted to find a way to use these imperfect pseudo-labels without being strictly bound by their errors.

Methods

The core innovation is Contrastive Semi-supervised Learning (CSL). Instead of forcing the student model to predict the exact label (e.g., "this frame is the phoneme /k/"), CSL uses the labels to organize the representation space:

  1. Contrastive Loss (InfoNCE): The model is trained to move representations of audio segments with the same pseudo-label closer together (positive pairs) and push segments with different labels further apart (negative pairs).
  2. Soft Constraints: Unlike standard Cross-Entropy (CE), which is a "hard" constraint, the contrastive loss acts as a "soft" constraint. It focuses on the relative similarity between sounds rather than absolute classification, making it more resilient to the teacher's labeling noise.
  3. Label-Aware Batching (LAB): Contrastive learning requires diverse batches. To ensure that rare speech sounds have "positive pairs" to learn from within a single training batch, the authors developed LAB. This technique samples rare labels more frequently when building mini-batches to ensure the model sees enough examples of every sound.
  4. Short Time Masking (STM): An augmentation strategy that masks segments of the input to force the model to learn more robust contextual features.

Datasets

The researchers used a massive dataset of 75,000 hours of unlabeled public social media videos (Facebook videos) in British English and Italian.

To test the "low-resource" effectiveness, they limited the supervised teacher data to:

  • 10 hours (Low-resource)
  • 1 hour (Ultra-low-resource)

Evaluation was conducted on 14 hours of human-labeled video. They also tested out-of-domain performance using datasets for short message dictation and long-form conversations to see how well the model handled audio styles it hadn't seen during supervised training.

Key Insights and Ideas

  • Resilience through Relativity: The "aha moment" is that we don't need pseudo-labels to be 100% accurate to learn from them. By treating them as a guide for similarity rather than ground truth, the model learns the underlying structure of speech without over-fitting to the teacher's specific errors.
  • Better Generalization: CSL-trained models performed significantly better on out-of-domain data. The authors argue that the contrastive loss over normalized representations emphasizes "hard" examples, which helps the model learn features that are more universal and less specific to the training domain.
  • Complementary to Iteration: The paper shows that CSL can be used in multiple "generations" (iterative labeling). As the teacher gets better, CSL continues to provide gains over standard methods.

Key Results

  • 10-Hour Setup: CSL reduced the Word Error Rate (WER) by 8% compared to standard cross-entropy pseudo-labeling.
  • 1-Hour Setup: In the ultra-low-resource scenario, the improvement jumped to 19%.
  • Out-of-Domain: CSL showed up to a 17% WER reduction on datasets like long-form conversations, proving it is much more robust to "domain shift" than traditional methods.
  • Closing the Gap: Using just 1 hour of labels and two iterations of CSL, the researchers closed about 80% of the performance gap between a 1-hour model and a 650-hour fully supervised model.

Limitations and Future Work

  • Computational Cost: The authors note that the CSL loss introduces a 15% increase in training time compared to standard pseudo-labeling due to the complexity of pairwise similarity computations.
  • Batch Size Dependency: Like most contrastive methods, CSL relies on large mini-batches to provide enough negative samples, which requires significant GPU memory.
  • Future Directions: The paper suggests further optimizing the selection of positive and negative examples (perhaps by reducing the number of samples for very frequent speech units) to speed up training. There is also potential to combine CSL with fully self-supervised methods (like wav2vec 2.0) for even better results.

Summary updated

Abstract

Pseudo-labeling is the most adopted method for pre-training automatic speech recognition (ASR) models. However, its performance suffers with degrading quality of the supervised teacher model.Inspired by the successes of contrastive representation learning for both computer vision and speech applications, and more recently for supervised learning of visual objects [1], we propose Contrastive Semi-supervised Learning (CSL). CSL eschews directly predicting teacher generated pseudo-labels in favor of utilizing them to select positive and negative examples.In the challenging task of transcribing public social media videos, using CSL reduces the WER by 8%, compared to the standard Cross-Entropy pseudo-labeling (CE-PL), when 10hr of supervised data is used to annotate 75,000hr of videos. The WER reduction jumps to 19% under the ultra low-resource condition of using 1hr labels for teacher supervision. In out-of-domain conditions, CSL generalizes much better showing up to 17% WER reduction compared to the strongest CE-PL pre-trained model.
BibTeX
@inproceedings{xiao2021contrastive,
  author = {Alex Xiao and Christian Fuegen and Abdelrahman Mohamed},
  title = {{Contrastive Semi-Supervised Learning for ASR}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=yrEGtYaVOo},
}

Loading related content...

Figure