Continuous pseudo-labeling from the start

Dan Berrebbi, Ronan Collobert, Samy Bengio, Navdeep Jaitly, Tatiana Likhomanenko

Introduction

In the domain of Automatic Speech Recognition (ASR), self-training (or pseudo-labeling) has emerged as a powerful technique to leverage massive amounts of unlabeled data. Traditionally, this involves taking a small amount of labeled audio, training a "teacher" model, and using it to generate "pseudo-labels" (PLs) for a large unlabeled dataset. These PLs are then used to train a more robust "student" model.

However, self-training faces a significant hurdle known as the "Cold Start" problem. If self-training begins too early, the model generates poor-quality labels, learns from its own mistakes, and quickly collapses into a degenerate state—often outputting empty strings. To circumvent this, current state-of-the-art continuous training methods (like slimIPL) rely on a supervised pre-training "warm-up" phase. The authors of this paper argue that in low-resource settings (e.g., only 10 hours of labeled data), this initial phase causes the model to overfit to the tiny labeled set, ultimately degrading its final generalization performance. Researchers from Apple and CMU propose a novel approach to stabilize continuous pseudo-labeling from the very first step of training, entirely skipping the traditional supervised warm-up.

Methods: Stabilizing Early Pseudo-Labeling

To prevent model collapse without relying on supervised pre-training, the authors introduce two key techniques:

1. Dynamic Cache Curriculum

In continuous pseudo-labeling frameworks, a cache of pseudo-labels is maintained and periodically updated. Instead of using a fixed probability to update this cache, the authors introduce a dynamic curriculum based on the Levenshtein distance (edit distance) between the old pseudo-label and the model's current prediction.

Comparison of cache update mechanisms between standard slimIPL and the proposed dynamic curriculum.

If a label is changing rapidly between training steps, it indicates the model is still actively learning about that specific sample. By monitoring this "evolution," the system dynamically adjusts the probability of updating the cache ($1 - p_{out}$). This keeps the training stable and prevents the model from getting stuck on early, incorrect guesses.

2. Alignment Sampling with Temperature

Standard pseudo-labeling typically uses the "best" transcription (the argmax of the output distribution). Early in training, however, the model is often overconfident about incorrect answers. To counteract this, the authors sample transcriptions from the model's predictive distribution rather than taking the absolute best result.

They employ a temperature schedule ($\tau$): starting with high randomness (high temperature) to inject noise and prevent collapse, and gradually cooling down to a near-deterministic state ($\tau = 0.1$) as the model becomes more accurate. This sampling acts as a crucial regularizer that keeps the model from self-reinforcing its early errors.

Key Insights

  • More Pre-training Isn't Always Better: One of the most surprising findings was that increasing the supervised pre-training phase from 10k to 40k steps actually increased the Word Error Rate (WER) from 14.3% to 22.9%. The model overfits to the small labeled set so severely that it loses the ability to generalize to the unlabeled data, validating the need to start self-training immediately.
  • Label Evolution as a Signal: The rate at which a pseudo-label changes over time serves as an excellent proxy for how "useful" that sample is.
Correlation between the evolution of pseudo-labels and their actual error against ground truth.

As shown above, there is a strong correlation between the change in pseudo-labels (WER between step $k$ and $k'$) and the actual error of those labels compared to the ground truth. This justifies using label evolution to drive the dynamic cache curriculum.

Datasets and Results

The researchers evaluated their approach across several benchmarks, including LibriSpeech (10-hour and 100-hour subsets), Libri-Light, and Common Voice (French) to test cross-lingual applicability.

  • Matching State-of-the-Art: On the LibriSpeech 10h benchmark, their "from the start" method matched or outperformed models like wav2vec 2.0 and slimIPL, which rely on complex multi-stage pre-training pipelines.
  • Low-Resource Wins: On the Common Voice French 10h task, the proposed method achieved a 26.0% WER, marking a significant 18% relative reduction over the slimIPL baseline (31.1% WER).
  • Simplicity: Notably, these results were achieved without the need for an external Language Model (LM), which is frequently used as a crutch to compensate for poor acoustic model performance in early training stages.

Limitations and Future Work

While highly effective, the current work focuses on models trained specifically with the Connectionist Temporal Classification (CTC) loss. Although the authors hypothesize that these techniques should transfer to Transducer or Sequence-to-Sequence models, this remains to be empirically proven. Additionally, removing the pre-training phase introduces new hyperparameters, such as the temperature schedule and cache update functions, which may require careful tuning for novel datasets. Future directions include exploring how these dynamic cache strategies could be adapted to handle out-of-domain unlabeled data.

Summary updated

Abstract

Self-training (ST), or pseudo-labeling has sparked significant interest in the automatic speech recognition (ASR) community recently because of its success in harnessing unlabeled data. Unlike prior semi-supervised learning approaches that relied on iteratively regenerating pseudo-labels (PLs) from a trained model and using them to train a new model, recent state-of-the-art methods perform `continuous training' where PLs are generated using a very recent version of the model being trained. Nevertheless, these approaches still rely on bootstrapping the ST using an initial supervised learning phase where the model is trained on labeled data alone. We believe this has the potential for over-fitting to the labeled dataset in low resource settings and that ST from the start of training should reduce over-fitting. In this paper we show how we can do this by dynamically controlling the evolution of PLs during the training process in ASR. To the best of our knowledge, this is the first study that shows the feasibility of generating PLs from the very start of the training. We are able to achieve this using two techniques that avoid instabilities which lead to degenerate models that do not generalize. Firstly, we control the evolution of PLs through a curriculum that uses the online changes in PLs to control the membership of the cache of PLs and improve generalization. Secondly, we find that by sampling transcriptions from the predictive distribution, rather than only using the best transcription, we can stabilize training further. With these techniques, our ST models match prior works without an external language model.
BibTeX
@inproceedings{berrebbi2023continuous,
  author = {Dan Berrebbi and Ronan Collobert and Samy Bengio and Navdeep Jaitly and Tatiana Likhomanenko},
  title = {{Continuous pseudo-labeling from the start}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=m3twGT2bAug},
}
Figure