Continuous Pseudo-Labeling from the Start

Dan Berrebbi, Ronan Collobert, Samy Bengio, Navdeep Jaitly, Tatiana Likhomanenko

Abstract

Self-training (ST), or pseudo-labeling has sparked significant interest in the automatic speech recognition (ASR) community recently because of its success in harnessing unlabeled data. Unlike prior semi-supervised learning approaches that relied on iteratively regenerating pseudo-labels (PLs) from a trained model and using them to train a new model, recent state-of-the-art methods perform 'continuous training' where PLs are generated using a very recent version of the model being trained. Nevertheless, these approaches still rely on bootstrapping the ST using an initial supervised learning phase where the model is trained on labeled data alone. We believe this has the potential for over-fitting to the labeled dataset in low resource settings and that ST from the start of training should reduce over-fitting. In this paper we show how we can do this by dynamically controlling the evolution of PLs during the training process in ASR. To the best of our knowledge, this is the first study that shows the feasibility of generating PLs from the very start of the training. We are able to achieve this using two techniques that avoid instabilities which lead to degenerate models that do not generalize. Firstly, we control the evolution of PLs through a curriculum that uses the online changes in PLs to control the membership of the cache of PLs and improve generalization. Secondly, we find that by sampling transcriptions from the predictive distribution, rather than only using the best transcription, we can stabilize training further. With these techniques, our ST models match prior works without an external language model.

Introduction

In semi-supervised learning for Automatic Speech Recognition (ASR), Self-Training (ST)—or pseudo-labeling—is a dominant technique. Traditionally, this involves a two-stage process: first, "bootstrapping" a model on a small amount of labeled data, and then using that model to generate labels for a larger unlabeled pool. Recent "continuous" methods generate these pseudo-labels (PLs) on the fly. However, even these methods typically require a significant number of initial supervised pre-training steps.

This paper argues that in low-resource scenarios, this initial phase causes the model to overfit to the tiny labeled set, limiting its eventual performance. The challenge is that starting pseudo-labeling from step zero is notoriously unstable; without a decent starting point, the model often collapses into a "degenerate" state, such as predicting only empty transcriptions. We propose a method to perform continuous pseudo-labeling from the very first step of training ($M=0$).

Stabilizing Self-Training from Step Zero

To stabilize training from the start, we introduce two primary techniques:

1. Dynamic Cache Control via PL Evolution

Standard continuous ST uses a cache to store unlabeled samples and their PLs. Usually, samples are replaced in the cache with a constant probability. We propose a curriculum where the replacement probability ($1 - p_{out}$) is a function of the Levenshtein distance (edit distance) between the old PL in the cache and a new PL generated by the current model. This allows the system to prioritize samples based on how much the model's "opinion" of them is changing.

Comparison of the proposed continuous pseudo-labeling framework against the slimIPL baseline. Our method uses a dynamic replacement probability based on label evolution.

2. Alignment Sampling

Instead of always using the most likely transcription (argmax), we sample transcriptions from the model’s predictive distribution. We use a temperature schedule ($ au$), starting with high temperature (more noise) and decaying to low temperature (more "hard" labels). This prevents the model from prematurely converging on wrong labels and provides a lower bound on label entropy, which stops the model from collapsing into the "all-blanks" failure mode common in Connectionist Temporal Classification (CTC) training.

PL Evolution as a Quality Proxy

A key insight of this work is that the rate at which a pseudo-label changes over time (its "evolution") correlates strongly with the actual error rate of that label. By tracking this, the model can effectively "know what it doesn't know." As shown in the figure below, the Word Error Rate (WER) between PLs at different training steps is a reliable proxy for the ground truth error.

Correlation between the evolution of pseudo-labels (WER between PLs at different steps) and their actual error relative to ground truth.

Key Results

  • Performance: The proposed method matches or outperforms previous state-of-the-art models (like wav2vec 2.0 and slimIPL) in 10h and 100h settings, notably without requiring an external language model or a complex multi-stage pipeline.
  • Low-Resource Gains: On the Common Voice French 10h task, the method achieved an 18% relative reduction in WER compared to the slimIPL baseline.
  • Simplicity: By removing the pre-training steps, the training pipeline becomes a single, continuous process from start to finish.

Conclusion

By eliminating the supervised bootstrapping phase, we reduce overfitting in low-resource ASR and simplify the training process into a single continuous stage. Our results demonstrate that with proper stabilization—namely dynamic cache control and alignment sampling—self-training can be effective from the very first training step.

Summary updated

Abstract

Self-training (ST), or pseudo-labeling has sparked significant interest in the automatic speech recognition (ASR) community recently because of its success in harnessing unlabeled data. Unlike prior semi-supervised learning approaches that relied on iteratively regenerating pseudo-labels (PLs) from a trained model and using them to train a new model, recent state-of-the-art methods perform `continuous training' where PLs are generated using a very recent version of the model being trained. Nevertheless, these approaches still rely on bootstrapping the ST using an initial supervised learning phase where the model is trained on labeled data alone. We believe this has the potential for over-fitting to the labeled dataset in low resource settings and that ST from the start of training should reduce over-fitting. In this paper we show how we can do this by dynamically controlling the evolution of PLs during the training process in ASR. To the best of our knowledge, this is the first study that shows the feasibility of generating PLs from the very start of the training. We are able to achieve this using two techniques that avoid instabilities which lead to degenerate models that do not generalize. Firstly, we control the evolution of PLs through a curriculum that uses the online changes in PLs to control the membership of the cache of PLs and improve generalization. Secondly, we find that by sampling transcriptions from the predictive distribution, rather than only using the best transcription, we can stabilize training further. With these techniques, our ST models match prior works without an external language model.
BibTeX
@inproceedings{berrebbi2023continuous,
  author = {Dan Berrebbi and Ronan Collobert and Samy Bengio and Navdeep Jaitly and Tatiana Likhomanenko},
  title = {{Continuous Pseudo-Labeling from the Start}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=m3twGT2bAug},
}

Loading related content...

Figure