Abstract
Self-training (ST), or pseudo-labeling has sparked significant interest in the automatic speech recognition (ASR) community recently because of its success in harnessing unlabeled data. Unlike prior semi-supervised learning approaches that relied on iteratively regenerating pseudo-labels (PLs) from a trained model and using them to train a new model, recent state-of-the-art methods perform 'continuous training' where PLs are generated using a very recent version of the model being trained. Nevertheless, these approaches still rely on bootstrapping the ST using an initial supervised learning phase where the model is trained on labeled data alone. We believe this has the potential for over-fitting to the labeled dataset in low resource settings and that ST from the start of training should reduce over-fitting. In this paper we show how we can do this by dynamically controlling the evolution of PLs during the training process in ASR. To the best of our knowledge, this is the first study that shows the feasibility of generating PLs from the very start of the training. We are able to achieve this using two techniques that avoid instabilities which lead to degenerate models that do not generalize. Firstly, we control the evolution of PLs through a curriculum that uses the online changes in PLs to control the membership of the cache of PLs and improve generalization. Secondly, we find that by sampling transcriptions from the predictive distribution, rather than only using the best transcription, we can stabilize training further. With these techniques, our ST models match prior works without an external language model.
Introduction
In semi-supervised learning for Automatic Speech Recognition (ASR), Self-Training (ST)—or pseudo-labeling—is a dominant technique. Traditionally, this involves a two-stage process: first, "bootstrapping" a model on a small amount of labeled data, and then using that model to generate labels for a larger unlabeled pool. Recent "continuous" methods generate these pseudo-labels (PLs) on the fly. However, even these methods typically require a significant number of initial supervised pre-training steps.
This paper argues that in low-resource scenarios, this initial phase causes the model to overfit to the tiny labeled set, limiting its eventual performance. The challenge is that starting pseudo-labeling from step zero is notoriously unstable; without a decent starting point, the model often collapses into a "degenerate" state, such as predicting only empty transcriptions. We propose a method to perform continuous pseudo-labeling from the very first step of training ($M=0$).
Stabilizing Self-Training from Step Zero
To stabilize training from the start, we introduce two primary techniques:
1. Dynamic Cache Control via PL Evolution
Standard continuous ST uses a cache to store unlabeled samples and their PLs. Usually, samples are replaced in the cache with a constant probability. We propose a curriculum where the replacement probability ($1 - p_{out}$) is a function of the Levenshtein distance (edit distance) between the old PL in the cache and a new PL generated by the current model. This allows the system to prioritize samples based on how much the model's "opinion" of them is changing.
2. Alignment Sampling
Instead of always using the most likely transcription (argmax), we sample transcriptions from the model’s predictive distribution. We use a temperature schedule ($ au$), starting with high temperature (more noise) and decaying to low temperature (more "hard" labels). This prevents the model from prematurely converging on wrong labels and provides a lower bound on label entropy, which stops the model from collapsing into the "all-blanks" failure mode common in Connectionist Temporal Classification (CTC) training.
PL Evolution as a Quality Proxy
A key insight of this work is that the rate at which a pseudo-label changes over time (its "evolution") correlates strongly with the actual error rate of that label. By tracking this, the model can effectively "know what it doesn't know." As shown in the figure below, the Word Error Rate (WER) between PLs at different training steps is a reliable proxy for the ground truth error.
Key Results
- Performance: The proposed method matches or outperforms previous state-of-the-art models (like wav2vec 2.0 and slimIPL) in 10h and 100h settings, notably without requiring an external language model or a complex multi-stage pipeline.
- Low-Resource Gains: On the Common Voice French 10h task, the method achieved an 18% relative reduction in WER compared to the slimIPL baseline.
- Simplicity: By removing the pre-training steps, the training pipeline becomes a single, continuous process from start to finish.
Conclusion
By eliminating the supervised bootstrapping phase, we reduce overfitting in low-resource ASR and simplify the training process into a single continuous stage. Our results demonstrate that with proper stabilization—namely dynamic cache control and alignment sampling—self-training can be effective from the very first training step.
Summary updated