Introduction
In the domain of Automatic Speech Recognition (ASR), self-training (or pseudo-labeling) has emerged as a powerful technique to leverage massive amounts of unlabeled data. Traditionally, this involves taking a small amount of labeled audio, training a "teacher" model, and using it to generate "pseudo-labels" (PLs) for a large unlabeled dataset. These PLs are then used to train a more robust "student" model.
However, self-training faces a significant hurdle known as the "Cold Start" problem. If self-training begins too early, the model generates poor-quality labels, learns from its own mistakes, and quickly collapses into a degenerate state—often outputting empty strings. To circumvent this, current state-of-the-art continuous training methods (like slimIPL) rely on a supervised pre-training "warm-up" phase. The authors of this paper argue that in low-resource settings (e.g., only 10 hours of labeled data), this initial phase causes the model to overfit to the tiny labeled set, ultimately degrading its final generalization performance. Researchers from Apple and CMU propose a novel approach to stabilize continuous pseudo-labeling from the very first step of training, entirely skipping the traditional supervised warm-up.
Methods: Stabilizing Early Pseudo-Labeling
To prevent model collapse without relying on supervised pre-training, the authors introduce two key techniques:
1. Dynamic Cache Curriculum
In continuous pseudo-labeling frameworks, a cache of pseudo-labels is maintained and periodically updated. Instead of using a fixed probability to update this cache, the authors introduce a dynamic curriculum based on the Levenshtein distance (edit distance) between the old pseudo-label and the model's current prediction.
If a label is changing rapidly between training steps, it indicates the model is still actively learning about that specific sample. By monitoring this "evolution," the system dynamically adjusts the probability of updating the cache ($1 - p_{out}$). This keeps the training stable and prevents the model from getting stuck on early, incorrect guesses.
2. Alignment Sampling with Temperature
Standard pseudo-labeling typically uses the "best" transcription (the argmax of the output distribution). Early in training, however, the model is often overconfident about incorrect answers. To counteract this, the authors sample transcriptions from the model's predictive distribution rather than taking the absolute best result.
They employ a temperature schedule ($\tau$): starting with high randomness (high temperature) to inject noise and prevent collapse, and gradually cooling down to a near-deterministic state ($\tau = 0.1$) as the model becomes more accurate. This sampling acts as a crucial regularizer that keeps the model from self-reinforcing its early errors.
Key Insights
- More Pre-training Isn't Always Better: One of the most surprising findings was that increasing the supervised pre-training phase from 10k to 40k steps actually increased the Word Error Rate (WER) from 14.3% to 22.9%. The model overfits to the small labeled set so severely that it loses the ability to generalize to the unlabeled data, validating the need to start self-training immediately.
- Label Evolution as a Signal: The rate at which a pseudo-label changes over time serves as an excellent proxy for how "useful" that sample is.
As shown above, there is a strong correlation between the change in pseudo-labels (WER between step $k$ and $k'$) and the actual error of those labels compared to the ground truth. This justifies using label evolution to drive the dynamic cache curriculum.
Datasets and Results
The researchers evaluated their approach across several benchmarks, including LibriSpeech (10-hour and 100-hour subsets), Libri-Light, and Common Voice (French) to test cross-lingual applicability.
- Matching State-of-the-Art: On the LibriSpeech 10h benchmark, their "from the start" method matched or outperformed models like wav2vec 2.0 and slimIPL, which rely on complex multi-stage pre-training pipelines.
- Low-Resource Wins: On the Common Voice French 10h task, the proposed method achieved a 26.0% WER, marking a significant 18% relative reduction over the slimIPL baseline (31.1% WER).
- Simplicity: Notably, these results were achieved without the need for an external Language Model (LM), which is frequently used as a crutch to compensate for poor acoustic model performance in early training stages.
Limitations and Future Work
While highly effective, the current work focuses on models trained specifically with the Connectionist Temporal Classification (CTC) loss. Although the authors hypothesize that these techniques should transfer to Transducer or Sequence-to-Sequence models, this remains to be empirically proven. Additionally, removing the pre-training phase introduces new hyperparameters, such as the temperature schedule and cache update functions, which may require careful tuning for novel datasets. Future directions include exploring how these dynamic cache strategies could be adapted to handle out-of-domain unlabeled data.
Summary updated