Semi-supervised speech recognition usually runs in two stages. A model is first trained on the small labeled set (the "warm-up"). It is then trained on its own guesses, called pseudo-labels, for a much larger pile of unlabeled audio. Theory says the warm-up is necessary: self-training only improves a model that already starts below some error threshold. Practice says the warm-up is dangerous when labels are very scarce. With 10 hours of labeled LibriSpeech, extending warm-up from 10k to 40k steps raised final WER from 14.3% to 22.9% (Continuous pseudo-labeling from the start). That paper's answer is to remove warm-up altogether and add a temperature-sampling schedule and an edit-distance cache curriculum. This proposal argues the reverse. Keep the warm-up, but find out what goes wrong during it, and stop that specific thing.
Our hypothesis is that the damage comes from a failure specific to CTC, not from generic overfitting. Connectionist temporal classification (CTC) is the standard training loss for mapping long audio frame sequences to short transcripts without knowing where each word starts. It does this by letting the model emit a "blank" symbol on most frames. On a tiny labeled set, CTC tends to lock each utterance onto one single frame-to-character alignment. The output becomes a train of one-frame, near-certain spikes separated by long stretches of blank (Connectionist Temporal Classification with Maximum Entropy Regularization). A model in that state produces confident but wrong pseudo-labels. Self-training then hardens those errors. If the hypothesis holds, two things follow. First, holding a floor on alignment uncertainty during warm-up should make final accuracy largely insensitive to warm-up length. Second, the right moment to switch stages can be read from unlabeled audio alone, without a labeled development set.
Core Claim
The claim has two parts, and both can be falsified. The first is that the WER-versus-warm-up-length curve flattens under an entropy floor. With a floor on CTC alignment entropy during warm-up, a standard second stage (slimIPL or momentum pseudo-labeling) should lose little accuracy as warm-up grows from 10k to 40k steps. Without the floor, the degradation reported by Berrebbi et al. should reappear. The floor must also beat ordinary regularizers given the same tuning budget, such as heavier dropout, heavier SpecAugment, or supervised CR-CTC. If those regularizers do equally well, the "peakiness" story is wrong and the problem is plain overfitting.
The second part is a label-free stopping rule. A simple statistic measured on a fixed set of unlabeled probe utterances should predict when the model has entered the stable region where self-training helps. The statistic is "churn": how much the model's greedy transcripts of the probe set change between checkpoints. A switch rule based on when churn stops falling should match the best hand-tuned warm-up length.
The closest existing work removes warm-up (Continuous pseudo-labeling from the start), or keeps a fixed, hand-chosen warm-up (slimIPL, Momentum Pseudo-Labeling, InterMPL, Semi-Supervised ASR by End-to-End Self-Training, which switches at a fixed epoch 35). The entropy regularizers we reuse (EnCTC, AdaMER-CTC) were only tested in fully supervised training. None of these papers asks whether the warm-up itself can be made robust.
Research Program
Warm-up objective. The warm-up stage trains a Conformer CTC model on the labeled set with an entropy floor rather than an entropy bonus:
$$\mathcal{L}_{\text{warm}} = \mathcal{L}_{\text{CTC}} + \lambda \max\big(0,\; h_0 - \tfrac{1}{T}\mathcal{H}(\pi \mid Y, X)\big).$$
The terms are defined as follows:
- $\mathcal{H}(\pi \mid Y, X)$ is the entropy over valid alignments of transcript $Y$ given audio $X$. It is computed exactly with the CTC forward-backward recursion, as in EnCTC.
- $T$ is the number of frames.
- $h_0$ is a per-frame entropy target.
- $\lambda$ is a weight.
We use a floor because of a property of AdaMER-CTC. That method learns a multiplier that shrinks over training so the model can sharpen late (AdaMER-CTC). Late sharpening is exactly what we want to prevent while the model still has only 10 hours of labels. The floor stops alignment collapse but does not reward unbounded uncertainty. We set $h_0$ from the entropy the model reaches at the point where unlabeled churn first plateaus. This ties $h_0$ to an observable quantity instead of searching over it freely.
Switch rule. Throughout warm-up, every $K$ steps we decode a fixed set of 500 unlabeled utterances greedily. We record the normalized edit distance between each checkpoint's transcripts and the previous checkpoint's. We switch to stage two when the smoothed rate of decrease in churn falls below a small fraction of its initial value.
The rule is motivated by theory. Self-training converts a weak learner into a strong one only once error falls below a constant threshold (Self-training Converts Weak Learners to Strong Learners in Mixture Models). Stabilizing predictions on unlabeled data is a measurable proxy for having crossed into that region. Berrebbi et al. use edit distance to decide which cache entries to refresh. We use it for a different purpose: deciding when the training stage ends.
Stage two. Stage two is held fixed as the controlled variable. It is either slimIPL with its dynamic cache, or momentum pseudo-labeling with its exponential-moving-average teacher. Both use greedy hard labels and the published recipes. We release the entropy floor at the switch.
Diagnostics. Alongside WER, we log three measurements of peakiness at the end of warm-up:
- the fraction of frames predicted as blank;
- the mean width of a non-blank emission;
- alignment entropy given the model's own hypothesis.
We then test whether these measurements predict final WER across all runs. This is the mechanistic test of the hypothesis.
Baselines
Naive reference. The model is trained only on the labeled 10 hours, with no unlabeled data. It shows how much any semi-supervised method adds at all.
Strongest existing two-stage method. This is slimIPL with its published warm-up schedule on LibriSpeech 10h labeled plus the remaining LibriSpeech audio as unlabeled data. It reports 11.4% WER on test-clean without a language model (slimIPL). We rerun it with our encoder, data, and step budget rather than comparing only against the published figure. Beating it means lower test-clean and test-other WER than our rerun, with no language model, over three seeds and at equal total training steps. Our warm-up uses fewer steps if the switch rule fires early, and that saving counts in our favor.
Strongest no-warm-up competitor. This is continuous pseudo-labeling from the start. On Common Voice French 10h it reports 26.0% WER against slimIPL's 31.1% (Continuous pseudo-labeling from the start). No reported result was found on Lacuna for this method on LibriSpeech 10h test-clean, so we reimplement it on both datasets with its temperature schedule and give it the same hyperparameter-search budget as our $h_0$ and $\lambda$. Beating it means matching or lowering its WER on Common Voice French while being less sensitive to hyperparameters. We measure sensitivity as the spread in final WER across the searched settings.
Ablations. These isolate the key ingredient by replacing the entropy floor in an otherwise identical pipeline:
- plain CTC warm-up;
- heavier dropout and SpecAugment, matched in tuning budget;
- supervised CR-CTC in warm-up (CR-CTC), which also flattens spikes, through agreement between two augmented views;
- the switch rule applied to plain CTC.
The last ablation separates the value of when to switch from the value of how to warm up.
Why not just…?
Why not just skip warm-up, as continuous pseudo-labeling from the start does? This is the real threat, and we treat it as a head-to-head baseline rather than dismissing it. Its reported gain over slimIPL on Common Voice French is large. Our argument for keeping warm-up rests on two points: the warm-up model is simpler, and it does not depend on a temperature cooling schedule that must be recalibrated per dataset. If skipping warm-up beats our method on both datasets and is no more sensitive to hyperparameters, we will report that the warm-up should go. The peakiness diagnostics would still explain why skipping it helps.
Why not just tune warm-up length on a development set? Tuning would avoid the 22.9% failure if dev WER tracked the final result. But it costs a full stage-two run for each candidate length. It also needs a labeled dev set, which in a true 10-hour setting is labeled data taken away from training. Our sweep over 5k, 10k, 20k, and 40k steps produces exactly this oracle curve. If the oracle-best length beats our switch rule by more than seed noise, the rule fails. The entropy-floor claim then stands or falls only on whether the curve flattens.
Why not just regularize harder with dropout and augmentation, or CR-CTC? If the harm is ordinary overfitting, these should work as well as the floor, and slimIPL already relies heavily on dropout in its early phase. The ablations decide this directly. If heavier generic regularization flattens the curve as much as the entropy floor, the CTC-specific explanation is wrong. We would then report that, not argue around it.
Why not just filter pseudo-labels by confidence? Confident collapsed spikes are exactly the failure mode, so per-frame CTC confidence is inflated in the collapsed state. Seq-UPS shows that raw sequence confidence is poorly calibrated even for autoregressive decoders (Seq-UPS). We will add a confidence-filtered slimIPL run to confirm this does not rescue long warm-ups.
Proposed Experiments
Main experiment: LibriSpeech. We use the 10h labeled split with the remaining LibriSpeech audio unlabeled, and report WER on dev and test, clean and other, without a language model.
- We sweep warm-up length over 5k, 10k, 20k, and 40k steps under each warm-up variant: plain, entropy floor, matched dropout and augmentation, and CR-CTC.
- Every run is followed by identical slimIPL and momentum pseudo-labeling second stages.
- The decisive outcome is whether the entropy-floor curve stays within about one absolute WER point across lengths while plain CTC reproduces a degradation like 14.3% to 22.9%.
Second check: the switch rule. We compare the step where the churn rule fires against the oracle-best length from the sweep. The rule should land within one sweep interval of the oracle and match its final WER.
Third check: peakiness as a predictor. We pool all runs and regress final WER on the warm-up peakiness statistics and on labeled-set training loss. Peakiness should explain final WER better than training loss does. If it does not, the mechanism is not supported.
Transfer to other settings. We repeat the main comparison on Common Voice French 10h against continuous pseudo-labeling from the start, and on WSJ si84 to si284. The WSJ case reported a gain from 11.43% to 9.78% eval92 WER with a fixed epoch-35 switch (Semi-Supervised ASR by End-to-End Self-Training). This tests whether the rule removes the need to hand-pick the switch point in a setting with more labels.
Total compute is reported for every system, so any accuracy gain can be weighed against cost.
References
- Continuous pseudo-labeling from the start
- Connectionist Temporal Classification with Maximum Entropy Regularization
- slimIPL: Language-Model-Free Iterative Pseudo-Labeling
- Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition
- Intermpl: Momentum Pseudo-Labeling With Intermediate CTC Loss
- Semi-Supervised ASR by End-to-End Self-Training
- AdaMER-CTC: Connectionist Temporal Classification with Adaptive Maximum Entropy Regularization for Automatic Speech Recognition
- Self-training Converts Weak Learners to Strong Learners in Mixture Models
- CR-CTC: Consistency regularization on CTC for improved speech recognition
- Seq-UPS: Sequential Uncertainty-aware Pseudo-label Selection for Semi-Supervised Text Recognition