Semi-supervised speech recognition increasingly relies on a slow corrector to clean up a fast model's guesses. A small speech recognizer transcribes unlabeled audio. A large audio-aware language model then rewrites those transcripts, and the recognizer is retrained on the rewritten versions. Here a pseudo-label is a model-generated label treated as training data, and iterative refinement is a loop in which a model drafts an output, critiques it, and revises it. The loop works: ReHear reaches 6.34±0.26% word error rate on Earnings-21, compared with 7.79±0.08% for rule-filtered iterative pseudo-labeling. The cost is that every unlabeled utterance goes through the heavy corrector in every round. Most of that compute is plausibly wasted. Many first-pass transcripts are already correct, some errors cannot be fixed by the corrector, and some "corrections" are hallucinations that make the label worse.
This proposal treats refinement compute as a budget and allocates it one utterance at a time. A cheap predictor, trained on the small labeled set, estimates for each unlabeled utterance whether one more corrector pass will lower its transcript error. A second estimate decides whether to keep or revert the corrector's edit. Refiner passes go to the utterances with the highest predicted gain until the budget runs out. The claim is that this matches ReHear's accuracy with a fraction of the corrector calls. It might also beat ReHear, because it declines harmful edits that uniform refinement accepts.
Core Claim
In refinement-based pseudo-labeling, the expected benefit of a refiner pass varies widely across examples and can be negative. That benefit can be predicted cheaply from signals available before or just after the pass. Allocating passes by predicted benefit should shift the accuracy-versus-refiner-compute curve. Uniform refinement, random subsets, and allocation by the student's confidence should all fall below it.
The new part is the object being decided. Existing pseudo-labeling work decides which labels to accept: SSRM uses a fixed 0.8 confidence cutoff, and Rethinking Pseudo Labels for Semi-supervised Object Detection uses class-specific thresholds. Existing refinement work decides how to make the deployed model fast: Meta-Reflection internalizes reflection into one pass, and Parallel-Distill-Refine bounds sequential tokens. Neither line decides how many refinement passes each unlabeled example deserves while the training labels are being produced. That question sits squarely between the two directions.
Prior Work and Its Role
ReHear supplies the method we reuse and also motivates the proposal. It iterates between a recognizer and an audio LLM corrector that conditions on both the waveform and the hypothesis. It refines the whole corpus each round, its gains saturate around the second iteration, and it names hallucinated corrections as an open threat. Saturation suggests most passes in later rounds add little. The hallucination concern suggests some passes subtract value.
StreamHear sets a constraint in the opposite direction. It shows that a domain-adapted offline teacher, used once and without iteration or confidence filtering, can train a streaming student to 9.56% WER on Earnings-22, against 9.55% for the teacher. Its finding that confidence filtering hurt warns us that student confidence may be a poor proxy for label usefulness in speech. This is exactly why we predict refinement gain rather than reuse confidence.
Self Iterative Label Refinement via Robust Unlabeled Learning and Agent-R supply the cautionary evidence that unguided self-refinement can spiral downward. On Green Patent, GPT-4o drops from about 68% to about 61%. This motivates the option to revert an edit.
Self-Refine and Volcano document the per-query cost of refinement. Volcano goes from 2.7s to 5.8s per response, and that cost here multiplies across an entire unlabeled corpus. We do not change their test-time loops. We move the cost question to training time, where it is paid per corpus rather than per user.
Research Program
Each self-training round starts with the current recognizer transcribing the unlabeled pool. For each utterance $i$, we compute a cheap feature vector from the recognizer alone. It includes token posterior statistics, disagreement within the recognizer's top few candidate transcripts (the N-best list), speaking rate, and the density of rare words and entities.
A pre-refinement gain predictor $g(x_i)$ estimates the expected word-error reduction from one corrector pass. After a pass, a post-refinement predictor sees additional features: edit distance between draft and revision, the corrector's likelihood of its edit, and whether the edit introduces words absent from the recognizer's N-best list. It then outputs keep, revert, or refine again.
Allocation under a budget of $B$ corrector passes per round is a greedy knapsack:
$$\max_{S} \sum_{i \in S} \hat{g}_i \quad \text{subject to} \quad |S| \le B,$$
where $S$ is the set of (utterance, pass) pairs chosen and $\hat{g}_i$ is the predicted gain of that pass. Passes with $\hat{g}_i \le 0$ are never spent. Because each pass is unit cost, this reduces to sorting by predicted gain.
The predictors are trained on the labeled seed. On held-out folds of that seed, we run the recognizer and corrector, measure the true per-utterance WER change for each pass, and regress on the features. We cross-fit, using a recognizer trained without that fold. Without cross-fitting, the recognizer's confidence on its own training audio is overconfident, and the predictor would learn the wrong relationship.
The predictors are refit every round, because the recognizer changes. Refitting is cheap relative to corrector passes over the full pool.
The main risk is mismatch between the labeled seed and the unlabeled pool. We check the gain predictor's calibration on a portion of labeled audio held out and treated as unlabeled. This gives us the predictor's actual ranking quality, separately from its effect on downstream WER.
Baselines
The naive reference is the recognizer fine-tuned only on the labeled seed, with no pseudo-labels. The non-refining reference is StreamHear-style pseudo-labeling: one pass from a domain-adapted offline teacher, with all labels kept (StreamHear). It is run on the same labeled and unlabeled splits as our method, with the same recognizer architecture. Rule-filtered iterative pseudo-labeling is also included; ReHear reports 7.79±0.08% WER for it on Earnings-21.
The strongest competitor is ReHear itself with full-corpus refinement every round. It reports 6.34±0.26% WER on Earnings-21, and it is the method our approach must match at lower cost. We will reimplement it on our hardware and data splits rather than rely only on the published figure, reporting both. All arms will use the same recognizer, the same corrector checkpoint, the same decay schedule, the same number of rounds, and the same tuning budget for recognizer training.
Two matched-budget baselines isolate whether where passes go matters: random allocation of $B$ passes, and allocation to the lowest-confidence utterances. An oracle that allocates using true labels on a fully labeled held-out pool gives the ceiling for any allocation rule.
Two ablations isolate the key ingredients. The first removes the revert option, accepting every edit, which tests whether avoiding harmful edits matters. The second replaces the learned gain predictor with a convergence stop that halts when a pass leaves the transcript unchanged.
Beating the strongest baseline means one of two results, each across three seeds. Either we match ReHear's WER within its reported ±0.26 band using at most a third of its corrector passes. Or we achieve lower WER than ReHear at an equal number of passes, with the gap exceeding seed variance.
Why not just…?
Why not just refine the low-confidence utterances? This is the most serious threat, because confidence is free and often correlates with error. Two things make it a weaker target. First, confidence measures whether the recognizer is unsure, not whether the corrector can fix the error or will hallucinate. Second, StreamHear found confidence filtering harmful in a closely related speech setting. The matched-budget confidence arm settles it. If confidence allocation matches the learned predictor across budgets, we will report that plainly. The contribution then shrinks to the budget curve and the revert ablation.
Why not just run one full round of refinement and stop? ReHear's saturation near iteration two makes this plausible as a cheap compromise. The budget sweep includes "one full round" and "two full rounds" as points. If allocated refinement does not dominate them, uniform early stopping is enough.
Why not skip refinement and use a better one-pass teacher? StreamHear shows an adapted teacher alone gets the student to teacher parity. However, ReHear's 7.79% to 6.34% gap indicates that refinement adds something on Earnings-21. The StreamHear-style arm on identical splits tests directly whether that gap survives against a strong adapted teacher. If it does not, refinement compute of any allocation is unjustified on this domain.
Why not compress the refiner into the recognizer, as Meta-Reflection does? Meta-Reflection targets test-time latency and requires teacher-verified reflections plus white-box layer access. Our deployed recognizer is already single-pass. The cost we attack is the training-time corrector sweep, which internalization does not remove.
Proposed Experiments
The primary setting follows ReHear on Earnings-21, with Earnings-22 and SPGISpeech as additional earnings-call domains. AMI is the harder shift test: spontaneous multi-party meetings, where hallucinated corrections should be more frequent and the revert option should matter most. Metrics are test WER, pseudo-label WER on held-out audio, and total corrector passes and GPU-hours per round. We also measure the gain predictor's ranking quality against true per-utterance gain.
The decisive check is the WER-versus-corrector-passes curve at budgets from 10% to 200% of one full corpus round. It plots the learned allocator against the random, confidence, convergence-stop, uniform-round, and oracle arms. The proposal succeeds if the learned curve sits strictly between confidence allocation and the oracle at most budgets, and if it reaches ReHear-level WER at a minority of ReHear's passes.
A secondary check counts edits that increase pseudo-label error, measured on the held-out labeled audio. We compare full refinement with the revert-enabled allocator; the revert option should cut harmful edits substantially. A failure check asks whether gain-predictor calibration degrades on AMI relative to Earnings-21. That outcome would mean the approach needs in-domain labeled seed data to work.
References
- ReHear: Iterative Pseudo-Label Refinement for Semi-Supervised Speech Recognition via Audio Large Language Models
- Semi-Supervised Reward Modeling via Iterative Self-Training
- Rethinking Pseudo Labels for Semi-supervised Object Detection
- Meta-Reflection: A Feedback-Free Reflection Learning Framework
- Rethinking Thinking Tokens: LLMs as Improvement Operators
- StreamHear: Domain-Adapted Pseudo-Labeling for Semi-Supervised Streaming Speech Recognition
- Self Iterative Label Refinement via Robust Unlabeled Learning
- Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training
- Self-Refine: Iterative Refinement with Self-Feedback
- Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision