Escalate When Meaning Is at Risk: Semantic Gating for Compute-Adaptive Speech Recognition

Speech recognizers that save compute do it by deciding, per utterance, how much model to spend. They run a small model first and hand the audio to a larger one only when needed (a cascade), or they stop a deep network at an intermediate layer once its output looks reliable enough (an early exit). Every such decision rule in speech recognition today asks one question: is the transcript likely to have many wrong words? That question is tied to word error rate (WER), the share of words substituted, deleted or inserted. WER counts dropping "not" from "should not take insulin" exactly like turning "the" into "a." A system tuned to save compute at a fixed WER can therefore keep the cheap transcript precisely when it carries a confident, meaning-breaking error, and escalate when the small model is merely unsure about filler words.

We propose to train the escalation decision on meaning rather than word count. The decision rule is a small, reference-free predictor that estimates how much the meaning of the cheap transcript would improve if the utterance were sent to the larger model. Utterances are escalated in order of predicted meaning gain per unit of compute. The claim is testable and could fail. At a fixed compute budget, a gate trained on a semantic target should keep more of the large model's meaning preservation than a gate with the same inputs trained on a WER target, and more than the confidence rules used today. The advantage should be largest on disordered and conversational speech, where WER and meaning diverge most.

Core Claim

The most direct prior work is the early-exit study of Training Early-Exit Architectures for Automatic Speech Recognition. It attaches recognition heads to intermediate layers of Conformer and wav2vec-style models and gates exits with a sentence confidence score computed from a 300-best list. Both its training and its exit rule are judged purely by WER. Our method borrows its exit and confidence machinery and changes what the gate is trained to predict.

Model Cascading supplies the cascade recipe for text classifiers. With BERT models from Mini to Large and gating on how far the output distribution is from uniform, it matched BERT-Large on QQP at 11.07% of its compute. It does not cover sequence outputs, where no single categorical distribution summarizes confidence.

ASR Error Detection via Audio-Transcript entailment shows that a model asking "does this audio entail this transcript?" detects errors better than recognizer confidence. It reports 26.2% classification error overall and 23% on medical errors, relative gains of 12% and 15.4%. That study uses the detector only as a standalone post-hoc flag and never to allocate compute. It motivates our proposal, because it is direct evidence that posterior confidence misses errors that matter.

What is new here has two parts. The first is the training target, which is the expected semantic gain from escalation rather than the expected word error of the cheap output. The second is the evaluation, which reports semantic quality against compute rather than WER against compute. The gain framing matters on its own. On impaired speech the large model often fails too, and a confidence rule spends compute there for nothing.

Research Program

The work proceeds in three stages. The first stage is a measurement study that is cheap and informative regardless of what follows. We take existing compute-saving policies, confidence-gated Whisper cascades and N-best-gated early exits, at operating points matched on WER. We then ask whether they are also matched on meaning. Meaning is scored with SemScore, the challenge metric of The Interspeech 2025 Speech Accessibility Project Challenge. It combines logical entailment, BERTScore contextual similarity and Soundex phonetic similarity. We also tag a set of "critical" errors by rule: negations, numbers, named entities and drug names.

If WER-matched policies turn out to be meaning-matched as well, the rest of the program loses its motivation, and we report that. The prior evidence suggests they will not match, for several reasons:

The second stage builds the gate. Its inputs are things the cheap model already produces or that cost little to compute:

  • per-word posteriors and N-best disagreement, with each word weighted by an estimate of its importance from a small text language model's surprisal (following the parity paper's finding that machine errors are high-surprisal);
  • flags for negation, number and entity tokens in the N-best;
  • the spread of a small sentence encoder's embeddings across the top few hypotheses, which measures whether competing hypotheses mean different things or only spell things differently.

The training label for an utterance is the semantic score of the large model's transcript minus that of the small model's, both scored against the reference. Labels are computed offline on training data. At inference no reference and no large-model pass is needed. The gate's own cost is counted in every compute figure, and we cap it at a few percent of the small model's FLOPs.

One fact rules out the shortcut of simply running a fast semantic metric at inference. SeMaScore is 41 times faster than BERTScore (0.047 s versus 1.95 s per sample on Torgo), but it still needs a reference transcript, and at runtime there is none.

The third stage moves the gate inside a single network. The same predictor decides early exits in a multi-exit Conformer trained from scratch with the compound loss of the early-exit paper. This tests whether the result depends on having separate models.

Disordered speech raises a known tension, noted by the SAP challenge and by Aligning ASR Evaluation with Human and LLM Judgments: larger models smooth away repetitions that clinicians need. A semantic gate trained on SemScore could reward that smoothing. We therefore report verbatim disfluency retention alongside semantic scores and treat any loss of it as a cost, not a win.

Baselines

The naive reference points are the two ends of the compute range. "Small only" spends nothing extra. "Large only" escalates everything. On the Speech Accessibility Project data, the untuned whisper-large-v2 baseline reported in The Interspeech 2025 Speech Accessibility Project Challenge scores 17.82% WER and 75.85% SemScore. The challenge's winning Parakeet-based system reaches 8.11% WER and 88.44% SemScore. These numbers anchor the large end. Our own large-only runs on identical normalization will be the comparison actually used. We also include random escalation at each budget, which every gate must beat.

The strongest existing method is confidence gating with the N-best sentence confidence of Training Early-Exit Architectures for Automatic Speech Recognition, applied both to exits and to cascade escalation. We complement it with the distance-to-uniform rule from Model Cascading, adapted to average per-token distributions. No reported result was found on Lacuna for a confidence-gated ASR cascade on SAP, and the file gives no per-layer LibriSpeech WER numbers for the early-exit paper. We therefore rerun both rules ourselves on the same models, the same N-best size and the same data splits. Their thresholds are tuned on the same development set with the same search budget as our gate.

The ablation that isolates the key ingredient is a gate with identical inputs, architecture and training budget, but trained to predict the WER reduction from escalation instead of the semantic gain. A second ablation trains the gate to predict the cheap transcript's semantic error rather than the gain, which isolates the value of modeling whether the large model will actually help.

Beating a baseline means the following. Over a sweep of compute budgets, the semantic gate's curve of quality against compute must lie above the baseline's curve. The difference in area under the curve must have a bootstrap confidence interval excluding zero. At the operating point using 30% of large-only compute, the gate must also leave fewer critical errors un-escalated, judged by the same rule-based tagging.

Why not just…?

Why not just gate on posterior or N-best confidence, which is cheap and well understood? This is the main threat. Confident outputs usually are correct, so confidence may already capture most of the meaning risk. The case against it rests on the audio-transcript entailment results, where an entailment model beat recognizer confidence by 12% relative overall and 15.4% on medical errors. Disordered speech adds further evidence: models produce confident repetitions with WER above 200% while staying intelligible, so confidence wastes escalations there. Still, this is indirect evidence. The WER-target ablation and the head-to-head frontier settle the question directly. If the semantic gate does not beat the WER-trained gate with the same inputs, we will conclude that WER-style confidence suffices and report it.

Why not just run the largest model on everything, since scale is what closed the gap on SAP? The SAP challenge shows large fine-tuned models nearly halving WER, but it never counts compute. Model Cascading shows that a well-gated cascade can match the large model at a fraction of the cost, and sometimes beat it. Large-only is one point on our curves. The question is how close to it a gated system gets, and at what cost. If large-only dominates even at modest budgets, the proposal has no practical value, and the curves will show that.

Why not just compute a fast semantic metric such as SeMaScore, or run the Phukon-style integrated score, at inference? Both compare the hypothesis to a reference transcript, and none exists at runtime. Using the large model's output as a stand-in reference would require running the large model on every utterance, which defeats the purpose. Our gate is effectively a distillation of such metrics into a reference-free predictor.

Why not just send every cheap transcript through an LLM corrector? ASR Error Correction Using Large Language Models shows that this reduces WER, but only as a uniform post-pass on every utterance. Zero-shot correction needs N-best or lattice constraints to avoid hallucination, and the SAP challenge reports that such correction smooths away disfluencies. We include LLM correction as an extra arm with its compute counted, and we report disfluency retention for it.

Proposed Experiments

The primary setting is the Speech Accessibility Project corpus with its five etiologies. We use a Whisper tiny, small and large-v2 cascade, both zero-shot and fine-tuned on the SAP training split. Results are reported per etiology, because Parkinson's disease makes up about three quarters of samples and could mask the others.

The secondary settings are chosen for different reasons. VoxPopuli tests accented, spontaneous speech, which is where WER and downstream answer error rate are known to diverge. TED-LIUM 3 and LibriSpeech are lower-divergence controls on which we expect little advantage. The ATIS accented corpus gives a downstream intent-accuracy check. The early-exit variant runs on LibriSpeech, TED-LIUM 3 and VoxPopuli with multi-exit Conformers trained from scratch.

To avoid grading the gate with the metric it was trained on, we hold out evaluation signals. The gate trains on an entailment-weighted semantic score. It is then judged on four things: SemScore, ATIS intent accuracy, answer error rate on a VoxPopuli subset, and the critical-error escalation rate. We add a fresh human study as well, rating intelligibility on a 1-to-5 scale for 300 utterances where the semantic and confidence gates disagree. The protocol follows the Phukon et al. design, which reached a Pearson correlation of 0.890 with an integrated metric on only 100 pairs. Compute is reported as total FLOPs per utterance, including the gate, along with wall-clock latency on one GPU.

Four results are decisive:

  • At WER-matched operating points, the measurement stage must show a meaning gap between policies. If it does not, the program stops there with that finding.
  • The semantic gate must beat the WER-target ablation on quality-versus-compute area for SAP, and in the human study.
  • The advantage must shrink on LibriSpeech. An advantage of equal size everywhere would suggest the gate is merely a better learned gate, not one that captures meaning.
  • The gain-target gate must escalate fewer utterances on which the large model also fails, a pattern concentrated in severe dysarthria, without lowering disfluency retention.

A finding of no gap at the first or second check would mean that confidence gating already protects meaning. That would be a useful negative result for anyone deploying adaptive speech recognition in clinical settings.