Speech systems that recognize words or identify speakers are usually forced to answer every input, even when background noise has made the audio nearly useless. A natural fix is selective prediction: give the system the option to say "I don't know" and hand the case to a human, while guaranteeing a maximum error rate on the cases it does answer. The hard part is the score that decides when to abstain. The model's own confidence is known to be overconfident on noisy, shifted inputs.
We propose a score built from a classical noise-removal method, spectral subtraction. This method estimates the average noise spectrum and subtracts it from the noisy spectrum. Its one free choice is how aggressively to subtract. The core idea is to run the recognizer on several versions of the same input, from the raw audio up to heavily subtracted audio, and to abstain when the answer depends on how hard we subtracted. If the decision is stable across subtraction strengths, we answer. If it flips, the input sits in the regime where denoising is known to both help and hurt, and we defer. The abstention threshold is then set with a finite-sample guarantee on the error among answered cases.
Core Claim
Spectral subtraction has a well-documented two-sided effect on downstream recognizers. It helps at low signal-to-noise ratio (SNR), the ratio of speech power to noise power. At high SNR it hurts, because it creates spectral artifacts. Assessment of single-channel speech enhancement techniques for speaker identification under mismatched conditions finds that subtraction improves speaker identification only at or below 10 dB and sometimes hurts above it. Impact of noise reduction and spectrum estimation on noise robust speaker identification finds that front-end gains under mismatched training vanish or reverse under matched conditions.
We treat this instability as signal rather than nuisance. Let $y$ be a noisy utterance. Let $x_k = \mathrm{SS}(y;\alpha_k,\beta_k)$ for $k=1,\dots,K$ be its spectrally subtracted versions, where $\alpha_k$ is the over-subtraction factor, $\beta_k$ is the spectral floor, and $\alpha_1=0$ is the untouched raw signal. Let $f$ be the recognizer. Our confidence score is the agreement of the decisions along this sweep:
$$ \kappa(y) = \frac{1}{K}\sum_{k=1}^{K} p_f\big(\hat{y}^{*} \mid x_k\big), \qquad \hat{y}^{*} = \arg\max_{c} \sum_{k} p_f(c \mid x_k). $$
Here $p_f(c \mid x_k)$ is the recognizer's probability for label $c$ on version $k$, and $\hat{y}^{*}$ is the consensus label. The system outputs $\hat{y}^{*}$ when $\kappa(y)\ge\theta$ and abstains otherwise. For speech recognition, we replace label posteriors with the mean pairwise word agreement among the $K$ hypotheses, and the consensus is the hypothesis closest to all the others.
The threshold $\theta$ is chosen with Selection with Guaranteed Risk (SGR) from Selective Classification for Deep Neural Networks. SGR inverts a binomial bound on a held-out calibration set so that, with probability $1-\delta$, the error rate among accepted inputs stays below a user-set target $r^*$. Because noise conditions drift, we also calibrate separately within strata defined by the SNR that the subtraction front-end itself estimates. This borrows the stratification logic of Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models, which shows that pooled conformal selection can collapse on hard subgroups.
The new element is a confidence signal that lives on a physically meaningful axis of the input, namely subtraction strength. It is paired with a calibrated, utterance-level abstention rule. Prior speech work rejects frames or time-frequency bins. It never rejects whole utterances, and it never uses sensitivity to the front-end as evidence of unreliability.
Relation to Prior Work
Several papers motivate the gap by showing that subtraction artifacts drive errors that forced-answer systems cannot escape. The synergy between bounded-distance HMM and spectral subtraction for robust speech recognition shows that subtraction-induced outlier features account for 20–40% of accumulated log-probability during decoding. Their remedy is to cap each outlier's penalty inside the acoustic model, so every utterance still receives a transcript. Missing feature theory with soft spectral subtraction for speaker verification uses subtraction to mark unreliable time-frequency regions and drops them from scoring. The authors report that at low SNR so much is dropped that verification falls below baseline. That is exactly the case where an utterance-level abstention should fire instead.
A second group supplies design constraints. An Efficient Feature Selection Method for Speaker Recognition uses subtraction only to decide which frames contain speech, then extracts features from the raw signal to avoid artifacts. It reports 11–16% relative equal error rate gains on distant-microphone data and under 1% on clean telephone speech. Robust voice activity detection based on the entropy of noise-suppressed spectrum thresholds spectral entropy after noise whitening, using a heuristic 91%-of-maximum cutoff. Both papers tell us that raw and subtracted views carry complementary information, which is why our sweep includes $\alpha=0$. Both also select frames with no error guarantee.
The noise-aware acoustic models are the strongest forced-answer backends we could wrap. These are Robust speech recognition using DNN-HMM acoustic model combining noise-aware training with spectral subtraction, which reports a 28.6% relative WER reduction over multi-condition training on CENSREC-1, and Speech-enhanced and Noise-aware Networks for Robust Speech Recognition. Both feed noise estimates as extra inputs. Neither outputs a calibrated reliability signal.
The closest design in spirit is Multiple Confidence Gates For Joint Training Of SE And ASR. It computes several speech masks from loose to strict and fuses them inside the recognizer. We keep the strictness variants separate and read their disagreement as uncertainty. Fusion hides exactly the information we want.
On the selective-prediction side, Towards Improving Selective Prediction Ability of NLP Systems shows that a random-forest calibrator trained on external difficulty features beats raw maximum probability, especially out of domain. We adopt that calibrator as a baseline fed with speech-specific features. This tests whether sweep agreement adds anything over cheap per-input statistics.
Research Program
We will build the sweep on a single noise tracker so that every version shares one noise estimate. The tracker uses minimum-statistics noise tracking as used by Vicente-Peña et al. The grid is $\alpha\in\{0,1,2,4\}$ and $\beta\in\{0.01,0.1\}$ on the nonzero settings, giving $K=7$ versions. The recognizer runs once per version, so the method costs $K$ forward passes and no retraining.
A critical design question is whether the backend has seen subtracted inputs during training. If it was trained only on raw audio, heavily subtracted versions are out of distribution, and disagreement measures domain mismatch rather than true ambiguity. We therefore train every backend in two variants: multi-condition on raw audio only, and multi-condition with subtraction strength randomly sampled from the grid as augmentation. We report results for both.
There are two task settings. The primary one is closed-set speaker identification on YOHO mixed with NOISEX-92 noise at 0–20 dB. The loss there is binary (misidentified or not), which fits the binomial bound directly, and the downstream two-sided effect of subtraction is documented on exactly this setup. The secondary one is utterance-level recognition on Aurora-4. Here the loss is sentence error, meaning any word wrong, and we also report WER restricted to accepted utterances.
For each setting we will analyze which inputs the sweep flags. We expect two regions:
- Disagreement concentrated at low SNR, where all versions are poor.
- A distinct high-SNR pocket where raw and subtracted versions disagree because of artifacts.
The second region is what an SNR threshold cannot catch.
Baselines
The naive reference is the forced-answer system at full coverage: the multi-condition backend on raw audio, answering everything. Its error rate is the risk every selective method must beat at matched coverage. On Aurora-4 the strongest forced-answer number in the file is WER = 3.55% for CNN-TDNNF + SENAN with SpecAugment, from Speech-enhanced and Noise-aware Networks for Robust Speech Recognition. On YOHO the file reports clean matched MFCC identification accuracy of 99.82% from Sadjadi & Hansen. No reported result was found on Lacuna for risk–coverage curves or guaranteed-risk coverage on Aurora-4, YOHO+NOISEX-92, or any speech benchmark. The only guaranteed-risk reference points in the file are from vision, for example 1% error at 78% coverage on CIFAR-10 with SGR in Selective Classification for Deep Neural Networks. We will therefore produce all selective baselines ourselves.
The first standard competitor is softmax response: the maximum class probability, or for ASR the decoder's average token posterior. We compute it both on raw audio and on fixed-strength subtracted audio, with the threshold set by SGR. Geifman and El-Yaniv found this simple score matched more expensive alternatives in vision. That makes it the bar any new score must clear.
The second is an estimated-SNR threshold taken from the same noise tracker. It is the obvious physical reliability signal and needs only one forward pass.
The third, and the strongest existing method, is a random-forest meta-calibrator in the style of Varshney et al. It is trained on a held-out split with these features:
- Maximum probability.
- Estimated SNR.
- The fraction of time-frequency bins hit by the spectral floor.
- The missing-feature mask density from Padilla et al.
The fourth is Monte Carlo dropout, meaning variance across stochastic forward passes.
The key ablation is generic test-time augmentation. It runs $K$ passes with random additive noise and SpecAugment-style masking instead of the subtraction sweep. This isolates whether the physical axis of subtraction strength matters or whether any input perturbation would do.
Fairness rules apply to every comparison. All methods share the same backend weights, calibration split, $r^*$, and $\delta=0.05$. Multi-pass methods get the same $K=7$ passes. The meta-calibrator gets the same held-out data that SGR uses, split once and reused for all methods, plus a matched hyperparameter search budget.
Beating the strongest baseline means two things on both tasks:
- A lower area under the risk–coverage curve than both the meta-calibrator and the matched-$K$ augmentation baseline, averaged over repeated calibration splits.
- Higher guaranteed coverage at fixed $r^*$.
If the sweep gains disappear once the meta-calibrator also receives the sweep-agreement feature, the contribution reduces to "a useful feature." We will report it honestly as that.
Why not just…?
Why not just threshold on estimated SNR? SNR is a strong predictor of error and is cheap. It is a real threat at low SNR. However, the documented failure at high SNR comes from subtraction distortion in clean-ish audio, as Sadjadi & Hansen and Godin et al. show, and an SNR score ranks those inputs as safe by construction. The decisive test is risk among accepted inputs in the top SNR stratum. If SNR thresholding matches the sweep there, the sweep is not needed, and we will say so.
Why not just use the recognizer's own confidence after multi-condition training? Multi-condition training is our default backend, so this is the softmax baseline, not an alternative to the proposal. The evidence that it under-performs is indirect. Varshney et al. find maximum probability weak out of domain, and noise-aware front-ends still give large gains over multi-condition training (28.6% relative on CENSREC-1 in Abe et al.), so multi-condition models leave noise structure unmodeled. Our held-out noise-type experiment measures this directly.
Why not just use test-time augmentation or dropout ensembles? This is the most serious alternative, because any $K$-pass disagreement score might capture most of the benefit. The matched-$K$ augmentation ablation exists to settle it. Our hypothesis is that random perturbations probe robustness in generic directions. The subtraction sweep instead probes the specific direction along which deployment front-ends actually alter the input. If augmentation matches the sweep, the idea reduces to ensemble uncertainty and should be framed that way.
Why not just replace spectral subtraction with a neural denoiser? We could sweep a neural enhancer's strength the same way, and we treat that as an extension. However, Multiple Confidence Gates For Joint Training Of SE And ASR reports that joint neural enhancement can generalize poorly to real noise. Its plain Conformer baseline reached 44.97% WER on the real-noise APY set, and competing joint enhancement models did worse than that plain baseline. Classical subtraction has one interpretable strength knob and no training data dependence, which keeps the sweep's meaning clean.
Decisive Experiments
The first check is in-distribution value. On YOHO+NOISEX-92 and on Aurora-4, does sweep agreement beat every baseline on area under the risk–coverage curve? Does SGR at $r^*\in\{1\%,2\%,5\%\}$ return higher certified coverage while realized risk stays under $r^*$ in at least $1-\delta$ of 100 random calibration splits?
The second check is shift. We calibrate on stationary noises (white, car) and test on non-stationary ones (babble, factory). We measure how often the guarantee is violated with pooled calibration versus SNR-stratified calibration. If stratification does not reduce violations, the guarantee claim under drift must be dropped and replaced with an empirical robustness claim.
The third check is mechanism. Within the highest SNR stratum, we ask whether the inputs where raw and subtracted decisions disagree are disproportionately the backend's errors. We also ask whether this effect shrinks when the backend is trained with subtraction augmentation. If it does, sweep disagreement on the raw-trained backend mostly reflects domain mismatch, and the honest recommendation becomes the augmented-backend variant.
The fourth check uses the consensus prediction $\hat{y}^{*}$ alone at full coverage. This shows whether the sweep also resolves the "subtract or not" dilemma as a router. The comparison is against both raw-only and fixed-subtraction systems.
References
- Assessment of single-channel speech enhancement techniques for speaker identification under mismatched conditions
- Impact of noise reduction and spectrum estimation on noise robust speaker identification
- Selective Classification for Deep Neural Networks
- Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models
- The synergy between bounded-distance HMM and spectral subtraction for robust speech recognition
- Missing feature theory with soft spectral subtraction for speaker verification
- An Efficient Feature Selection Method for Speaker Recognition
- Robust voice activity detection based on the entropy of noise-suppressed spectrum
- Robust speech recognition using DNN-HMM acoustic model combining noise-aware training with spectral subtraction
- Speech-enhanced and Noise-aware Networks for Robust Speech Recognition
- Multiple Confidence Gates For Joint Training Of SE And ASR
- Towards Improving Selective Prediction Ability of NLP Systems