Speech recognizers trained to cope with hard audio often get worse on easy audio. Adversarially trained image classifiers show the same pattern: they lose accuracy on unperturbed inputs. These two literatures rarely talk to each other. We propose to treat them as one phenomenon measured along a single axis, namely how hard the training perturbation is. At one end sits mild random background noise. At the other sit worst-case adversarial perturbations, small changes to the waveform crafted to maximize recognition error. We will fine-tune one speech foundation model, a large network pre-trained on unlabeled audio such as HuBERT, along this whole axis. At each point we will measure the "clean tax", the increase in word error rate on pristine speech relative to a clean-only model. We will also test which mechanisms from adversarial machine learning explain the tax and close it.
The core bet is that the tax tracks how far training perturbations push the model's internal representations away from where clean speech lives, and that the type of perturbation matters less. If this holds, two things follow. First, the fixes the adversarial literature invented, chiefly anchoring perturbed representations to a fixed copy of the clean ones, should help a speech model exactly where its representations drift, and nowhere else. Second, a cheap displacement measurement taken before or early in training should predict whether a given augmentation recipe will cost clean accuracy.
Core Claim
The research file suggests the clean penalty in speech is not universal. pMCT shows that patched multi-condition training on LibriSpeech gets 2.41% clean WER against 2.42% for standard multi-condition training. Braun et al. find Gaussian per-epoch noise mixing gives 13.6% clean WER on WSJ versus 13.8% for clean-only training. Mild random noise therefore seems nearly free. The penalty appears as training gets harder:
- Braun et al.'s hard-to-easy curriculum starting at 0 dB pays a clean cost.
- Enhancement front-ends that inject artifacts into clean audio cause large drops (FAT-HuBERT; Yang et al.).
- In vision, worst-case adversarial training is where the tax is largest (TRADES).
We claim this is one curve. The clean tax is a monotone function of layer-wise representation displacement between clean and perturbed inputs. The asymmetric clean-anchored objective from AR-AT reduces the tax in proportion to that displacement. It should give roughly nothing at mild SNR and the most under adversarial perturbation. Nobody in the file has measured the speech clean tax under worst-case perturbations, or placed noise, enhancement artifacts and adversarial perturbations on a common severity scale.
What We Take From Each Line of Work
The adversarial literature supplies candidate mechanisms. AR-AT attributes clean degradation in vision to two causes:
- Gradient conflict: the invariance term pulls clean features toward perturbed ones, against the classification gradient.
- Mixed normalization statistics: batch normalization averages clean and adversarial batches.
AR-AT fixes these with a stop-gradient on the clean branch, a predictor head, and split batch normalization. It reports 90.89% clean and 50.77% AutoAttack accuracy on CIFAR-10, against TRADES' 84.33% clean. CURE attributes the tax to global updates overwriting weights that natural data needs. It freezes early layers and selects which weights to update with a gradient-importance score, reaching 87.05% natural and 55.25% C&W accuracy. We reuse both as methods, but on a different architecture and loss: Transformer encoders trained with connectionist temporal classification (CTC), an alignment-free sequence loss.
One mechanism needs correcting before it can be ported. The research file suggests a "Split-LayerNorm" as the speech analogue of split batch normalization. Layer normalization, however, computes statistics per token, so no clean–noisy batch mixing occurs at test time. The only shared quantities are the learned gain and bias. We therefore treat condition-specific gain and bias as an ablation. We predict it matters far less than it did for batch normalization, and a positive result would itself be surprising.
The speech literature supplies the regime and the closest competitors:
- HuBERT-VIC aligns a noisy student to a frozen clean teacher with variance and covariance terms. It reports a 23.3% relative normalized-WER gain on test-clean over plain noisy pre-training.
- deHuBERT applies a symmetric Barlow-Twins-style loss across two noisy views, taking noisy-LibriSpeech WER from 14.45% to 13.52%.
- FAT-HuBERT randomizes the enhancement front-end, cutting CHiME-4 real WER from 8.6% to 5.7%.
All three are evaluated only on random environmental noise or enhancement artifacts. None reports adversarial audio, and none relates its clean behavior to perturbation severity. HuBERT-VIC's frozen teacher already acts as an asymmetric anchor; AR-AT itself argues that distillation works for this reason. So "port asymmetry to speech" is not new in itself. The new parts are the severity axis, the displacement predictor, and the decomposition of which mechanism matters where.
Research Program
Base model and data. We fix the backbone as HuBERT Base and fine-tune it with CTC on LibriSpeech 100h. This keeps every condition comparable and within academic compute.
Training perturbation families. We build families indexed by severity:
- MUSAN additive noise at 20, 10, 5 and 0 dB SNR.
- Room-impulse-response reverberation.
- Audio passed through a pool of enhancement networks, following FAT-HuBERT's Conv-TasNet and DCUNet setup.
- Projected-gradient (PGD) waveform perturbations against the CTC loss. These are computed with a few inner steps and set at budgets matched in signal-to-perturbation ratio to the noise levels, so that energy is held fixed while "worst-caseness" varies.
Displacement measure. For every family, before fine-tuning, we compute layer-wise displacement between clean and perturbed representations in the pre-trained model. We use one minus linear centered kernel alignment, plus mean cosine distance.
Objectives. Each family is crossed with five training objectives:
- Plain mixed training.
- Symmetric invariance in the deHuBERT style.
- Frozen-teacher alignment in the HuBERT-VIC style.
- Self-anchored asymmetric alignment. The same network processes the clean utterance, its output is detached, and a small predictor head maps perturbed-branch features onto it. This is AR-AT's recipe without a second model in memory.
- Objective 4 plus CURE-style conservation, freezing the convolutional feature encoder and lower Transformer blocks and updating weights chosen by gradient importance.
Diagnostics. During training we log the cosine between the CTC gradient and the invariance gradient at each layer. This tests whether gradient conflict, AR-AT's main explanation, actually appears in speech Transformers and grows with severity.
Baselines
Do-nothing reference. HuBERT Base fine-tuned on clean LibriSpeech 100h. This defines zero clean tax and the robustness floor.
Plain multi-condition training. This uses the same perturbations without any auxiliary loss. It is the honest default, and pMCT's evidence suggests it already pays almost no tax at mild SNR. We include pMCT's patching variant as a stronger version, since pMCT reports 2.41% clean WER and 9.89% on VOiCES against 12.86% for standard multi-condition training.
Strongest existing method for clean preservation under noise. This is HuBERT-VIC-style frozen-teacher alignment. Its reported result is relative: 23.3% N-WER improvement on test-clean and 13.2% on test-other. We re-implement it at fine-tuning scale with the same data, steps, learning-rate sweep (five values) and loss-weight sweep (three values) as our method. It also gets the same GPU budget, which charges it for the teacher's memory.
Strongest robustness reference on real noise. This is FAT-HuBERT, at 5.7% CHiME-4 real WER. We train it on our enhancement family and report it on CHiME-4.
Adversarial training references. For the adversarial family we use PGD adversarial training and a TRADES-style objective. We found no reported result on Lacuna for adversarially trained HuBERT or for PGD-robust WER on LibriSpeech, so these numbers will be produced here under the identical budget.
Ablations. - The predictor head and stop-gradient are removed one at a time, which isolates the anchoring ingredient. - Condition-specific layer-norm gain and bias are added, which tests the normalization hypothesis.
What counts as beating the baselines. The method must beat HuBERT-VIC on the clean tax at matched robust WER, with confidence intervals from three seeds, and at lower peak memory. It must also cut the clean tax of adversarial training by at least half while keeping PGD-robust WER within one absolute point.
Why not just…?
Why not just use a frozen clean teacher, as HuBERT-VIC does? This is the real threat. AR-AT's own analysis says frozen-teacher distillation succeeds because it is an implicit stop-gradient, so the teacher may perform as well as our self-anchored variant. If it matches us at every severity, we will report that. The remaining contribution would then be the severity curve and the displacement predictor, plus a memory saving that we will measure rather than assume. The experiment settles this directly, because objectives 3 and 4 run on identical data and budgets.
Why not just do multi-condition training, perhaps patched? At mild SNR it probably suffices: pMCT and Braun et al. both show essentially no clean penalty. Our prediction is that this stays true only at low displacement, and the tax grows under enhancement artifacts and adversarial perturbations. If plain multi-condition training also shows no tax under PGD training, the central claim is falsified.
Why not just detect noise and route clean audio around the robust model? A gate cannot help with adversarial perturbations, which are designed to look clean, by construction. For environmental noise, the file's analogy to perplexity filters is suggestive: those filters show roughly 10% false rejections of benign inputs in Baseline Defenses. That is evidence from text, though, not speech. We will train an SNR-threshold router between the clean-only and multi-condition models as an additional baseline and report its WER at intermediate SNRs, where we expect misrouting to hurt.
Why not just use TRADES-style adversarial training? In vision it pays the largest clean cost (84.33% versus AR-AT's 90.89%). Whether that gap carries over to CTC speech models is exactly one point on our curve, so we run it rather than assume it.
Decisive Checks
The study succeeds or fails on four checks.
- Across all perturbation families, the pre-training displacement measure should rank-correlate with the clean tax of plain multi-condition training. We will report Spearman correlation across at least twelve family-severity cells. A correlation below 0.6 refutes the unifying claim.
- The tax reduction from asymmetric anchoring should grow with displacement: near zero for 20 dB noise, largest for PGD. A flat benefit would mean anchoring is a generic regularizer rather than a fix for drift.
- Gradient conflict should appear in speech Transformers, with negative CTC–invariance cosine concentrated in upper layers and increasing with severity. Its absence would mean AR-AT's mechanism does not transfer, even if the method still helps.
- Robust WER under PGD must be measured with attacks that differentiate through the final trained model. Because stop-gradient and the predictor are used only during training, the deployed network is an ordinary encoder, and robustness cannot come from obscured gradients.
Evaluation covers LibriSpeech test-clean and test-other, WHAM!-mixed test-other at 0–5 dB, CHiME-4 real, VOiCES, and TED-LIUM 3 with FSD50K office noise. This tests whether the conclusions hold on noise not seen in training.
References
- pMCT: Patched Multi-Condition Training for Robust Speech Recognition
- A curriculum learning method for improved noise robustness in automatic speech recognition
- FAT-HuBERT: Front-End Adaptive Training of Hidden-Unit BERT For Distortion-Invariant Robust Speech Recognition
- FiLM Conditioning with Enhanced Feature to the Transformer-based End-to-End Noisy Speech Recognition
- Theoretically Principled Trade-off between Robustness and Accuracy
- Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off
- Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Training
- HuBERT-VIC: Improving Noise-Robust Automatic Speech Recognition of Speech Foundation Model via Variance-Invariance-Covariance Regularization
- De'hubert: Disentangling Noise in a Self-Supervised Model for Robust Speech Recognition
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models