Adapting a Multi-Task Denoising Recognizer to Unseen Noise at Test Time

Speech recognizers that are trained to clean up noisy audio tend to break when they meet a kind of noise they never heard in training. The clearest documented case is a multi-task autoencoder, a network that learns to predict both the clean speech and the background noise from a noisy recording. It cut word errors on familiar noise types but lost to a hand-engineered front-end on unfamiliar ones, because its noise estimates became biased Multi-Task Autoencoder for Noise-Robust Speech Recognition. We propose to keep the multi-task structure and make it useful exactly where it currently fails. A convolutional network with one shared encoder and three output heads (clean speech, background noise, and speech-recognition targets) is trained jointly. At test time, on each unlabeled utterance, a small input layer of the network is briefly updated using two label-free checks that the clean-speech and noise heads make possible.

The first check is corruption consistency. If the network is shown the same utterance with extra noise added, noise that resembles what is already present, its clean-speech estimate should not change. The second check is additivity. The predicted speech and the predicted noise should add back up to the observed input. Neither check requires a transcript, and both depend on the denoising and noise-prediction tasks being trained alongside recognition. That joint training is what links them to recognition accuracy. The question is whether these checks give a gradient, computed on unfamiliar noise, that points toward fewer recognition errors, and whether we can measure ahead of time when they will.

Core Claim

A jointly trained convolutional denoising autoencoder, meaning an encoder–decoder network with convolutional layers that maps corrupted input back to clean input, does more than act as a fixed front-end. Its auxiliary heads supply a self-supervised objective that can be computed on any test utterance. When a shallow normalization front-end is adapted per utterance to minimize corruption-consistency and additivity losses, word error rate should fall on unseen-noise conditions, specifically Aurora 2 Test Set B and the simulated-to-real shift in CHiME 3. Accuracy on matched noise (Aurora 2 Test Set A) should not degrade. We also claim that the benefit is predictable. It should track the cosine similarity between the gradient of the test-time loss and the gradient of the recognition loss on the adapted parameters, measured on a small labeled probe set. This turns a multi-task-learning diagnostic, gradient agreement between tasks, into a deployment-time tool.

Write the encoder as $E_{\phi,\theta}$, where $\phi$ is a shallow per-channel affine front-end and $\theta$ the rest. The clean-speech decoder is $D_s$ and the noise decoder is $D_n$. For a test input $x$ in the log-filterbank domain, let $\hat s = D_s(E(x))$ and $\hat n = D_n(E(x))$. The test-time objective is

$$\mathcal{L}_{\text{tta}}(\phi) = \big\| D_s(E(x \oplus \tau(\hat n))) - \mathrm{sg}[\hat s] \big\|^2 + \lambda \big\| \hat s \oplus \hat n - x \big\|^2 ,$$

where $\oplus$ adds signals in the linear power domain and returns to log, and $\tau$ is a random time shift and gain applied to the network's own noise estimate. The operator $\mathrm{sg}$ stops gradients, and $\lambda$ weights the additivity term. Only $\phi$ is updated, for a fixed small number of steps per utterance. The recognition head then decodes from the adapted encoder.

What Exists and What This Changes

Three lines of prior work motivate the proposal. Joint Visual Denoising and Classification using Deep Learning trained a Y-shaped network that both reconstructs and classifies corrupted digits. It reports a 12.7% error rate against 29.9% for a separately trained denoiser feeding a classifier. That is evidence that joint training preserves task-relevant detail, but the model used restricted Boltzmann machines on toy images and did nothing at test time. Multi-Task Autoencoder for Noise-Robust Speech Recognition added the noise-prediction ("deSpeeching") task with fully connected layers. It reached 5.52% average word error rate (WER) on Aurora 2 Test Set A against 6.39% for a plain denoising autoencoder, and 26.93% against 29.71% on CHiME 3 et05_real. It openly reports failure on unseen noise. Multi-Channel Speech Enhancement Using Time-Domain Convolutional Denoising Autoencoder shows that convolutional denoisers on audio are strong, but it trains only the noisy-to-clean mapping and has no recognition head. We reuse the convolutional denoiser design and the noise-prediction task. Our addition is a use for the noise head after training: it supplies the realistic extra corruption $\tau(\hat n)$ and the additivity anchor.

One paper constrains the design. Robust medical image segmentation by adapting neural networks for each test image adapts a shallow normalization network per test image, using a denoising autoencoder trained separately on segmentation masks. It found that adapting all layers of the segmenter destroys performance. We adopt its restriction to a shallow front-end and will test that finding directly. We differ from it in two ways. Our self-supervised signal comes from the input audio, not from a prior over outputs. And the auxiliary heads are trained jointly with the main task rather than offline.

Two multi-task optimization papers supply diagnostics and a competing training method rather than the core idea. RotoGrad: Gradient Homogenization in Multi-Task Learning balances task gradient magnitudes and rotates the shared representation to reduce directional conflict between tasks. We use its notion of inter-task gradient alignment as the predictor of test-time benefit, and we run it as a training-time baseline. DeMT addresses negative transfer with task-specific attention in dense vision. It is relevant only as evidence that structured sharing matters, and we do not adopt its architecture.

Outside this file, test-time training with self-supervised heads is a known idea in image classification. What is specific here is a speech-recognition setting in which the auxiliary signal comes from a jointly trained noise-prediction head, and a stated test for when the method should work.

Research Program

The model is a 2D convolutional encoder over log-filterbank frames, using dilated convolutions and no pooling in the time axis, following the morphology-preserving lesson of Removing Noise from Extracellular Neural Recordings Using Fully Convolutional Denoising Autoencoders. The encoder feeds a clean-speech decoder, a noise decoder, and a frame-level acoustic-model head trained with cross-entropy on forced alignments. Each utterance is then decoded with the standard Aurora 2 / CHiME 3 recognizer backend. Training minimizes a weighted sum of the three losses. The weights are tuned on a held-out matched-noise development set and kept fixed.

The program has three stages. First, we establish that a convolutional version of the multi-task autoencoder at least reproduces the original's matched-noise advantage, so the base model is not a straw man. Second, we add test-time adaptation and measure it on seen and unseen noise. Third, we study when the adaptation helps. For each Test Set B noise type, we compute the cosine between $\nabla_\phi \mathcal{L}_{\text{tta}}$ and $\nabla_\phi$ of the recognition loss on a small labeled probe set (we will use roughly a hundred utterances). We then check whether this cosine ranks noise types by realized WER gain. If it does, the cosine becomes a cheap gate for whether to adapt at all. We will also test whether training with a RotoGrad-style alignment between the denoising and recognition gradients on $\phi$ raises the test-time cosine and therefore the gain. This is the one point where gradient homogenization enters the method itself.

Baselines

The naive reference point is a convolutional acoustic model with no denoising heads, trained on the Aurora 2 multi-condition set and decoded on its own. This is plain noise augmentation. It shares the encoder architecture and the training budget of our model. Our method beats it only if WER on Test Set B is lower at equal parameter count.

The strongest existing method on this exact question is the triangular multi-task autoencoder of Multi-Task Autoencoder for Noise-Robust Speech Recognition. It reports 5.52% average WER on Aurora 2 Test Set A (0–20 dB) and 26.93% on CHiME 3 et05_real. For Test Set B, no reported result was found on Lacuna. The paper states only qualitatively that the ETSI advanced front-end outperformed it there. We will reimplement it, confirm we recover numbers close to 5.52% on Set A, and run it on Set B with the same recognition backend. We will also run the ETSI advanced front-end ourselves, since it is the method reported to win on unseen noise. Beating both on Set B, while staying within the reimplemented baseline's Set A WER, is the bar.

A second strong competitor is a cascade, in the spirit of the decoupled pipeline that Joint Visual Denoising and Classification using Deep Learning compared against. A convolutional denoiser is trained alone, and the acoustic model is trained on its outputs. We run it with and without end-to-end fine-tuning through the denoiser. Crucially, we also run it with the same test-time objective applied to the denoiser's front-end. This arm isolates whether joint training adds anything beyond test-time adaptation itself.

The ablations that isolate the key ingredient are as follows. We replace $\tau(\hat n)$ with noise drawn from the training noise bank, which tests whether the network's own noise estimate matters. We drop the additivity term, setting $\lambda = 0$, which tests the collapse anchor that only the noise head provides. We remove the noise head entirely, using consistency only with bank noise. And we adapt all encoder layers rather than $\phi$, a direct test of the Karani et al. finding. Every adapted arm uses the same number of test-time steps and learning rate, chosen on a matched-noise development set and never on Test Set B.

Why not just…?

Why not just train on more kinds of noise? Heavier multi-condition augmentation is the cheapest fix and a real threat. Its weakness is structural: it helps only to the extent that the training noise covers the test noise, and Test Set B is built to violate that. The multi-condition baseline above, trained on the full multi-condition set, settles this. If it matches our method on Set B, the proposal fails on its central claim, and we will report that.

Why not just re-estimate normalization statistics on the test utterance? Recomputing per-utterance feature statistics, or the running statistics in normalization layers, adapts to new noise with no gradient steps. It plausibly captures much of the gain from a shallow affine front-end. We include it as a baseline applied to the same base model. If it matches gradient-based adaptation, the honest conclusion is that the multi-task heads add nothing at test time beyond what their training already provides. The ablation that drops both test-time losses but keeps statistic re-estimation measures exactly this.

Why not just use gradient homogenization during training and skip test-time adaptation? RotoGrad-style training reduces conflict between denoising and recognition on the training distribution RotoGrad: Gradient Homogenization in Multi-Task Learning. Nothing in that work addresses noise types absent from training, and it has not been evaluated with a reconstruction task paired with a recognition task. We run it as a training-only arm. If it closes the Set B gap on its own, test-time adaptation is unnecessary.

Why not just apply test-time adaptation to a cascade? This is the alternative most likely to work as well, because consistency adaptation of a standalone denoiser is perfectly well defined. The cascade-plus-adaptation arm answers it directly. Our hypothesis is that joint training makes the adaptation gradient agree better with the recognition gradient, which the cosine diagnostic measures in both systems. If the cascade arm matches the joint arm, we will reframe the contribution as test-time adaptation for denoisers rather than as a multi-task result.

Proposed Experiments and Decisive Checks

The primary benchmark is Aurora 2. We train on the multi-condition set, evaluate on Test Set A as the matched control and Test Set B as the unseen-noise target, and report average WER over 0–20 dB along with per-SNR and per-noise breakdowns. The secondary benchmark is CHiME 3. We train on tr05_simu and evaluate on dt05_real and et05_real, treating the simulated-to-real gap as distribution shift; the reference points there are the 26.93% and 29.71% WERs reported for the original multi-task autoencoder and denoising autoencoder. We use the first channel to stay comparable with the monaural multi-task autoencoder, and add a multi-channel variant only as a follow-up. Alongside WER, we report reconstruction error of $\hat s$ against clean references where these exist. We also report added latency per utterance, and compare against an equal-compute test-time augmentation ensemble so that extra inference cost is not mistaken for adaptation.

Four checks decide the outcome. The first is the main success criterion. Joint training with test-time adaptation must give at least 10% relative WER reduction on Aurora 2 Test Set B over the best of the multi-condition, reimplemented multi-task autoencoder, ETSI front-end, and statistic re-estimation baselines. This is a target we set, not a prior result, and the Set A WER must not exceed the non-adapted model's by more than its seed-to-seed spread. The second is that joint-plus-adaptation must beat cascade-plus-adaptation; otherwise the multi-task claim is withdrawn. The third is that the self-estimated noise ablation must outperform bank noise on Set B, and removing the additivity term must cause measurable collapse or degradation; otherwise the noise head is not the operative ingredient. The fourth is that the per-noise-type gradient cosine must rank-correlate with per-noise-type WER gain, so that adaptation can be switched on only where it helps. A null correlation would mean the gain cannot be predicted in advance, which limits deployment even if the average improves.