Teaching Speaker Adapters to Ignore Enhancement Artifacts

Personalizing a speech recognizer to one user usually means fine-tuning a small set of extra parameters on a minute or so of that user's speech. Every published method for this assumes the minute is clean. In practice it is recorded in a kitchen or a car. The natural fix is to run it through a speech enhancer, a neural network that suppresses background noise. That fix has a known flaw: enhancers do not just remove noise, they add their own synthetic distortions, or "artifacts." Careful decomposition studies show these artifacts, not leftover noise, are what actually hurt recognizers. We propose an adaptation procedure for this setting. It builds several differently enhanced versions of the same noisy enrollment clip. It weights each version by a measure of how much artifact it contains, a measure that needs no clean reference. It then trains the speaker adapter to behave the same across versions. The intended effect is that the adapter learns only what the versions share, which is the speaker, rather than the quirks of any one enhancer.

The opportunity follows from one asymmetry in the literature. How bad are artifacts? splits enhancement error into residual noise and an artifact term that lies outside the space spanned by speech and noise. Shrinking the artifact term cuts word error rate (WER) sharply, while shrinking residual noise barely matters. A follow-up study finds the same ordering for human listeners. If this holds for adaptation, a few-shot adapter should see enrollment audio with artifacts minimized and moderate noise left in. Enhancers aim for the opposite. Our second point is that a small adapter fit on seconds of data may absorb a single enhancer's artifact signature as if it were part of the speaker's voice. We test that directly.

Core Claim

When enrollment audio is noisy, low-resource speaker adaptation of an ASR model fails mainly because the adapter fits enhancer-specific artifacts and environment-specific noise, not because the speaker information is missing. Two ingredients should recover most of the gap to clean-enrollment adaptation. The first is training across multiple views of each enrollment clip that differ in artifact content. The second is down-weighting views and segments by a reference-free artifact estimate. Both should hold under noise types and enhancers never seen in training.

This differs from the closest work in specific ways:

  • Clean-enrollment adaptation. KL-regularized speaker-dependent enhancement adapts the top layers from 10 seconds of clean speech. Personality-memory Gated Adaptation adapts parallel adapters from 10 minutes of clean speech. Both supply our adaptation machinery; neither handles corrupted enrollment.
  • Speaker-independent artifact robustness. FAT-HuBERT makes a self-supervised model invariant to a pool of enhancers. Distortion-independent acoustic modeling trains on enhanced speech. Both work at full training scale and are speaker-independent. We borrow the "pool of enhancers" idea but apply it at adaptation time, on seconds of one speaker's data, with the backbone frozen.
  • Noisy-enrollment voice cloning. Domain-adversarial voice cloning does handle noisy enrollment for text-to-speech, but with a binary clean-versus-noisy classifier that cannot tell artifacts from noise.
  • Enrollment augmentation for extraction. Self-estimated enrollment augmentation feeds artifact-laden outputs back as enrollment for target-speaker extraction. It treats them as a black box and never measures artifacts.

Research Program

Backbone and adapter. The base model is a self-supervised encoder fine-tuned for recognition. We use both a standard HuBERT model and a FAT-HuBERT-style distortion-robust variant, so that any gain we report survives the strongest robust backbone. The adapter is a set of parallel bottleneck modules beside frozen layers, as in Gu et al.. A conditional layer-norm variant in the spirit of AdaSpeech checks that conclusions do not depend on adapter type. Enrollment budgets are 10 s, 30 s, 60 s and 5 min, with transcripts. A pseudo-label variant covers the unlabeled case.

View construction. For each noisy enrollment utterance $y$, we make views $x_v$ from three ingredients:

  • the raw observation itself;
  • outputs of several enhancers (Conv-TasNet, DCUNet and a band-split model);
  • observation-adding mixtures $\hat s + \alpha y$, the recipe from Sakai et al. that trades artifact for noise, at several values of $\alpha$.

The views share the speaker and the words. They differ in how much and which kind of noise and artifact they carry.

Reference-free artifact proxy. The decomposition used by Sakai et al. needs the clean signal, so it cannot run at deployment. We approximate it as follows. In each frequency band and each block of about 100 ms, we fit the enhanced signal as a short linear filter applied to the noisy observation. We take the fraction of enhanced-signal energy this fit cannot explain as the artifact score $a_v$.

The rationale is that a smooth, slowly varying gain on the observation looks like noise suppression, which is benign by the Sakai results. Musical noise, phase errors and within-band spectral reshaping cannot be explained this way and look like artifacts. The proxy will partly confuse very aggressive suppression with artifacts. That is why its first test is correlation with the oracle artifact component on simulated data, where the clean reference is known.

Adaptation objective. With adapter parameters $\theta$, transcript $z$, and $p_\theta(\cdot \mid x_v)$ the adapted model's frame-level output distribution on view $v$, we minimize

$$ \mathcal{L}(\theta) = \sum_v \omega_v \, \mathcal{L}_{\text{CTC}}(x_v, z; \theta) + \lambda \sum_v \mathrm{KL}\big(\bar p \,\|\, p_\theta(\cdot \mid x_v)\big) + \beta \, \mathrm{KL}\big(p_0 \,\|\, p_\theta\big). $$

The terms are defined as follows:

  • $\omega_v \propto \exp(-a_v/\tau)$ down-weights views and segments that carry heavy artifacts, with $\tau$ a temperature.
  • $\bar p$ is the stop-gradient average of the outputs across views.
  • The middle term enforces cross-view agreement, an adaptation-time analogue of the consistency idea in Mixup-Breakdown training, which uses consistency to cope with unseen interference.
  • The last term tethers the adapted model to the unadapted model's output $p_0$, as Chai et al. do, to limit forgetting.

Diagnostic for what the adapter learned. We train an adapter on speaker A's enrollment processed by enhancer E. We then apply it to a different speaker B whose test audio is also processed by E, and compare against B's audio processed by a different enhancer E′.

  • A large gain on same-enhancer, different-speaker audio would show the adapter learned the enhancer, not the speaker.
  • An analogous swap of noise type separates noise-fitting from speaker-fitting.

This diagnostic tests the premise of the whole proposal, independently of whether our method wins.

Data and Metrics

Simulated setting with oracle decomposition. Each LibriSpeech test-clean and test-other speaker has several minutes of speech. We use the first 10 s to 5 min as enrollment, mixed with noise, and the remainder as that speaker's test set. Test utterances are scored clean, and also mixed with WHAM! noise at 0–5, 5–10 and 10–20 dB, the conditions in FAT-HuBERT.

Unseen-noise setting. Enrollment and test noises come from NOISEX-92, which none of the enhancers saw during training, following the unseen-noise protocol of Chai et al.. WSJ0 speakers serve as a second speaker pool.

Unseen-enhancer setting. MambAttention (Kühne et al.) is held out of the view pool and used only at test time. This checks generalization to a new enhancer, analogous to the unseen front-ends in FAT-HuBERT.

Real setting. The CHiME-4 single-channel real development and evaluation sets are split per speaker into enrollment and test portions. This gives real bus, cafe, pedestrian and street noise with genuinely noisy enrollment.

Metrics. We report per-speaker WER with confidence intervals over speakers. We also report WER on held-out other speakers' data to measure forgetting, Spearman correlation between the artifact proxy and oracle artifact energy, and the cross-speaker and cross-enhancer transfer matrix described above.

Baselines

Naive reference points. The do-nothing baseline is the unadapted backbone. On the robust backbone, the reference numbers are those reported by FAT-HuBERT: 5.7% WER on CHiME-4 single-channel real evaluation, and 32.7% WER on LibriSpeech test-other mixed with WHAM! at 0–5 dB. We will reproduce these before adapting; our per-speaker splits differ, so the comparison is against our own reproduction. The ceiling is adaptation on the clean enrollment audio with the same adapter and budget. That defines the gap we aim to close.

Adapting on raw noisy audio. The adapter is fit directly on raw noisy enrollment. Sakai's finding that residual noise is benign makes this a serious competitor, especially on the robust backbone.

Adapting on enhanced audio. The adapter is fit on single-enhancer output, which is the off-the-shelf pre-filter pipeline that Cong et al. found harmful for cloning.

Best-tuned observation adding. We choose the single $\alpha$ that minimizes WER on development speakers. This is the strongest simple competitor. Sakai et al. report roughly 20% relative WER improvement on CHiME-3 real test from observation adding over raw enhanced speech. No reported result exists on Lacuna for observation adding used to prepare adaptation data.

Pooled multi-condition adaptation. The adapter is trained on the union of noisy and enhanced views with no weighting and no consistency term. This mirrors the "noisy plus enhanced" training that gave about 40% relative improvement in Kataria et al. for speaker embeddings.

Strongest clean-enrollment method. Gated adapters as in Gu et al. are fit on enhanced enrollment. Gu et al. report a 21.62% relative CER reduction from 10 minutes of clean dialect audio. No reported result exists on Lacuna for any speaker-adapted ASR method trained on noisy enrollment, which is the gap itself.

Ablations. These isolate the two key ingredients: views with consistency but uniform weights, weights without consistency, and oracle artifact weights computed from the clean reference in simulation.

Fairness rules. Every method gets the same adapter parameter count, the same number of update steps, and the same development-speaker tuning budget. That budget is a fixed grid over learning rate, plus $\alpha$, $\lambda$, $\beta$ and $\tau$ where applicable, never tuned on test speakers.

What counts as winning. The proposal succeeds if it beats best-tuned observation adding and pooled multi-condition adaptation, with non-overlapping speaker-level confidence intervals, under unseen NOISEX-92 noise and the held-out enhancer. It must also close at least half of the noisy-to-clean enrollment gap at 30–60 s, with forgetting no worse than the KL-tethered baseline.

Why not just…?

Why not just apply observation adding to the enrollment audio and adapt as usual? This is the most plausible competitor, and our method contains it as one view family. Sakai et al. show that one global $\alpha$ is a blunt trade-off, since large values reintroduce enough noise to hurt intelligibility. The best $\alpha$ should also vary across utterances and segments as noise level and enhancer behavior change. Our weighting can follow that variation; a fixed $\alpha$ cannot. If tuned observation adding matches our full method, the honest conclusion is that a per-utterance $\alpha$ chosen by the artifact proxy is the whole contribution. The ablation without consistency, run with only observation-adding views, measures exactly this.

Why not just adapt on the raw noisy audio, since residual noise is benign? The Sakai results concern recognition of fixed inputs, not fitting parameters. Few-shot adapters can absorb a consistent background, which is the failure reported for noisy voice cloning by Cong et al.. The noise-swap row of the transfer diagnostic shows directly whether raw-noisy adapters encode the noise. The comparison on the FAT-HuBERT backbone shows whether a robust enough backbone makes the problem disappear. If it does, that is a publishable negative result, and we would report it as such.

Why not just pool noisy and enhanced audio as ordinary augmentation? Pooling teaches the adapter to handle every view, but nothing stops it from fitting each view's artifacts separately. The consistency term demands agreement across views, which removes view-specific components. Kataria et al. show pooling works for training embedding extractors at scale. Whether it suffices with seconds of data is open, and the uniform-weight, no-consistency ablation answers it.

Why not just use domain-adversarial training with a clean/noisy classifier? The single binary label in Cong et al. lumps artifacts and noise together. Artifacts are correlated with the speech itself, so pushing representations away from "enhanced-looking" input risks erasing speaker cues along with them. An adversarial classifier also needs more than seconds of data to train stably. We include it on the 5-minute budget only, where it is feasible.

Decisive Checks

Three results decide the proposal:

  1. Proxy validity. If the reference-free artifact proxy correlates weakly with oracle artifact energy on simulated LibriSpeech/NOISEX mixtures, the weighting has no basis. In that case the oracle-weight ablation tells us whether a better estimator is worth pursuing.
  2. Premise. The cross-speaker, same-enhancer transfer diagnostic must show that adapters fit on singly enhanced enrollment carry a measurable enhancer signature. If they do not, the premise that adapters absorb artifacts is wrong, and any gains must come from elsewhere.
  3. Head-to-head. Under unseen noise and the held-out enhancer, the full method must beat best-tuned observation adding and pooled adaptation on CHiME-4 real and on LibriSpeech-based splits, across all four enrollment budgets. If it wins only at 5 minutes, the method is not a low-resource method, and we would say so.