Distill the Noise, Not the Speech: Compressing Robust Enhancers with Teacher-Harvested Real-World Noise

Speech enhancement models remove background noise from a recording of someone talking. Small ones that run on hearing aids and phones tend to fail on noise they never heard in training. The usual fix is knowledge distillation: a large, robust "teacher" model trains a tiny "student" by having the student copy the teacher's outputs. On unseen noise this copying is risky, because the teacher's cleaned speech carries its own artifacts, and the student learns those too. We propose to use the teacher differently. Run it on unlabeled real-world noisy recordings and keep only what it removes, its estimate of the noise. Then mix that harvested noise into clean speech for which we know the true answer. The student learns from real, never-before-seen noise, but its target is genuine clean speech rather than the teacher's guess.

In two sentences: the teacher's job is to mine realistic noise from audio nobody labeled, not to supply the student's targets. Speech targets come only from clean recordings, so the student is not capped by the teacher's speech errors and can, in principle, beat the teacher out of domain while being an order of magnitude smaller and strictly causal.

Core Claim

Let $y$ be an unlabeled in-the-wild noisy recording and $T(y)$ the teacher's enhanced speech estimate. The residual $\hat n = y - T(y)$ is an estimate of the real noise. It is imperfect, but it is drawn from the deployment distribution. For a clean utterance $s$ from a paired corpus and a random gain $g$, we form a training pair

$$ x = s + g\,\hat n, \qquad \text{target} = s, $$

and train the student $S$ on $\ell(S(x), s)$ alongside ordinary supervised data. Any error the teacher makes shows up only as slightly wrong noise. It never shows up in the target. Speech that leaks into $\hat n$ becomes speech-like interference the student learns to suppress, which is a genuine side effect we measure rather than assume away. The claim is that, at a fixed student size, this "noise harvesting" improves out-of-domain quality more than distilling the teacher's outputs or features on the same unlabeled audio.

Relation to Prior Work

The motivation comes from two observations. First, copying the teacher's regression outputs is brittle. LaMi-KD reports that output-only distillation can degrade speech enhancement because teacher outputs contain artifacts, and it switches to mixup over intermediate features. ROSE-CD finds one-step students overfit the teacher's sampling trajectory and amplify its errors out of domain. Correlation-metric distillation shows that compressed HuBERT students lose noise robustness under plain distance-matching. Second, the compressed enhancers themselves are tested almost only in domain. RT-SEMamba distills an 8-layer causal Mamba (a recurrent state-space sequence model) into a 1-layer, 1.05M-parameter, 25 ms-latency student. DFKD splits spectra into adaptive bands for distillation. Both rely only on paired synthetic data, and RT-SEMamba evaluates only on VoiceBank+DEMAND. All four change how the student matches the teacher. None changes what the teacher is trusted for, and none uses unlabeled real recordings.

The closest method is Mixup-Breakdown Training. It has a moving-average teacher split unlabeled out-of-domain mixtures into sources, remixes them, and asks the student to reproduce the teacher's split. It recovers separation from 1.86 to 13.52 SI-SNRi on environmental noise. We differ in two ways. MBT's targets are entirely the teacher's estimates, whereas ours pair a teacher-estimated noise with a true clean target. MBT's teacher and student are architectural clones, whereas we cross a large non-causal teacher into a tiny causal student. For the teacher we use MambAttention, whose attention-before-Mamba design is reported to generalize across DNS 2020 and EARS-WHAM_v2 but is non-causal and quadratic in sequence length. It supplies robustness we cannot deploy directly. Two lines of work constrain the design. The MMI-SPLICE result, where larger discriminative models over-train on seen noise, warns that a bigger teacher is not automatically a better noise miner. Whisper-style scaling, which Park et al. note does not cover extreme conditions, is the brute-force alternative we must beat on the noise side.

Research Program

We first train the MambAttention teacher on paired data, either VoiceBank+DEMAND or the DNS 2020 training mixtures, and build an RT-SEMamba-style 1-layer causal student. We then assemble an unlabeled pool of real noisy recordings that is disjoint from every test set. The teacher processes this pool, and we keep residuals $\hat n$ along with a per-segment reliability score. The score comes from the teacher's agreement with itself under small input perturbations, or from a non-intrusive quality predictor such as NORESQA-MOS. It is used to down-weight residuals likely to contain leaked speech. The student then trains on three mixed streams: original paired data, harvested-noise pairs, and RT-SEMamba's progressive output and feature distillation on the paired data only.

The analysis has two parts. On synthetic mixtures where the true noise is known, we measure how the residual differs from the true noise, specifically how much speech energy leaks into it, as a function of SNR and noise type. We then relate that leakage to downstream student quality, which tells us when harvesting helps and when it teaches the student to remove speech. We also test whether the student can surpass the teacher out of domain. This is a falsifiable consequence of the claim: if the student only inherits the teacher's speech errors, it cannot.

Baselines

The naive reference is the same 1-layer causal student trained from scratch on paired data, the RT-SEMamba scratch condition. It is reported at 3.06 PESQ on VoiceBank+DEMAND, against 3.18 for the distilled student and 3.32 for the 8-layer teacher (RT-SEMamba). We re-run it with our pipeline rather than relying on the reported number. The strongest supervised distillation competitor is RT-SEMamba's progressive output-plus-feature distillation with our MambAttention teacher swapped in. It answers whether a better teacher alone is enough. We will also run LaMi-KD latent mixup on the same student, since it is the documented fix for regression distillation.

The strongest competitor on the actual question, using unlabeled audio, is output distillation on the unlabeled pool: the student matches $T(y)$ on the real recordings. It sees exactly the same audio and teacher as our method and differs only in what serves as the target. The second is MBT applied to the small student, with its own moving-average copy as teacher, on the same pool. No reported result was found on Lacuna for MBT on speech enhancement or on DNS 2020; its reported numbers are separation results on WSJ0-noise and WSJ0-music. Two ablations isolate the key ingredient. One harvests noise with the student's own moving-average copy instead of the large teacher, testing whether teacher quality matters. The other harvests noise only from speech-absent segments found by a voice activity detector, with no teacher at all. Every method gets the same unlabeled pool, the same student architecture, the same number of training steps, and the same hyperparameter search budget, with selection on a held-out in-domain validation set so that test-domain data never informs tuning. Beating a baseline means a consistent gain on all three out-of-domain test sets, outside seed variance over three seeds, without losing more than 0.05 PESQ in domain.

Why not just…?

Why not just distill the teacher's outputs on the unlabeled recordings? This is the most direct threat, and it is run as a baseline on identical data. The evidence against it is indirect but specific. LaMi-KD and ROSE-CD both report that students inherit teacher regression errors, and those errors should be largest exactly on unseen noise. If output distillation matches harvesting, our core claim is false and we will report that.

Why not just harvest noise from silent stretches with a voice activity detector? This could genuinely work for stationary noise, and it needs no teacher. It cannot capture noise that occurs only while someone talks, such as typing during a call or nonstationary noise overlapping speech. It also fails where the detector itself breaks, at low SNR and in babble. We split the analysis by noise stationarity. If the detector baseline matches us on nonstationary noise as well, the teacher adds nothing and the contribution reduces to data collection.

Why not just train the small student on a bigger noise corpus? The DNS 2020 training set already pools 65,000 noise clips. Park et al. and the MMI-SPLICE capacity results suggest that coverage alone does not reach specialized conditions, and a 1M-parameter student has little capacity to absorb breadth. We test this directly by training a student on the full DNS noise set and then checking whether harvesting from a different real pool still helps it.

Why not just apply MBT to the tiny model? MBT shows that self-consistency rescues separation out of domain. A 1-layer student is a weak breakdown source, though, and its errors feed back into its own targets. The moving-average-teacher ablation measures exactly this gap.

Proposed Experiments

We run two training regimes, VoiceBank+DEMAND and DNS 2020 synthetic, and evaluate out of domain on three sets. The first is TIMIT+NOISE92, following ROSE-CD's protocol. The second is VB-DemandEx, for low SNR and wider noise variety. The third is a held-out split of DNS 2020 real recordings, disjoint from the unlabeled pool. We also report the in-domain VoiceBank+DEMAND test set. Metrics are PESQ, SI-SDR, and ESTOI where clean references exist, and NORESQA-MOS plus a small listening test on the real recordings. No reported result was found on Lacuna for numeric DNS 2020 or VB-DemandEx scores of any method in the file, so those baselines are produced in-house. The ROSE-CD reported figure of 3.99 PESQ on VoiceBank+DEMAND, from a one-step diffusion model trained with a PESQ loss and with a documented SI-SDR penalty, is quoted for context only, since it is neither causal nor the same model size.

Three checks would decide the question. Harvesting must beat output distillation on the identical unlabeled pool on all three out-of-domain sets. The large-teacher residual must beat both the student's own residual and voice-detector harvesting on nonstationary noise. And the student should match or exceed the teacher on at least one out-of-domain set at about 1M parameters and 25 ms latency. The leakage study supplies the failure map: if gains vanish below 0 dB SNR, where residuals carry the most speech, that bounds where the method can be trusted.