Noise-suppression networks have learned to remove background noise almost perfectly, yet they still damage the speech they are meant to save. In the ICASSP 2022 Deep Noise Suppression Challenge, no submitted system improved human-rated speech quality or word accuracy over the original noisy recordings (ICASSP 2022 DNS Challenge). This proposal tests one explanation for that failure and one remedy. The explanation is that a single network applies one kind of processing to every kind of damage. Noise, clipping, reverberation and band-limiting need different repairs, and an average repair over-suppresses speech. The remedy is a mixture of experts, meaning several small sub-networks plus a gate that picks which of them process each stretch of audio. The new element is what the gate looks at. Instead of hand-made acoustic features, it reads the internal features of a frozen speech-quality model. A recent study found that these features already sort recordings by impairment type without ever being given impairment labels (Impairments are Clustered in Latents of DNN-based Speech Quality Models).
The work therefore joins the two directions. It uses the evaluation side, which can recognize what is wrong with audio better than the enhancement side can, to decide which specialist does the repair. It also keeps the enhancer away from the known traps of training directly on a metric. The enhancer never maximizes a quality score. It only uses the quality model's view of the damage to choose experts. Success is judged by human ratings on the dimension that has resisted progress: speech distortion, not noise removal.
Core Claim
A sparse mixture of enhancement experts whose gate reads frozen speech-quality-model features will raise human-rated speech quality on the DNS blind test set above that of the unprocessed noisy input, while keeping background-noise suppression competitive. It will do this at the per-frame compute of a single small expert. The claim makes two separable bets, and the experiments are built to tell them apart. The first bet is about information: quality-model features describe the damage better than the cues earlier gates used. The second bet is about structure: sending different damage to different weights works better than feeding the same information into one shared network.
Why This Gap, and What Prior Work Contributes
Two speech-enhancement mixtures of experts motivate the architecture and also expose its weak point. Speech Enhancement using a Deep Mixture of Experts trained experts and a gate jointly on log-spectra, with Mel-cepstral features fed to the gate. The experts split on their own into voiced and unvoiced specialists and beat a single deep expert, but the paper reported only PESQ and saw diminishing returns beyond two experts. Sparse Mixture of Local Experts for Efficient Speech Enhancement trained specialists for each signal-to-noise level and each speaker gender, used a single-expert hard gate at test time, and showed that a 512x2 specialist ensemble beat a 1024x2 generalist. It also found that gate accuracy caps the whole system, at roughly 80% for noise level and 90% for gender. Both papers route on cues that are only loosely tied to how the audio sounds damaged. Neither measures speech distortion separately from noise suppression. We reuse their joint training and the temperature-sharpened softmax that lets a soft gate become hard at test time. We change what the gate reads and what is measured.
The routing signal comes from the evaluation literature. DNSMOS P.835 predicts human ratings of speech distortion (SIG), background noise (BAK) and overall quality (OVRL) without a clean reference. The impairment-clustering study shows that its improved variant, DNSMOS+, separates 16 impairment classes with 94% k-nearest-neighbor accuracy, against 86% for the original DNSMOS. On 50 real-world noise classes it reaches 54%, against 18% for Mel-cepstral features. That last comparison is the reason to expect a better gate than the one in Speech Enhancement using a Deep Mixture of Experts.
The metric-optimization line constrains the design. MetricGAN+ needs replay buffers and a mask floor to stop a generator from exploiting its learned metric surrogate. Improving speech synthesis for noisy environments shows that pushing an intelligibility index directly produces harsh audio. MetricGAN-OKD shows that a single generator chasing both quality and intelligibility surrogates receives conflicting gradients. It resolves them with several full generators, one per metric, coupled by distillation, but with no input-dependent gate. We take the negative lesson and do not backpropagate any quality score into the enhancer. MetricGAN-OKD is the closest prior work at this intersection, and it differs from our design on two axes. It specializes by metric rather than by impairment, and it duplicates whole models rather than routing frames to small experts.
Research Program
The system has three parts. The first is a frozen copy of DNSMOS+. On each causal analysis window of the noisy input it emits a latent vector $z_t$. The second is a gate that maps this vector to expert weights $g_t = \mathrm{softmax}(W z_t / \tau)$. Here $W$ is a learned projection, and $\tau$ is a temperature that is annealed toward a one-hot choice during training, following Sparse Mixture of Local Experts for Efficient Speech Enhancement. The third is a set of $K$ small causal mask-estimating experts plus one always-on shared expert. The shared expert follows the shared-plus-routed division of labor described in Decoding Knowledge Attribution in Mixture-of-Experts. That work found shared experts carry general structure while routed experts refine. Our hypothesis is that the shared path handles generic noise removal and the routed experts handle impairment-specific speech repair.
At test time only the shared expert and one routed expert run. Training uses only reference-based losses on synthetic mixtures where clean speech is known: a spectral reconstruction loss plus a separate speech-distortion term on the speech component. No training loss is computed by DNSMOS or by any other metric surrogate.
Two engineering risks are studied first rather than assumed away. The first is that DNSMOS+ was trained on clip-length audio, so its impairment clustering may weaken on the short causal windows that a real-time enhancer can afford. We will measure k-nearest-neighbor impairment accuracy on LibriAugmented as a function of window length. We will then pick the shortest window that keeps most of the reported clip-level accuracy. If none does, we fall back to routing at the segment level, which is still compatible with streaming. The second risk is router collapse onto one expert. We monitor this with expert-usage entropy and counter it with a standard load-balancing penalty.
The analysis also asks whether the experts are specialized in a meaningful way. We will compute the mutual information between expert assignment and the 16 LibriAugmented impairment labels. We will also check whether separate experts absorb compound impairments, such as noise plus clipping, which a shared network has to handle with one set of weights.
Baselines
The naive reference is the unprocessed noisy input, scored by human P.835 ratings and word accuracy. On the DNS blind set it is not a trivial bar for SIG and word accuracy, since ICASSP 2022 DNS Challenge reports that no submission beat it on either. Beating it means a statistically significant positive SIG difference across the crowdsourced test. For context on overall quality, the best system in INTERSPEECH 2021 DNS Challenge reached an overall MOS of about 3.78. That number came from a different test set and listener pool, so we cite it only as orientation and not as a target.
The strongest dense competitor is a causal mask-estimating network trained by us on the same DNS data, with the same losses and the same hyperparameter-search budget. It is run at two sizes: one matching the mixture's active per-frame compute, and one matching its total parameter count. The DNS challenge reports show that a small model can win, with 1.5M parameters for the Track 1 winner (ICASSP 2022 DNS Challenge). A well-tuned dense model is therefore a serious opponent and not a strawman.
The strongest existing mixture methods are our reimplementations of Speech Enhancement using a Deep Mixture of Experts, with a Mel-cepstral gate, and Sparse Mixture of Local Experts for Efficient Speech Enhancement, with a gate supervised on noise level. Both use our expert sizes and data. Neither paper's summary gives numbers we could compare against: no reported result found on Lacuna for either on DNS, VoiceBank-DEMAND or P.835. On VoiceBank-DEMAND, the strongest metric-trained reference is MetricGAN+, which reports a PESQ of 3.15. We also run MetricGAN-OKD at matched total parameters. Its paper summary gives no table values here, so no reported result found on Lacuna for its exact PESQ.
The key ablations form a two-by-two design that crosses gate input (Mel-cepstral features versus DNSMOS+ latents) with use of that input (sparse routing versus FiLM conditioning, where FiLM means feature-wise scaling and shifting of a dense network's activations). Two further ablations remove the shared expert and replace the learned gate with random assignment. Beating these ablations means the DNSMOS+-routed mixture outperforms both the Mel-routed mixture and the DNSMOS+-conditioned dense model on human SIG at matched active compute.
Why not just…?
Why not just scale a dense network? Sparse Mixture of Local Experts for Efficient Speech Enhancement found that specialists beat a generalist of double width. However, that comparison used noise-level specialists and signal-level losses, not human SIG. We do not assume the result carries over. Instead we test it directly with the dense model at matched total parameters. If that model matches our SIG, the routing argument fails and we will report that.
Why not just feed the quality-model latent into one dense network? This is the most serious threat. It is plausible that most of the gain comes from the information and not from routing. The FiLM arm of the factorial design answers this directly. We expect routing to help most on compound impairments, because there the needed operations disagree: declipping must fill in missing signal, while denoising must remove it. We will report results for each impairment on LibriAugmented. If FiLM conditioning matches routing everywhere, the honest conclusion is that quality-model conditioning is the contribution. In that case the mixture is justified only by its compute savings, and we will present it that way.
Why not just train against DNSMOS or metric surrogates, as in MetricGAN-style work? MetricGAN+ needed replay buffers and a mask floor to contain surrogate exploitation. MetricGAN-OKD documents conflicting gradients across metrics. Improving speech synthesis for noisy environments shows that direct metric pushing makes audio harsh. Even so, we include MetricGAN-OKD as a baseline, so that its human SIG and the gap between its DNSMOS and human scores can be read off directly next to ours.
Why not just run several specialists and keep the one DNSMOS likes best? Choosing the best of K outputs per clip is simple. It costs K times the compute, it cannot switch experts within a clip, and it selects for exactly whatever DNSMOS rewards, which invites the gaming the metric literature warns about. We run it at clip level as an upper-bound reference. We expect it to look strong on DNSMOS and weaker on human SIG. A large gap between the two would itself be evidence of the metric limitation.
Proposed Experiments
Training uses the ICASSP 2022 DNS data, which covers more than 3,200 talkers and about 62,000 noise clips. Primary evaluation is a crowdsourced ITU-T P.835 test on the DNS blind set, rating SIG, BAK and OVRL, together with word accuracy. Each system in the factorial design receives an equal number of listener ratings, and differences are tested at the clip level.
Mechanism analysis uses LibriAugmented, where clean references allow intrusive PESQ, STOI and composite scores for each of the 16 impairment classes. Comparability with the metric-trained literature uses VoiceBank-DEMAND, scored by PESQ, CSIG, CBAK, COVL and STOI against MetricGAN+'s 3.15 PESQ. Here we expect to trail on PESQ, because we do not optimize it. The point of this comparison is to show where PESQ and human SIG disagree.
The decisive checks are four. First, SIG relative to noisy input must be positive on the blind set, the threshold no 2022 submission crossed. Second, SIG must be higher than both the FiLM-dense and Mel-routed arms at matched active compute. Third, expert assignment must carry substantial mutual information about LibriAugmented impairment labels, beyond what random-gate and Mel-gate assignments carry. Fourth, the improvement in DNSMOS SIG must track the improvement in human SIG. That last check matters because DNSMOS's clip-level correlation with human SIG is only 0.71–0.83 (DNSMOS P.835), and because our gate depends on a DNSMOS-family model. A rise in DNSMOS without a matching rise in human ratings would show circularity, not progress.
References
- Icassp 2022 Deep Noise Suppression Challenge
- Impairments are Clustered in Latents of Deep Neural Network-based Speech Quality Models
- Speech Enhancement using a Deep Mixture of Experts
- Sparse Mixture of Local Experts for Efficient Speech Enhancement
- Dnsmos P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors
- MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement
- Improving speech synthesis for noisy environments
- MetricGAN-OKD: Multi-Metric Optimization of MetricGAN via Online Knowledge Distillation for Speech Enhancement
- INTERSPEECH 2021 Deep Noise Suppression Challenge