Making Reference-Free Quality Losses Hard to Cheat: Annealed Invariance Averaging for Speech Enhancement

A speech enhancer can be trained without any clean reference recordings by maximizing the score of a learned quality predictor. This predictor is a neural network that listens to audio and outputs an estimated quality score, much as a human listener would. The approach is attractive because clean references do not exist for real calls, hearing aids, or in-the-wild recordings. It has a known failure, though. After enough optimization, the enhancer learns artifacts that fool the predictor: the predicted score climbs while true quality falls (Learning With Learned Loss Function). We propose to treat this failure the way 3D registration treats false local minima. Point-cloud aligners escape bad solutions by first optimizing a coarse, smoothed objective and then sharpening it. We apply the same schedule to the quality predictor. The enhancer is trained to maximize the predictor's score averaged over transformations that humans do not notice, such as small time shifts, phase changes, gain changes, and mild re-encoding. The range of those transformations starts wide and narrows over training.

The underlying bet is that a real quality improvement survives these perturbations and an adversarial artifact usually does not. The perturbations are chosen from exactly the properties that break intrusive metrics: alignment, phase, and silence handling. A human judges quality without a reference and without sample-level alignment, so a trustworthy reference-free loss should not care about these details either. We also borrow an evaluation habit from global registration. Instead of reporting one training run, we report the fraction of runs across many initializations and seeds that end in a genuine improvement rather than a trap.

Core Claim

Exploiting a frozen quality predictor is largely a problem of the optimizer being pulled into narrow, high-scoring valleys. These valleys are artifacts of the predictor's fine structure, and which valley a run falls into depends on where the enhancer starts. Averaging the predictor over a coarse-to-fine family of perceptually irrelevant transformations widens the basin that leads to real improvement and fills in the artifact valleys. As a result, more runs succeed, and the optimized score keeps agreeing with held-out judges far longer than under existing safeguards.

Concretely, at training stage $k$ we maximize

$$J_k(\theta) = \mathbb{E}_{x \sim D}\,\mathbb{E}_{\tau \sim T(r_k)}\big[\,Q(\tau(f_\theta(x)))\,\big] - \lambda\, R(f_\theta(x), x),$$

where the terms are defined as follows. $D$ is a set of noisy recordings with no clean targets. $f_\theta$ is the enhancer with parameters $\theta$. $Q$ is a frozen differentiable non-intrusive quality predictor. $T(r)$ is a distribution over human-invariant transformations whose magnitude is bounded by a radius $r$. $r_k$ is a schedule that shrinks from large to small. $R$ is a self-reconstruction penalty of the kind used in MetricGAN-U, weighted by $\lambda$.

The schedule plays the role of the kernel bandwidth in MMD-Reg, which uses Maximum Mean Discrepancy, a distance between two point distributions measured through a smoothing kernel. That paper shows that large kernel scales flatten spurious minima and small scales recover fine alignment. The transformations take the place of the correspondence-free invariance that Feature-Metric Registration obtains by comparing global features instead of exact point matches.

Relation to Prior Work

Three papers motivate the proposal by documenting the trap and the fixes tried so far. Fu et al. trained Quality-Net with spectral normalization, which makes the network 1-Lipschitz (its output cannot change faster than its input). They froze it, used it as a loss, and gained 0.18 PESQ over a Mean Squared Error (MSE) loss on TIMIT, yet still observed adversarial over-optimization. This is direct evidence that bounding the predictor's slope does not prevent the enhancer from finding artifact valleys.

MetricGAN-U optimizes DNSMOS, a learned reference-free quality predictor, through an adversarial network that imitates the metric. It needs a self-reconstruction constraint to stop collapse. That constraint limits how far the output may move from the input, but it does nothing to reshape the objective itself.

PESQNet avoids drift by repeatedly re-training the predictor against true PESQ on synthetic pairs. This works, but it reintroduces paired data. Our method keeps the predictor frozen and requires no pairs.

Two registration papers supply the method rather than the motivation. MMD-Reg gives the coarse-to-fine smoothing principle. Go-ICP supplies the evaluation habit of measuring success rate across many random initializations; it reached 100% success over 100 initializations where plain ICP failed.

Finally, ViSQOL v3 constrains which transformations we choose. Its authors show that intrusive scores break under micro time shifts, silences, and waveform-divergent generative codecs. Those same factors define our invariance family, since human judgments are stable under them.

We deliberately do not pursue the "certified Lipschitz basins" or branch-and-bound directions the research file suggests. Certifying a basin for a deep enhancer optimizing a deep predictor is not tractable. Fu et al.'s result already shows that Lipschitz control of the predictor alone is insufficient.

Research Program

The first stage tests the premise before building anything. We reproduce the trap by optimizing an enhancer against frozen DNSMOS on VoiceBank-DEMAND noisy training audio, with checkpoints saved throughout. For outputs from late, over-optimized checkpoints, we apply each candidate transformation and measure how much of the predictor's score each one removes. We compare this with outputs from early checkpoints, where true quality is still improving. The candidates are sub-millisecond shifts, random all-pass phase, ±3 dB gain, low-bitrate re-encoding, and silence padding.

The idea is only viable if trapped outputs lose substantially more score under these transformations than genuine improvements do. If they do not, the artifacts are invariance-robust and the method is dead. That outcome would still be a useful negative result.

The second stage builds the annealed objective. We keep the transformations whose score loss best separates trapped outputs from genuine ones and define $T(r)$ over them. We then compare radius schedules: fixed large, fixed small, linear shrink, and shrink triggered when held-out validation stalls. Held-out validation here means a second, non-optimized predictor on a small development split.

The third stage is the initialization study. For each method, we train from a grid of starting points: random, MSE-pretrained on synthetic pairs, and supervised-pretrained at several levels of training. Each start is run with several seeds. We record the fraction of runs whose final outputs beat their own starting point on every held-out judge. We also record the training step at which the optimized score and the held-out judges stop moving together, which we call the divergence step.

Baselines

The naive reference points are the unprocessed noisy input and an MSE-trained enhancer. Fu et al. report the MSE-trained model on TIMIT as 0.18 PESQ below Quality-Net training, which fixes the scale of improvement a perceptual loss can buy. For the unprocessed input, no reported result was found on Lacuna for VoiceBank-DEMAND.

The strongest reference-free competitor is MetricGAN-U, which reports PESQ 2.45 and DNSMOS 3.151 on VoiceBank-DEMAND (MetricGAN-U). It is the strongest because it solves the same problem: enhancement from noisy audio alone, targeting DNSMOS, with a guard against collapse. We will rerun it with the same enhancer architecture, training data, DNSMOS version, and number of gradient steps. Its reconstruction weight will get the same tuning budget as our $\lambda$ and radius schedule. Beating it means higher PESQ than 2.45 at matched or higher DNSMOS, together with a better score on a second non-optimized predictor. Matching its DNSMOS while losing on held-out judges would count as failure.

A second direct competitor is the frozen spectrally normalized surrogate of Fu et al., rebuilt on a reference-free predictor. It isolates whether Lipschitz control alone suffices.

On real recordings, we compare against the alternating PESQNet protocol. It reports an overall MOS of 3.16 against a 3.07 baseline on the Interspeech 2021 DNS Challenge blind set (PESQNet). We treat it as an upper-reference that uses paired synthetic data our method does not. Approaching it without pairs would be the claim, not exceeding it.

Three ablations isolate the key ingredient. The first uses the same averaging with a fixed radius, with no annealing. The second averages over isotropic Gaussian waveform noise instead of human-invariant transformations, which tests whether the invariance family matters or any smoothing does. The third uses the annealed objective with $\lambda=0$, which tests whether the reconstruction term is still needed.

Why not just…?

Why not just penalize distance from the starting model, as is common when fine-tuning against a learned reward? This is the most serious threat. A proximal penalty keeps the enhancer inside the capture range of its initialization and may by itself prevent most traps. MetricGAN-U's self-reconstruction term is a version of this idea, and it still leaves a gap to supervised methods. The likely reason is that a distance penalty caps both artifacts and real improvements. We include a tuned L2-to-initialization penalty as a baseline and plot held-out quality against how far the output has moved from the input. If the proximal penalty matches our curve, our method adds nothing, and we will report that.

Why not just ensemble several quality predictors? Artifacts that fool one network may not fool another, and averaging over networks is simpler than averaging over transformations. However, predictors trained on similar data share blind spots. Averaging over predictors also does not produce a coarse-to-fine schedule. We run an ensemble of all available reference-free predictors as a baseline, then evaluate on a predictor held out from the ensemble. We also combine the ensemble with our averaging to test whether the two are complementary.

Why not just keep re-training the predictor, as PESQNet does? Re-anchoring works, but it needs synthetic clean-noisy pairs. Its authors note that it drifts when synthetic data differs from real acoustics. REAL-M shows how large that difference can be: separation drops from about 20 dB SI-SNR on synthetic data to an estimated 2.8 dB on real mixtures. Our method targets the setting where no pairs can be trusted. The PESQNet comparison on DNS blind data measures how much of the re-anchoring benefit we recover without pairs.

Why not just stop training early? Early stopping requires a trustworthy judge to decide when to stop, which is the missing piece in the reference-free setting. The divergence-step measurement quantifies this directly. If our method only delays divergence without raising the peak held-out quality, it amounts to early stopping in disguise.

Proposed Experiments

The main benchmark is VoiceBank-DEMAND. We train on its noisy audio alone and use its clean test references only for evaluation, with PESQ and STOI (Short-Time Objective Intelligibility) as intrusive judges the enhancer never sees. For real data we use the Interspeech 2021 DNS Challenge blind set, reporting DNSMOS P.835 speech, background, and overall scores from Gopal et al. when DNSMOS was not the optimized predictor. Otherwise we report a held-out predictor. For generalization to unseen noise we use TIMIT with its seen and unseen noise split, following Fu et al.

We run a small AB listening test on DNS blind outputs. The pairs are our method against MetricGAN-U and our method against the proximal-penalty baseline. We follow the AB preference protocol of Fu et al., who report p = 0.00027 for Quality-Net over MSE. Our p-value will come from our own test.

Four checks are decisive. The first is the stage-one premise test: trapped outputs must lose more predictor score under invariant transformations than genuine improvements do. The second is success rate across initializations and seeds, which must be clearly higher than for MetricGAN-U and the proximal penalty. The third is the divergence step between the optimized score and held-out judges, which must come later, with a higher held-out peak. The fourth is the Gaussian-noise ablation, which must lose to human-invariant averaging. If the first check fails, we stop. If the second and third pass but the proximal penalty ties us, the contribution reduces to a diagnostic study of initialization-dependent reward exploitation in reference-free enhancement. We would report it as such.