Refine Only While Contracting: Curvature-Certified Diffusion Refinement for Speech

Diffusion refiners are generators that learn to turn random noise into clean speech by removing noise step by step. They are now used to polish the output of ordinary speech enhancers. They make audio sound more natural to automatic quality predictors, but they drift away from what the speaker actually said, and the drift is currently controlled by a hand-tuned mixing knob. We propose to measure, per utterance and during sampling, whether the refiner's reverse process is still pulling nearby inputs together (contracting) or has started pushing them apart (expanding). Contraction removes artifacts without inventing content. Expansion is where the model starts choosing fine detail that the input does not determine. Our method stops the refinement at the point where it stops contracting.

The mathematical side of this comes from a line of probability theory. That work shows two things. First, the maps that reverse a diffusion are provably contractive when the smoothed distribution is curved like a single bump (log-concave). Second, for multimodal data the curvature only becomes favorable after enough smoothing. We turn that curvature condition into a quantity we can measure on a trained speech diffusion model, and we test whether it explains and controls the known gap in which generative refiners gain on reference-free perceptual scores while losing on reference-based fidelity.

Core Claim

Diffiner refines the output of an upstream enhancer or separator with a diffusion model (denoising diffusion probabilistic model, DDPM) trained on clean WSJ0 speech. It reports that reference-free metrics (NISQA, DNSMOS) consistently improve, "often surpassing even the original clean speech scores", while reference-based metrics slightly degrade. It manages this trade-off with a single linear blending weight between the upstream output and the refined output. Our claim is that the trade-off is not a single global dial. It has a per-utterance, per-noise-level cause: the sign of the curvature of the smoothed speech distribution along that utterance's reverse trajectory.

Consider the deterministic reverse flow $\Phi_{t_0\to\tau}$, which carries a signal from noise level $t_0$ down to level $\tau$. Let $\kappa_s(x_s)$ be the smallest eigenvalue of the symmetrized negative Jacobian of the reverse-time velocity field at the trajectory point $x_s$. A Grönwall argument gives the pathwise Lipschitz bound

$$ \|D\Phi_{t_0\to\tau}(x)\| \le \exp\!\Big(-\int_\tau^{t_0} \kappa_s(x_s)\,ds\Big). $$

For the Ornstein-Uhlenbeck probability flow, $\kappa_s$ is an affine function of the Hessian of $-\log p_s$, where $p_s$ is the data density smoothed to level $s$. This is the same multiscale curvature that bounds the Lipschitz constant of Langevin transport maps in Exact Renormalization Groups and Transportation of Measures. It also governs contraction in the reverse-heat-flow construction of A generalization of Caffarelli's contraction theorem via (reverse) heat flow.

Those papers use the bound to prove that the maps contract under assumptions that real audio does not satisfy. We use it in the other direction. On a trained speech prior, we estimate the accumulated log-expansion $E(\tau)=-\int_\tau^{t_0}\kappa_s\,ds$ along each utterance's actual trajectory. We then refine only while $E(\tau)\le\beta$, where $\beta$ is a budget near zero.

The payoff is a fidelity certificate. Fix the injected noise seed, and let $a$ be the scaling applied to the input before sampling. For a degraded input $x$ and its clean reference $c$, the refiner $R$ satisfies

$$ \|R(x)-c\| \le a\,e^{\beta}\,\|x-c\| + \|R(c)-c\|. $$

In words, within the contractive regime the refinement cannot move the output further from the clean signal than the input was, apart from the refiner's self-consistency error on clean speech. That error can be measured on held-out clean data. The certificate holds in waveform or spectrogram Euclidean distance, where the theory lives. Perceptual quality is evaluated separately.

Why the Theory Predicts a Stopping Point

Convexity of mutual information along the heat flow studies how information decays as noise is added. The companion Ornstein-Uhlenbeck result shows the same under OU dynamics. Together they establish that the information-decay curve is convex for log-concave sources. For mixtures and point masses, it is concave at small noise and becomes convex only past a threshold $t^*$ set by the data's geometry.

Speech is clearly such a mixture. Phonemes, voicing states, and pitch lie on separated low-dimensional pieces. The prediction is therefore that $\kappa_s$ is positive at high noise and turns negative below some utterance-dependent level. That negative-curvature region is exactly where the refiner decides between competing fine structures. Such decisions raise naturalness scores and lower SDR.

Two further results support the high-noise half of this picture. Spectral monotonicity under Gaussian convolution shows that for log-concave measures the Poincaré constant grows monotonically along the heat flow, which is consistent with curvature improving as noise increases. The Brownian transport map shows that transport built from diffusions can remain contractive even for Gaussian mixtures, where optimal-transport maps fail. This suggests that the diffusion-built map is the right object to certify.

These results motivate the proposal but do not supply it. None of them estimates curvature on a learned score or connects it to audio evaluation. That empirical characterization of $t^*$ on speech is itself one contribution of this work.

Research Program

We will train or reuse a clean-speech score model on WSJ0, following the setup of Diffiner, and run it as a deterministic probability-flow refiner with a fixed noise seed. The core tool is a trajectory-level expansion estimator. It integrates the variational equation of the reverse ODE with Jacobian-vector products, and it uses a few power iterations with vector-Jacobian products to track the largest singular value of $D\Phi_{t_0\to\tau}$ as $\tau$ decreases. This gives $E(\tau)$ directly, without forming the Hessian. The cost is a handful of additional ODE solves, which we report as a multiple of plain sampling.

Before touching speech, we validate the estimator on Gaussian mixtures, where the convexity threshold can be computed in the spirit of Wibisono and Jog. The estimated turn from contraction to expansion should occur at the analytically predicted noise level.

The controller then works as follows. From the upstream output, it injects noise at level $t_0$ and runs the reverse ODE while $E(\tau)\le\beta$. At the stopping level $\tau^*$, it returns the denoiser's estimate of the clean signal, that is, the conditional mean given the current noisy state. It does not continue into the expanding regime.

The analysis has three parts. First, we map how $\tau^*$ is distributed across utterances, noise types, and separation versus enhancement tasks. Second, we test whether the expansion accumulated below $\tau$ predicts, per utterance, two quantities: the SDR lost by refining to $\tau$, and the reference-free gain. Third, we test the certificate itself. We check the inequality above empirically with finite-difference perturbations, and we measure how often and by how much it is violated. A pathwise local bound does not cover the whole segment between $x$ and $c$, so this check is required rather than optional.

Baselines

The do-nothing reference is the upstream enhancer or separator output with no refinement, on VoiceBank-DEMAND, WSJ0+CHiME, and WSJ0-2mix. This fixes the SDR ceiling and the perceptual floor that every refiner trades between.

The strongest existing method is Diffiner itself: full reverse diffusion followed by its linear blending. We rerun it with the same score network, the same number of sampler steps, and a blending weight tuned on the same validation split and with the same tuning budget we give our budget $\beta$. The file contains no reported NISQA, DNSMOS, or SDR tables for Diffiner. We therefore found no reported result on Lacuna for Diffiner on these benchmarks, and all Diffiner numbers will come from our own reproduction.

The second strong competitor avoids generative refinement entirely: fine-tuning the enhancer with a learned perceptual loss. CDPAM reports that fine-tuning the DEMUCS denoiser with its loss raised subjective MOS from 3.73 to 4.07. We fine-tune the same upstream model with CDPAM under a matched compute budget.

Beating these baselines means one of two things. Our frontier of SDR against DNSMOS, NISQA, and human preference must strictly dominate the blending frontier at matched compute. Alternatively, at equal human preference we must retain a statistically significant amount of SDR that Diffiner gives up.

Three ablations isolate the key ingredient. The first uses a global stopping level fixed per dataset to the mean of $\tau^*$, which tests whether per-utterance adaptivity matters. The second triggers stopping on the magnitude of the score rather than the sign of the curvature, which tests whether contraction specifically, rather than generic "large updates", is the signal. The third continues the ODE past $\tau^*$ instead of returning the conditional mean.

Why not just…?

Why not just tune Diffiner's starting noise level and blending weight globally? This is the most serious threat. If utterances within a dataset share roughly the same $t^*$, a single tuned pair of starting noise and blending weight would reproduce our frontier. The fixed-mean-$\tau^*$ ablation settles the question directly. If it matches the adaptive controller within confidence intervals, we will drop the controller claim and present the expansion measure only as a per-utterance diagnostic of where refinement hallucinates.

Why not just measure the spread across several stochastic samples? Sample variance at a given noise level also reflects a multimodal posterior, and we expect it to correlate with our expansion measure. That makes it a real competitor. We will compare the two as per-utterance predictors of SDR loss at equal numbers of network evaluations, and we will use each as a stopping signal. If spread predicts equally well, the cheaper sampling-based rule wins, and our contribution narrows to the certificate, which spread cannot provide.

Why not just fine-tune the enhancer with a perceptual loss like CDPAM? CDPAM's MOS gain on DEMUCS shows this route works. However, CDPAM is an unconstrained network with no robustness guarantee, and the adversarial collapse documented for vision metrics in LipSim (DreamSim alignment falling from about 96% to 46% at $\epsilon=0.5$) gives reason to expect similar exploitability in audio. We include CDPAM fine-tuning as a baseline. We also test both systems under small input perturbations, where our certificate predicts bounded output change.

Why not just make the score network 1-Lipschitz, LipSim-style? Contraction of the flow depends on the sign of the curvature, not on the magnitude of the score. A score with small Lipschitz constant can still produce an expanding flow. Conversely, a speech prior forced to have a small-Lipschitz score cannot represent well-separated modes. LipSim itself pays a clean-accuracy cost, from 96.2% to 85.6% on NIGHT. The score-magnitude ablation tests the first point.

Why not just stop when a quality predictor such as DNSMOS stops improving? That optimizes the evaluation metric directly. Diffiner's scores already "surpass clean speech", which is itself a warning sign of predictor exploitation, and this rule provides no fidelity protection. We report the rule for completeness but judge it against human ratings.

Decisive Experiments

The first check is synthetic. On Gaussian mixtures, the estimated turning point must match the analytic threshold. If it does not, the estimator is wrong and nothing downstream is interpretable.

The second check tests the diagnostic claim. On VoiceBank-DEMAND and WSJ0-2mix, the expansion accumulated below $\tau$ should correlate with per-utterance SDR loss and with the CDPAM distance to the reference. The correlation must be significantly stronger than that of score norm and at least equal to that of sample spread at matched compute.

The third check tests the controller. A MUSHRA-style listening test on all three benchmarks, reported alongside NISQA, DNSMOS, and SDR, should show that contraction-budgeted refinement matches Diffiner's human preference while keeping more SDR. It must also beat the global-stopping ablation.

The fourth check tests the certificate. Measured violations of the fidelity inequality should be rare and small on held-out data. Output change under bounded input perturbations should stay within the certified factor for our method, while Diffiner and CDPAM fine-tuning show no such bound.

A clean negative result would also be informative. If speech priors show no stable turning point, or if the turning point carries no information beyond a global noise level, that bounds how far the transport-contraction theory can explain generative audio's evaluation gap.