Generative speech enhancers produce clean audio that listeners prefer, yet they score badly on fidelity metrics, and sometimes they change the words. A generative enhancer is a model that samples a plausible clean recording given a noisy one, rather than estimating a single average answer. This proposal tests a specific explanation for the low fidelity scores. Part of the fidelity loss is the price of realism that theory says is unavoidable. A further part is avoidable: the model outputs speech that could not have produced the recording the microphone actually heard. Room physics gives a way to detect that second part and remove it.
The plan has three pieces. We fit a small differentiable model of the room to each recording; "differentiable" here means it can be tuned by gradient descent through its own equations. We then steer the generative sampler so that its output, played back through that room model plus a modeled noise floor, reproduces the observation. Finally, we measure how far the result moves toward the best fidelity that any equally realistic method could reach. The main scientific question is how much physical structure the room model needs. If it is too flexible, it can explain away any hallucination. If it is too rigid, it forces distortion through model error.
Core Claim
The Perception-Distortion Tradeoff proves that no restoration method can be both maximally realistic and minimally distorted. That result fixes a reference point we can compute. Suppose a sampler draws $\hat{x}$ from the true posterior $p(x\mid y)$, independently of the clean speech $x$ given the observation $y$. Its expected error is exactly twice the error of the best average estimate:
$$\mathbb{E}\|x-\hat{x}\|^2 = 2\,\mathbb{E}\|x-\mathbb{E}[x\mid y]\|^2 .$$
This holds because the two independent posterior deviations add in variance. A perfectly realistic, perfectly faithful sampler therefore sits about 3 dB of signal-to-distortion below the best regression estimate. Anything worse is excess distortion. The excess comes from samples that leave the true posterior through phase drift, misalignment, or invented phonemes.
Our claim is that, in reverberant speech, much of this excess can be measured and removed by a consistency check under a physically structured room model. Reverberation repeats every phoneme in delayed and filtered copies, so a recording contains redundant evidence about its content. A hallucinated syllable would have to match its own echoes, and generally does not. An unconstrained filter cannot use this redundancy, because it can bend itself to fit whatever speech the sampler proposes. A room model with explicit early-reflection delays and a structured late tail can use it.
Research Program
We start from a conditional Schrödinger bridge enhancer trained with direct clean-signal prediction. This is a generative model that transports the noisy recording toward a clean one in a few steps. We use the configuration that Investigating Training Objectives for Generative Speech Enhancement found stable, but drop the differentiable-PESQ term, so that fidelity gains cannot come from a surrogate loss.
At test time we attach a forward model of how the observation was produced:
$$\mathcal{L}(x,\phi,\sigma)=\sum_{t,f}\frac{|Y_{t,f}-H_\phi(f)\,X_{t,f}|^2}{\sigma^2_{t,f}}+\log\sigma^2_{t,f}.$$
Here $X$ and $Y$ are the short-time spectra of the candidate clean speech and the observation, indexed by time frame $t$ and frequency $f$. $H_\phi$ is the transfer function of the room model with parameters $\phi$. $\sigma^2_{t,f}$ is a smooth, slowly varying noise power that is fitted jointly with $\phi$. Writing the room as a transfer function assumes it changes slowly relative to the analysis window.
The room model follows Room Impulse Response Synthesis via Differentiable Feedback Delay Networks. It combines a sparse set of delayed taps for early reflections with a feedback delay network for the dense tail; a feedback delay network is a small recursive filter bank that mimics many bouncing reflections. That paper used the model only for forward rendering. We invert it.
We evaluate the network's transfer function in closed form in the frequency domain instead of unrolling the recursion over time. This sidesteps the exploding and vanishing gradients that long recursive simulations suffer. During the bridge's reverse steps we alternate two updates. One adjusts the room and noise parameters by gradient descent on $\mathcal{L}$. The other nudges the sample toward lower $\mathcal{L}$, weighted by step size. We initialize the room parameters from the input with Weighted Prediction Error (WPE), a classical statistical dereverberation method.
The structure of this loop borrows from three sources, and we depart from each.
Score Matching via Differentiable Physics splits the reverse diffusion step into a physical operator and a learned correction, but only for fluid partial differential equations whose operators are known. We take the split but must identify the operator per recording.
Diffusion Posterior Sampling for Informed Single-Channel Dereverberation injects a measurement-consistency gradient but assumes the room impulse response is given. We use it as an oracle upper bound.
Unsupervised Blind Joint Dereverberation and Room Acoustics Estimation with Diffusion Models (BUDDy) is the closest prior work. It jointly estimates speech and a room model, but with three limitations. Its room model is subband exponential decays under a minimum-phase assumption, which cannot represent discrete early reflections. It omits additive noise. It does not ask where its outputs land relative to the theoretical bound. We differ on all three counts, and we vary model structure deliberately.
PhysWave showed that physics losses can guide a diffusion sampler without retraining, but under free-field propagation with no room. Our setting is the reverberant one it leaves open.
Two other room models set constraints on our design. Differentiable Acoustic Radiance Transfer models only acoustic energy, so it carries no phase. That makes it unsuitable as the consistency operator, since phase is exactly where generative samples drift. We use it only as an optional source of decay priors when room geometry is known.
Differentiable Consistency Constraints for Improved Deep Speech Enhancement enforces a purely algebraic consistency: the separated parts must sum back to the mixture. It gained 0.7 dB of scale-invariant signal-to-distortion improvement (SI-SDR, a sample-by-sample fidelity score) in a discriminative model. We treat it as the "no physics" version of our idea.
The analysis component turns the 3 dB identity into a measurement. On each test set we train a strong discriminative regressor, an MMSE proxy (MMSE is the minimum-mean-squared-error estimate, i.e. the best average answer). Its error, doubled, gives an estimated fidelity floor for realistic samplers. We then report each method's excess distortion above that floor alongside non-intrusive realism scores. The estimate is only as good as the proxy, so we report it for two regressors of different capacity.
The structural question is tested by sweeping the room model's expressiveness through five settings: subband decays only, as in BUDDy; our early taps plus a delay-network tail; the same with more taps; a free finite impulse response (FIR) filter of full length; and no room model at all. Our prediction is U-shaped. Excess distortion and hallucination should be lowest at intermediate structure, and rise at both extremes for the reasons given above.
Baselines
The do-nothing reference is the unprocessed observation, scored on every metric. The pure generative reference is the same Schrödinger bridge sampled with no guidance. Every gain we report is relative to that identical network, checkpoint, and number of steps.
The strongest fidelity competitor is a large discriminative regressor. It plays the role of the purely discriminative winner of the Interspeech 2025 URGENT Speech Enhancement Challenge, which led on intrusive and downstream metrics but trailed generative systems in listener preference. The file reports no numbers for that winner or for the generative entries; no reported result was found on Lacuna for either. We train our regressor on the same data as the bridge with a matched parameter count. Beating it is not the goal on SI-SDR. Beating it means landing within 3 dB of it on SI-SDR while matching the unguided bridge on non-intrusive quality.
The strongest perceptual competitor is the bridge trained with a differentiable PESQ loss (PESQ is a standard intrusive quality score), as in Investigating Training Objectives for Generative Speech Enhancement. Its model M6 reports PESQ 3.70 on Voicebank-Demand, which is the number to stay near on that benchmark while gaining SI-SDR.
The strongest physics competitor is BUDDy. We run it with WPE initialization and the same sampling budget, measured in network evaluations plus inner gradient steps, and give it the same added noise model so that the comparison isolates room structure. The physics upper bound is diffusion posterior sampling with the true room impulse response. On a WHAMR!-style setup, that method reports PESQ near 2.5 at 10 dB SNR, against below 2.0 for regularized inverse filtering with post-processing.
The ablations that isolate the key ingredient are the structure sweep above and an algebraic-only mixture-consistency projection applied to the bridge output. All guided methods get an equal hyperparameter search budget, tuned on a held-out room set disjoint from the test rooms.
Why not just…?
Why not just interpolate between the discriminative and generative outputs, or average several generative samples? Both trace a curve between realism and fidelity at almost no cost. This is the most serious threat. If the physics-guided points lie on or inside that curve, the proposal fails, and we will report it that way. The reason to expect otherwise is that interpolation can only blend two estimates; it cannot remove a hallucinated phoneme present in the generative one. On the unseen-language URGENT test set, character accuracy (CAcc, from a speech recognizer) should separate the two. Interpolation should raise SI-SDR while leaving the generative model's substitution errors partly intact. Consistency guidance should remove them.
Why not just enforce mixture consistency without any room physics? The Differentiable Consistency Constraints projection is cheap and helped discriminative models. In reverberant speech, however, it treats echoes as part of the speech or the noise rather than as evidence. A flexible enough residual can absorb a wrong phoneme. The full-length FIR and no-room-model ablations test this directly. If they match the structured model, the physics is unnecessary.
Why not just use BUDDy's simpler room model? It is published, unsupervised, and already generalized to real recordings and singing voice. Its exponential-decay, minimum-phase form cannot place discrete early reflections. Those reflections are where the redundancy that exposes hallucinations is strongest, and where misfit forces distortion. The measured, non-shoebox rooms from Real Acoustic Fields are the test. If BUDDy's model is as good there, our added structure buys nothing.
Why not just scale the training data? URGENT 2025 reports that going from about 2,500 to 60,000 hours did not significantly improve results, and generative hallucination on unseen languages persisted. Scale is therefore not an obvious fix. Our guidance needs no retraining, so the two approaches combine rather than compete.
Proposed Experiments
The first experiment uses WSJ0 speech with WHAMR!-style simulated rooms and noise, where true impulse responses are available. It tests two things. One is whether the fitted room model recovers decay time and clarity. The other is the main fidelity question: does excess distortion above the doubled-regressor floor fall while WavLM-based MOS and DNSMOS (non-intrusive quality predictors) stay within noise of the unguided bridge?
The second experiment repeats this with WSJ0 convolved with real measured impulse responses from Real Acoustic Fields. That is where the structure sweep should show its U shape, because real rooms violate simple decay models.
The third experiment uses the URGENT 2025 benchmark, with emphasis on the unseen Japanese set. The measures are CAcc and SDR, testing whether consistency guidance suppresses phoneme substitution.
The fourth experiment is a deliberate negative control on Voicebank-Demand. There is no reverberation there, so the room model has almost nothing to exploit. We predict little gain over noise-model-only guidance, which would show that any improvements elsewhere come from the room physics rather than from extra test-time computation. The fifth experiment runs the real DDS-DAPS recordings for a listening test against BUDDy.
The decisive checks are the following. On the third and fourth experiments together, we will show whether guided points move toward the doubled-regressor floor beyond the curve traced by interpolation and averaging. In the second experiment, a structured room model should beat both the free FIR and the subband-decay model on CAcc and excess distortion. Hallucination rate should track the consistency residual, which would make that residual usable as a per-utterance fidelity warning. Failure of the first check refutes the core claim. Failure of the second means room physics is not the active ingredient.
References
- The Perception-Distortion Tradeoff
- Investigating Training Objectives for Generative Speech Enhancement
- Room Impulse Response Synthesis via Differentiable Feedback Delay Networks for Efficient Spatial Audio Rendering
- Score Matching via Differentiable Physics
- Diffusion Posterior Sampling for Informed Single-Channel Dereverberation
- Unsupervised Blind Joint Dereverberation and Room Acoustics Estimation With Diffusion Models
- PhysWave: Physics-Guided Latent Diffusion Models for Controllable Spatial Audio Generation
- Differentiable Acoustic Radiance Transfer
- Differentiable Consistency Constraints for Improved Deep Speech Enhancement
- Interspeech 2025 URGENT Speech Enhancement Challenge
- Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark