TL;DR
This paper introduces the Regularized Schrödinger Bridge (RSB), a novel diffusion model for solving inverse problems that tackles the inherent distortion-perception tradeoff and exposure bias. RSB achieves this by employing a unique regularization strategy that perturbs both training targets and input states, effectively guiding the model towards optimal solutions and mitigating prediction errors during inference.
What Problem Does This Paper Address?
Solving "inverse problems" is a fundamental task in many fields, like image restoration and speech enhancement. Imagine trying to reconstruct a clear image from a blurry one, or a clean audio signal from a noisy one. The challenge is that these problems are "ill-posed," meaning there can be many possible solutions for a single degraded input. This leads to two major issues for powerful generative models like diffusion models:
- Distortion-Perception Tradeoff (DP Tradeoff): Improving the visual or auditory "realism" (perception) of the output often comes at the cost of accuracy or fidelity to the original signal (distortion). Finding a sweet spot that balances both is difficult.
- Exposure Bias: During training, diffusion models learn by seeing perfect, ground-truth data at each step. However, during inference (when the model is actually used), it has to rely on its own previous predictions, which might contain errors. These errors can accumulate over many steps, leading to a significant drop in output quality.
Existing diffusion models, even advanced ones like Schrödinger Bridges (SB), struggle to adequately address both these challenges simultaneously.
Methods
The core innovation of this paper is the Regularized Schrödinger Bridge (RSB). It builds upon the Schrödinger Bridge framework, which uses an optimal transport perspective to move from a measurement distribution (e.g., noisy speech) to a clean data distribution. RSB introduces a novel Distortion-Perception Regularization strategy:
- Perturbing Training Targets: Instead of training the model to predict the exact clean data (
X), RSB trains it to predict a "perturbed target" (X_tgt). This target is an interpolation between the minimal distortion estimate (the posterior mean,X*) and the ground-truth clean data (X). This interpolation is designed to guide the model's predictions along the optimal DP tradeoff curve. The interpolation weight (w_t) is time-varying, starting closer toX*and moving towardsX. - Perturbing Input States: To combat exposure bias, RSB also perturbs the input states (
X_state) used during training. This perturbation simulates the prediction errors that the model will encounter during inference. By exposing the model to these simulated errors during training, it becomes more robust and less susceptible to error accumulation. - Joint Perturbation: The paper emphasizes that perturbing both the targets and the input states simultaneously (joint perturbation) is crucial for achieving the best results.
Essentially, RSB creates a more robust training environment that forces the model to learn how to navigate the DP tradeoff and handle its own potential mistakes.
Datasets
The paper evaluates RSB on two common audio inverse tasks:
- Speech Denoising:
- WSJ0+WHAM: A synthetic dataset created by mixing clean WSJ0 utterances with noise from the WHAM corpus.
- VoiceBank+DEMAND: A benchmark dataset combining clean speech from VoiceBank with noise from the DEMAND corpus.
- Speech Dereverberation:
- WSJ0+Reverb: A synthetic dataset generated by convolving clean WSJ0 utterances with room impulse responses.
No new datasets were introduced in this work.
Key Insights and Ideas
- Simulating Inference Errors During Training: The most significant insight is that actively simulating the prediction errors that occur during inference during the training phase is key to mitigating exposure bias. This makes the model more resilient to its own imperfections.
- Interpolation for Optimal Tradeoff: By interpolating between the minimal distortion estimate and the ground-truth data, RSB effectively guides the model to find a better balance on the DP curve, rather than getting stuck in suboptimal solutions.
- Joint Perturbation is Superior: The ablation study clearly shows that perturbing both inputs and targets together yields better results than perturbing only one. This suggests a synergistic effect in addressing both exposure bias and the DP tradeoff.
- RSB as a "Label Smoothing" for Inverse Problems: The authors draw an analogy between their target perturbation and label smoothing techniques, highlighting how it helps the model generalize better and avoid overfitting to specific training examples.
Key Results
RSB consistently outperforms state-of-the-art methods on both speech denoising and dereverberation tasks.
- Speech Denoising (WSJ0+WHAM): RSB achieved the best perceptual scores (PESQ, CSIG, CBAK, COVL) and improved distortion metrics (SI-SDR, SI-SIR, SI-SAR) compared to a pure predictive model (NCSN++M) and the vanilla SB. It achieved a SI-SDR of 18.8 dB.
- Speech Dereverberation (WSJ0+Reverb): RSB matched or exceeded the best perception-oriented metrics and outperformed all baselines in distortion-oriented metrics.
- VoiceBank+DEMAND: RSB achieved the highest SI-SDR (18.8 dB) and matched SB in PESQ, demonstrating its effectiveness across different datasets.
- Exposure Bias Mitigation: Figure 3 shows that RSB exhibits significantly flatter prediction error curves during sampling, indicating slower error accumulation and thus reduced exposure bias.
- DP Tradeoff Improvement: Figure 4 demonstrates that RSB consistently achieves higher PESQ and better preserves SI-SDR across different sampling steps, indicating a more favorable DP tradeoff.
Limitations and Future Work
- Computational Cost: While RSB aims for efficiency, diffusion models, in general, can still be computationally intensive. The paper mentions that RSB uses the same SDE sampler as vanilla SB, and the models have around 25.2M parameters.
- Applicability to Other Inverse Problems: The paper suggests that RSB is readily applicable to a wide range of inverse problems beyond speech enhancement, such as image denoising, deblurring, and inpainting, which would be a natural direction for future work.
- Further Exploration of Tradeoff Strategies: While RSB shows promise, the paper notes that other strategies for improving the DP tradeoff (like using minimal MSE estimates as conditioning) were explored but underperformed RSB, suggesting that the specific interpolation approach in RSB is particularly effective.
The authors believe their work opens avenues for further research into more robust and effective diffusion models for inverse problems.
Summary updated