Abstract
Denoising Diffusion Probabilistic Models (DDPMs) have achieved significant success in generation tasks. Nevertheless, the exposure bias issue—the natural discrepancy between training (where inputs are ground-truth derived) and inference (where inputs are iteratively generated by the model)—harms performance. We propose Multi-Step Denoising Scheduled Sampling (MDSS) to alleviate this bias. MDSS comprehensively considers the influence of prediction errors on both the model's noise output and the resulting input signal for the next step. By efficiently modeling multi-step error accumulation through a mathematical formulation, MDSS enables the model to learn self-correction. Experimental results show that MDSS effectively mitigates exposure bias in DDPM, DDIM, and DPM-solver, achieving a state-of-the-art FID score of 3.86 on CIFAR-10 with 100 DDIM steps.
Introduction: The Exposure Bias Problem
Denoising Diffusion Probabilistic Models (DDPMs) generate images by starting with pure noise and iteratively removing it. However, there is a fundamental mismatch between how they are trained and how they are used, known as exposure bias.
During Training, the model is shown a "clean" noisy image $x_t$ derived directly from a ground-truth image $x_0$. It learns to predict the noise added at that specific step. During Inference, the model starts from random noise and uses its own previous (imperfect) predictions to generate the input for the next step. Small errors in early steps compound over time, leading to a "drift" from the data distribution seen during training. While common in text generation, this issue has been largely overlooked in diffusion models until now.
Methodology: Multi-Step Denoising Scheduled Sampling (MDSS)
MDSS improves upon standard training by simulating realistic multi-step error accumulation, allowing the model to "see" and correct for its own mistakes during the training phase. The method introduces three key innovations:
- Simulating Real Errors: Instead of adding random Gaussian noise, MDSS uses the model's actual one-step prediction to introduce realistic "model-based" errors into the training samples.
- Multi-Step Accumulation: To simulate how errors compound without the massive computational cost of full iterative loops, the authors developed a mathematical formulation using the posterior distribution. This allows the model to efficiently calculate what a sample would look like after $k$ steps of imperfect inference.
- Dual-Channel Denoising: The model is modified to predict two components: the noise in the current step (standard) and a "correction" signal to denoise the input signal itself. This helps the model handle both its current prediction error and errors inherited from previous steps.
Crucially, MDSS can be applied as a fine-tuning step on top of pre-trained models, making it a computationally efficient enhancement.
Experiments and Results
The researchers evaluated MDSS on standard benchmarks: CIFAR-10, ImageNet 64×64, and LSUN Tower 64×64. The results show that MDSS consistently outperforms previous state-of-the-art methods like Input Perturbation (IP) and standard Scheduled Sampling (SS).
Error Mitigation Analysis
The core strength of MDSS is its ability to maintain low Mean Squared Error (MSE) even when inputs are corrupted by accumulated errors.
As shown in the figure above, MDSS ("Ours") consistently maintains significantly lower MSE across the sampling trajectory compared to baselines, particularly in the most challenging scenario (c) where errors have accumulated over multiple steps.
Quantitative Performance
- Superior Image Quality: On CIFAR-10 using the DDIM sampler (100 steps), MDSS achieved a Frechet Inception Distance (FID) of 3.86, significantly better than the second-best score of 4.78.
- Massive Gains in Low-Step Regimes: Exposure bias is most damaging when using fast samplers like DPM-Solver. With only 10 steps, MDSS improved the FID from 26.08 (standard) to 12.11, effectively doubling performance in ultra-fast generation scenarios.
Conclusion
MDSS provides a robust framework for alleviating exposure bias by modeling multi-step error accumulation. It is particularly effective for fast-sampling regimes (DDIM, DPM-Solver), making it a versatile "plug-and-play" enhancement for modern diffusion pipelines. Future work will explore adjustments for higher-order solvers and non-additive noise distributions.
Summary updated