Abstract: Diffusion models have demonstrated compelling generation quality by optimizing the variational lower bound through a simple denoising score matching loss. In this paper, we provide theoretical evidence that the prevailing practice of using a constant loss weight strategy in diffusion models leads to biased estimation during the training phase. Simply optimizing the denoising network to predict Gaussian noise with constant weighting may hinder precise estimations of original images. To address the issue, we propose an elegant and effective weighting strategy grounded in the theoretically unbiased principle. Moreover, we conduct a comprehensive and systematic exploration to dissect the inherent bias problem deriving from constant weighting loss from the perspectives of its existence, impact and reasons. These analyses are expected to advance our understanding and demystify the inner workings of diffusion models. Through empirical evaluation, we demonstrate that our proposed debiased estimation method significantly enhances sample quality without the reliance on complex techniques, and exhibits improved efficiency compared to the baseline method both in training and sampling processes.
Introduction: The Hidden Bias in Diffusion Training
Denoising Diffusion Probabilistic Models (DDPMs) are the engines behind modern generative AI, yet they often suffer from two persistent issues: slow training convergence and "color shift." Color shift is a phenomenon where generated images exhibit unnatural global tints or a hazy, low-contrast appearance, particularly when using fewer sampling steps. This paper identifies the root cause as a fundamental bias in the standard training objective. While it is common practice to use a constant loss weight across all timesteps $t$, we demonstrate that this leads to poor image reconstruction at high noise levels.
Theoretical Analysis of Error Amplification
The core issue lies in the relationship between the predicted noise $\epsilon_\theta$ and the estimated original image $\hat{x}_0$. We derive that the error in the estimated image is proportional to the noise prediction error, scaled by an amplification factor:
$$\text{Error} \propto \frac{1}{\sqrt{SNR(t)}} (\epsilon_\theta - \epsilon)$$
As the timestep $t$ increases, the Signal-to-Noise Ratio ($SNR$) approaches zero, causing the $1/\sqrt{SNR(t)}$ coefficient to grow exponentially. Consequently, even a minor inaccuracy in noise prediction at high-noise timesteps results in a massive distortion in the estimated clean image. At very high noise levels ($t > 950$), standard models often perform worse than if they did nothing at all, as the loss is too small for the model to learn to identify the subtle signal buried in the noise.
The Debiased Estimation Strategy
To counteract this bias, we propose a simple, theoretically grounded weighting strategy. Instead of treating all timesteps equally, the training loss is re-weighted by the inverse square root of the SNR: $1/\sqrt{SNR(t)}$. This "debiased" objective focuses the model's learning on the timesteps where errors have the most significant impact on the final image structure. This fix is "plug-and-play," requiring no changes to the U-Net architecture or the sampling algorithm; it only changes a single line of code in the training loop.
Empirical Results and Efficiency
The proposed method was validated across standard benchmarks including FFHQ, CelebA-HQ, and ImageNet. The results show significant improvements in both image quality and training efficiency. For instance, on the FFHQ dataset, the Fréchet Inception Distance (FID) score—a key measure of image quality—dropped from 10.86 (standard) to 6.35 (debiased).
Beyond quality, the debiased model is remarkably efficient. It achieves better results in 400k iterations than the baseline does in 1,000k. Furthermore, it enables high-quality generation in as few as 20 sampling steps, effectively mitigating the structural and color biases that plague standard models.
Conclusion
By identifying and correcting the inherent bias in constant-weight diffusion training, this work provides a robust mathematical foundation for improving generative models. The proposed $1/\sqrt{SNR(t)}$ weighting strategy offers a simple yet powerful tool for achieving higher fidelity and faster convergence in diffusion-based generation without the need for complex auxiliary techniques.
Summary updated