Debias the Training of Diffusion Models

Hu Yu, Li Shen, Jie Huang, Man Zhou, Hongsheng Li, Feng ZhaoSubmission to ICLR 2024 · Withdrawn

TL;DR

Standard diffusion models optimize an unweighted MSE loss predicting Gaussian noise, but the implicit error on the reconstructed image is amplified by $1/\sqrt{\text{SNR}(t)}$ as noise increases. The authors show that this constant weighting causes biased estimates, color shifts, and early-step error propagation. By adding a single line of code—weighting the noise loss by $1/\sqrt{\text{SNR}(t)}$—they significantly boost sample fidelity and training efficiency across multiple datasets.

What Problem Does This Paper Address?

Most modern diffusion models (like DDPM) train a neural network to predict the added Gaussian noise $\epsilon$ using a simple, unweighted mean squared error (MSE) across all timesteps $t$. While this simplified loss is easy to optimize, it masks an underlying distortion: the actual generative goal is recovering the clean image $x_0$.

Because noisy inputs are formed as $x_t = \sqrt{\alpha_t}x_0 + \sqrt{1 - \alpha_t}\epsilon$, reversing the equation to estimate $x_0$ yields: $$x_0 = \hat{x}_0 + \frac{1}{\sqrt{\text{SNR}(t)}}(\epsilon_\theta(x_t, t) - \epsilon)$$

When $t$ is large (the very noisy, initial generation steps), the Signal-to-Noise Ratio ($\text{SNR}(t)$) is tiny, which blows up the amplification coefficient $\frac{1}{\sqrt{\text{SNR}(t)}}$. Consequently, an imperceptibly small error in noise prediction translates into a massive error in the estimated image $\hat{x}_0$. This causes severe early-step artifacts and color casts that propagate through the entire sampling chain.

Methods

To counter this bias, the authors propose a debiased weighting strategy that directly accounts for the amplification coefficient: $$\mathcal{L} = \sum_t \mathbb{E}_{x_0, \epsilon} \left[ \frac{1}{\sqrt{\text{SNR}(t)}} \|\epsilon - \epsilon_\theta(x_t, t)\|^2 \right]$$

  • Implementation: It requires changing literally one line of code in existing training pipelines: python weight = 1.0 / torch.sqrt(SNR) loss = weight * (target - model_output) ** 2
  • Why not $1/\text{SNR}(t)$? Mathematically, weighting by $1/\text{SNR}(t)$ corresponds to directly penalizing $\|x_0 - \hat{x}_0\|^2$. However, its dynamic range is extreme ($10^{-4}$ to $10^{4}$), causing gradients from high-noise steps to overpower the batch, sacrificing fine-detail learning at small $t$ and leading to training divergence. The square-root formulation strikes a balanced compromise between explicit noise fitting and implicit image recovery.

Datasets

The authors evaluate their method on unconditional image generation benchmarks at $256 \times 256$ resolution:

  • FFHQ (~70,000 human face images)
  • AFHQ-dog (~50,000 animal face images)
  • MetFaces (~1,000 portrait art images)

No new datasets or benchmarks are introduced.

Key Insights and Ideas

  • Optimization asymmetry at large $t$: At high timesteps ($t > 600$), the input $x_t$ is almost entirely pure noise. Under constant weighting, the network can virtually "do nothing" and still log a tiny MSE loss. In fact, for $t > 950$, baseline network outputs deviate further from the target noise than the raw noisy inputs do.
  • Unifying prior heuristics: Earlier works like P2 and Min-SNR recognized that uniform weighting was problematic, but addressed it empirically by clamping or reducing weights at small $t$. The authors provide a principled foundation showing that weights should monotonically increase with timestep $t$ to counteract SNR decay.
  • Biased generation starts early: Visualizing the first few reverse sampling steps ($t \approx 999$) reveals that constant-weight models suffer from chaotic structure and severe color drift. Once introduced, these errors are difficult to fix in subsequent steps via error propagation. The debiased weighting eliminates artifacts from step one.

Key Results

  • Generation Quality (FID): On FFHQ (500k iterations, $T=1000$), debiased weighting reduces FID from 10.86 (ADM baseline) to 6.35, outperforming P2 (6.52) and Min-SNR (6.50). With 1,000k iterations, it reaches 4.97.
  • Dramatic Gains on Small Datasets: On MetFaces ($T=1000$), the FID plummets from 41.42 (constant weight) and 14.20 (P2) down to 9.17.
  • Accelerated Sampling: In few-step regimes, where early-step accuracy is critical, debiased models show their largest advantages. At $T=20$ sampling steps on FFHQ, FID drops from 41.27 (constant) to 29.38.
  • Training Efficiency: The proposed weighting reaches the FID milestone of a 1,000k-iteration baseline model in only 400k iterations, cutting compute requirements by more than half.

Limitations and Future Work

  • Theoretical trade-off: The choice of $1/\sqrt{\text{SNR}}$ instead of $1/\text{SNR}$ is motivated partly by empirical training stability rather than a pure first-principles derivation.
  • Scope of experimentation: Experiments were conducted on unconditional pixel-space diffusion (ADM backbone). Validating this strategy across large latent text-to-image models (e.g., Stable Diffusion) remains an open avenue.
  • Future directions: The authors suggest exploring how debiased weighting interacts with custom noise schedules and extending the insight into downstream applications such as image editing, inpainting, and restoration.

Summary updated

Abstract

Diffusion models have demonstrated compelling generation quality by optimizing the variational lower bound through a simple denoising score matching loss. In this paper, we provide theoretical evidence that the prevailing practice of using a constant loss weight strategy in diffusion models leads to biased estimation during the training phase. Simply optimizing the denoising network to predict Gaussian noise with constant weighting may hinder precise estimations of original images. To address the issue, we propose an elegant and effective weighting strategy grounded in the theoretically unbiased principle. Moreover, we conduct a comprehensive and systematic exploration to dissect the inherent bias problem deriving from constant weighting loss from the perspectives of its existence, impact and reasons. These analyses are expected to advance our understanding and demystify the inner workings of diffusion models. Through empirical evaluation, we demonstrate that our proposed debiased estimation method significantly enhances sample quality without the reliance on complex techniques, and exhibits improved efficiency compared to the baseline method both in training and sampling processes.
BibTeX
@misc{yu2023debias,
  author = {Hu Yu and Li Shen and Jie Huang and Man Zhou and Hongsheng Li and Feng Zhao},
  title = {{Debias the Training of Diffusion Models}},
  year = {2023},
  eprint = {2310.08442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.08442},
}

Loading related content...

Figure