Eliminating Lipschitz Singularities in Diffusion Models

Zhantao Yang, Ruili Feng, Han Zhang, Yujun Shen, Kai Zhu, Lianghua Huang, Yifei Zhang, Yu Liu, Deli Zhao, Jingren Zhou, Fan Cheng

Abstract

Diffusion models have emerged as a powerful paradigm for high-fidelity image generation. However, they suffer from an inherent instability known as a "Lipschitz singularity" near the final stages of denoising ($t \to 0$). This paper identifies that the rapid change in standard noise schedules causes the denoiser's Lipschitz constant to explode, leading to training difficulties and significant discretization errors during inference. We introduce the Early Timestep-shared Diffusion Model (E-TSDM), a simple yet effective modification that shares timestep conditions across small intervals near zero. E-TSDM stabilizes the denoising path, achieving a 33% improvement in sampling quality for fast samplers and setting a new state-of-the-art FID of 4.22 on the FFHQ dataset.

Introduction

Diffusion models have revolutionized generative AI by transforming noise into realistic images through a reverse-time Stochastic Differential Equation (SDE). The stability of this process depends heavily on the "smoothness" of the neural denoiser. Mathematically, a low Lipschitz constant with respect to time ensures that small changes in the timestep $t$ produce predictable changes in output. However, researchers have identified a critical "Lipschitz singularity" where this constant explodes as the process approaches the clean image ($t=0$).

The Lipschitz Singularity Problem

The root cause of this instability lies in the noise schedules (e.g., linear, cosine) used by popular models like DDPM and Stable Diffusion. As $t$ approaches zero, the derivative of the noise schedule $d\sigma_t/dt$ tends toward infinity. This makes the network hyper-sensitive to time conditions near the end of the denoising process.

Lipschitz constants of DDPM vs. E-TSDM across timesteps. The standard DDPM (green) shows an explosion in the Lipschitz constant as timesteps decrease toward zero, while E-TSDM (purple) remains stable.

This singularity leads to significant discretization errors, particularly for "fast samplers" like DDIM or DPM-Solver that take large temporal steps. The resulting instability can manifest as training difficulties and reduced image quality.

The E-TSDM Method

To address this, the authors introduce the Early Timestep-shared Diffusion Model (E-TSDM). Instead of unique conditioning for every infinitesimal step near zero, E-TSDM identifies a "danger zone" interval $[0, \tilde{t})$. This interval is divided into $n$ sub-intervals, each sharing a single timestep condition (typically the left endpoint).

Comparison of the standard DDPM procedure and the proposed E-TSDM method. E-TSDM shares timestep conditions across small windows near $t=0$ to eliminate the singularity.

By forcing the network to use the same time condition over these windows, the derivative of the output with respect to time is effectively zeroed out, "killing" the singularity. This algorithmic change requires no additional parameters or computational overhead, making it a purely logic-based improvement to the training and inference process.

Experimental Results

E-TSDM was validated across a wide variety of standard image synthesis benchmarks, including FFHQ, CelebAHQ, and LSUN datasets. The quantitative improvements are substantial:

  • State-of-the-Art Performance: E-TSDM achieved a new SOTA FID of 4.22 on FFHQ 256×256, outperforming established models like LDM and P2-DM.
  • Massive Gains for Fast Samplers: When using acceleration methods like DDIM and DPM-Solver, E-TSDM reduced the FID by over 33%, as the smoother denoising path is more amenable to large temporal steps.
  • Robustness to Noise: The method demonstrated significantly lower prediction errors (up to 42% reduction) when inputs were subjected to small perturbations, indicating a more stable learned manifold.
  • Color Bias Mitigation: In super-resolution tasks, E-TSDM was found to mitigate the "color bias" (off-tinted hues) often seen in baseline models.

Conclusion

By identifying and eliminating Lipschitz singularities, E-TSDM provides a simple yet powerful fix for diffusion model instability. Its ability to improve both image quality and sampling speed without increasing complexity makes it a highly practical advancement for the field of generative modeling. Future work may explore its impact on massive, billion-parameter text-to-image models and further refine the optimal interval sharing strategies.

Summary updated

BibTeX
@inproceedings{yang2024eliminating,
  author = {Zhantao Yang and Ruili Feng and Han Zhang and Yujun Shen and Kai Zhu and Lianghua Huang and Yifei Zhang and Yu Liu and Deli Zhao and Jingren Zhou and Fan Cheng},
  title = {{Eliminating Lipschitz Singularities in Diffusion Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=WNkW0cOwiz},
}
Figure