Abstract
We introduce a Cascaded Diffusion Model (Cas-DM) that improves a Denoising Diffusion Probabilistic Model (DDPM) by effectively incorporating additional metric functions in training. Metric functions such as the LPIPS loss have been proven highly effective in consistency models derived from the score matching. However, for the diffusion counterparts, the methodology and efficacy of adding extra metric functions remain unclear. One major challenge is the mismatch between the noise predicted by a DDPM at each step and the desired clean image that the metric function works well on. To address this problem, we propose Cas-DM, a network architecture that cascades two network modules to effectively apply metric functions to the diffusion model training. The first module, similar to a standard DDPM, learns to predict the added noise and is unaffected by the metric function. The second cascaded module learns to predict the clean image, thereby facilitating the metric function computation. Experiment results show that the proposed diffusion model backbone enables the effective use of the LPIPS loss, improving the image quality (FID, sFID) of diffusion models on various established benchmarks.
Introduction
Standard Denoising Diffusion Probabilistic Models (DDPMs) typically learn to predict the noise ($\epsilon$) added to an image. While this approach is stable and has achieved remarkable success in generative modeling, it does not easily allow for the direct application of "metric functions"—perceptual losses like LPIPS that compare a predicted clean image to the ground truth. These metrics have been transformative for GANs and VAEs because they help models capture human-like semantic features and improve perceptual quality.
Integrating these metrics into diffusion models faces two primary obstacles. First, there is a mismatch: metric functions require a clean image ($x_0$), but DDPMs operate on noisy images ($x_t$) across many steps. Second, there is gradient interference: previous attempts to predict both noise and the clean image simultaneously (Dual Diffusion Models) often failed because the gradients from the metric function would "pollute" the noise prediction branch, leading to degraded overall performance.
Methodology: The Cascaded Diffusion Model (Cas-DM)
To address these challenges, we propose Cas-DM, which splits the generative task into two cascaded modules. This architecture allows the model to leverage perceptual signals without compromising the stability of the core denoising process.
Architecture Components
- Module $\theta$ (The Noise Predictor): This serves as a standard DDPM backbone. It takes the noisy image $x_t$ and predicts the added noise $\epsilon'$. This prediction is then used to derive an initial, "rough" estimate of the clean image ($x^*_0$).
- Module $\phi$ (The Refiner): This second network takes the rough estimate $x^*_0$ and the current timestep $t$ as input. It outputs a refined clean image ($x'_0$) and a dynamic mixing weight ($r_t$).
The "Stop-Gradient" Operation
The key innovation is the use of a "stop-gradient" ($sg$) operation between the two modules. During training, the metric function (e.g., LPIPS) is applied to the output of the refiner ($\phi$). By applying stop-gradient to the input of the refiner that comes from the noise predictor, we ensure that perceptual gradients update only the refiner $\phi$. This prevents the noise predictor $\theta$ from being "polluted" by the metric function's gradients, maintaining its specialized role in denoising.
Experiments and Results
We evaluated Cas-DM on four standard benchmarks: CIFAR10 (32×32), CelebAHQ (64×64), LSUN Bedroom (64×64), and ImageNet (64×64).
Quantitative Performance
Cas-DM consistently outperformed baseline DDPMs and Dual Diffusion models across all tested datasets. For instance, on CelebAHQ 64×64, Cas-DM with LPIPS achieved an FID of 4.95, a significant improvement over the standard DDPM (6.34). On ImageNet 64×64, the model achieved an FID of 27.54 without guidance, setting a strong baseline for non-guided models.
Qualitative Improvements
Visual comparisons demonstrate that models trained with LPIPS produce images with significantly fewer artifacts. The refinement is particularly noticeable in complex semantic areas such as eyeglasses, hair textures, and facial symmetry.
Key Insights
- Decoupling is Essential: Separate optimization paths are necessary because predicting noise and predicting semantic image features require different inductive biases.
- VGG Superiority: Interestingly, LPIPS based on the VGG network outperformed versions based on ResNet or Swin Transformers. VGG's lack of residual connections may force its features to represent more semantic information, providing a better training signal for the diffusion refiner.
- Efficiency: Cas-DM remains effective even with low sampling steps (e.g., 10 steps), showing larger performance gains over standard models in high-speed generation scenarios.
Conclusion
The Cascaded Diffusion Model (Cas-DM) provides a robust framework for incorporating perceptual metric functions into diffusion training. By decoupling noise prediction from image refinement through a cascaded design and stop-gradient operations, Cas-DM achieves state-of-the-art image quality across several standard benchmarks. While it introduces some computational overhead, the gains in perceptual quality and sampling efficiency make it a compelling architecture for future generative modeling research.
Summary updated