Abstract
Standard Denoising Diffusion Probabilistic Models (DDPMs) primarily optimize for pixel-wise Mean Squared Error (MSE) on predicted noise, which often fails to capture human-perceived image quality. Integrating perceptual metrics like Learned Perceptual Image Patch Similarity (LPIPS) into diffusion training is challenging because the noise-prediction task and image-reconstruction task can interfere with each other. We present Cas-DM (Cascaded Diffusion Model), a novel two-stage architecture that decouples these objectives. By using a dedicated refinement module shielded from the primary generative backbone via gradient blocking, Cas-DM allows for the stable integration of advanced perceptual metrics. Our results across multiple benchmarks, including CIFAR10 and CelebAHQ, demonstrate significant improvements in image sharpness and a reduction in visual artifacts.
Introduction
Denoising Diffusion Probabilistic Models (DDPMs) have set new standards in image generation. However, their reliance on pixel-wise MSE loss during training often leads to blurry results or artifacts that do not match human visual perception. In contrast, perceptual metrics like LPIPS, which leverage deep network features, have been highly successful in GANs and Consistency Models for enhancing realism.
The challenge in applying LPIPS to diffusion models lies in the fundamental objective: DDPMs learn to predict noise ($\epsilon$) from a noisy input ($x_t$), whereas LPIPS requires a clean image ($x_0$). Previous attempts to predict both simultaneously using a shared backbone—often called "Dual Diffusion"—suffered from performance degradation, as the optimization of one task "polluted" the other, leading to instability and lower image quality.
The Cas-DM Architecture
To overcome these limitations, we propose a cascaded approach that separates noise prediction from image refinement into two distinct modules.
Decoupled Modules
Cas-DM consists of two primary components:
- The Noise Module ($\theta$): This module functions as a standard DDPM backbone, taking the noisy input $x_t$ and predicting the added noise $\epsilon'$.
- The Refinement Module ($\phi$): This module receives a rough estimate of the clean image (derived from the Noise Module) and refines it into a high-quality version $x'_0$. It also outputs a dynamic weight $r_t$ to balance the contributions of the two modules during the sampling process.
Gradient Blocking
A key innovation in Cas-DM is the use of a stop-gradient (sg) operation. By blocking gradients from the metric function (e.g., LPIPS) from reaching the Noise Module, we ensure that the primary generative process remains stable. The LPIPS loss is exclusively used to train the Refinement Module, allowing it to learn how to "polish" the generated images without compromising the underlying noise distribution modeling.
Experimental Results
We evaluated Cas-DM on several benchmarks, including CIFAR10, CelebAHQ, LSUN Bedroom, and ImageNet (64×64).
Quantitative Improvements
Cas-DM consistently outperformed standard DDPMs and Dual Diffusion baselines. For instance, on the CelebAHQ dataset, Cas-DM achieved an FID of 4.95, a significant improvement over the 6.34 achieved by a standard DDPM. Notably, unlike previous methods where adding perceptual losses sometimes worsened the FID, Cas-DM showed consistent gains across all tested benchmarks.
Qualitative Refinement
The impact of perceptual guidance is most visible in complex image regions. Standard diffusion models often struggle with fine textures or structured objects like eyeglasses, leading to "ghosting" or blurred details. Cas-DM, guided by VGG-based LPIPS, produces much sharper results with fewer artifacts.
Discussion and Insights
Our findings suggest that decoupling is essential for integrating perceptual metrics into diffusion models. The mismatch between noise-prediction and image-reconstruction tasks requires separate parameter sets. Furthermore, we observed that LPIPS based on the VGG architecture is particularly effective, likely because VGG's lack of residual connections preserves more semantic information useful for guiding the refinement process.
While Cas-DM introduces some computational overhead due to the additional module, the gains in image quality and training stability make it a compelling choice for high-fidelity generative tasks. Future work will explore scaling this architecture to higher resolutions and applying it to other modalities like video or audio.
Summary updated