Bring Metric Functions into Diffusion Models

Jie An, Zhengyuan Yang, Jianfeng Wang, Linjie Li, Zicheng Liu, Lijuan Wang, Jiebo Luo

Abstract

Standard Denoising Diffusion Probabilistic Models (DDPMs) primarily optimize for pixel-wise Mean Squared Error (MSE) on predicted noise, which often fails to capture human-perceived image quality. Integrating perceptual metrics like Learned Perceptual Image Patch Similarity (LPIPS) into diffusion training is challenging because the noise-prediction task and image-reconstruction task can interfere with each other. We present Cas-DM (Cascaded Diffusion Model), a novel two-stage architecture that decouples these objectives. By using a dedicated refinement module shielded from the primary generative backbone via gradient blocking, Cas-DM allows for the stable integration of advanced perceptual metrics. Our results across multiple benchmarks, including CIFAR10 and CelebAHQ, demonstrate significant improvements in image sharpness and a reduction in visual artifacts.

Introduction

Denoising Diffusion Probabilistic Models (DDPMs) have set new standards in image generation. However, their reliance on pixel-wise MSE loss during training often leads to blurry results or artifacts that do not match human visual perception. In contrast, perceptual metrics like LPIPS, which leverage deep network features, have been highly successful in GANs and Consistency Models for enhancing realism.

The challenge in applying LPIPS to diffusion models lies in the fundamental objective: DDPMs learn to predict noise ($\epsilon$) from a noisy input ($x_t$), whereas LPIPS requires a clean image ($x_0$). Previous attempts to predict both simultaneously using a shared backbone—often called "Dual Diffusion"—suffered from performance degradation, as the optimization of one task "polluted" the other, leading to instability and lower image quality.

The Cas-DM Architecture

To overcome these limitations, we propose a cascaded approach that separates noise prediction from image refinement into two distinct modules.

Decoupled Modules

Cas-DM consists of two primary components:

  1. The Noise Module ($\theta$): This module functions as a standard DDPM backbone, taking the noisy input $x_t$ and predicting the added noise $\epsilon'$.
  2. The Refinement Module ($\phi$): This module receives a rough estimate of the clean image (derived from the Noise Module) and refines it into a high-quality version $x'_0$. It also outputs a dynamic weight $r_t$ to balance the contributions of the two modules during the sampling process.
The Cas-DM architecture separates the generative backbone ($\theta$) from the refinement module ($\phi$). A stop-gradient (sg) operation ensures that the perceptual metric loss only updates the refinement module, maintaining the stability of the core diffusion process.

Gradient Blocking

A key innovation in Cas-DM is the use of a stop-gradient (sg) operation. By blocking gradients from the metric function (e.g., LPIPS) from reaching the Noise Module, we ensure that the primary generative process remains stable. The LPIPS loss is exclusively used to train the Refinement Module, allowing it to learn how to "polish" the generated images without compromising the underlying noise distribution modeling.

Experimental Results

We evaluated Cas-DM on several benchmarks, including CIFAR10, CelebAHQ, LSUN Bedroom, and ImageNet (64×64).

Quantitative Improvements

Cas-DM consistently outperformed standard DDPMs and Dual Diffusion baselines. For instance, on the CelebAHQ dataset, Cas-DM achieved an FID of 4.95, a significant improvement over the 6.34 achieved by a standard DDPM. Notably, unlike previous methods where adding perceptual losses sometimes worsened the FID, Cas-DM showed consistent gains across all tested benchmarks.

Qualitative Refinement

The impact of perceptual guidance is most visible in complex image regions. Standard diffusion models often struggle with fine textures or structured objects like eyeglasses, leading to "ghosting" or blurred details. Cas-DM, guided by VGG-based LPIPS, produces much sharper results with fewer artifacts.

Qualitative comparison highlighting the reduction of artifacts in complex areas such as eyeglasses. Cas-DM produces sharper, more defined features compared to standard MSE-based models.

Discussion and Insights

Our findings suggest that decoupling is essential for integrating perceptual metrics into diffusion models. The mismatch between noise-prediction and image-reconstruction tasks requires separate parameter sets. Furthermore, we observed that LPIPS based on the VGG architecture is particularly effective, likely because VGG's lack of residual connections preserves more semantic information useful for guiding the refinement process.

While Cas-DM introduces some computational overhead due to the additional module, the gains in image quality and training stability make it a compelling choice for high-fidelity generative tasks. Future work will explore scaling this architecture to higher resolutions and applying it to other modalities like video or audio.

Summary updated

BibTeX
@inproceedings{an2024bring,
  author = {Jie An and Zhengyuan Yang and Jianfeng Wang and Linjie Li and Zicheng Liu and Lijuan Wang and Jiebo Luo},
  title = {{Bring Metric Functions into Diffusion Models}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=plaZPDqF7Y},
}
Figure