Efficient Conditional Diffusion Model with Probability Flow Sampling for Image Super-resolution

Yutao Yuan, Chun Yuan

Introduction

Image super-resolution (SR) is a fundamental yet ill-posed inverse problem in computer vision, where the goal is to reconstruct a high-resolution (HR) image from its low-resolution (LR) counterpart. Traditional methods, often optimized for pixel-wise metrics like Peak Signal-to-Noise Ratio (PSNR), frequently produce blurry results by averaging the manifold of possible HR solutions. While generative models such as Diffusion Probabilistic Models (DPMs) have demonstrated the ability to synthesize sharp, realistic details, they are typically hindered by high computational costs during inference and a tendency toward structural or color inconsistencies relative to the input LR image.

To address these limitations, this paper introduces the Efficient Conditional Diffusion Model with Probability Flow Sampling (ECDP). This framework enhances diffusion-based SR by optimizing for both inference speed and perceptual fidelity through a continuous-time formulation and a novel training objective.

Methodology

ECDP introduces three primary technical innovations designed to bridge the gap between generative quality and practical efficiency.

Continuous-Time SDE and Probability Flow

Rather than relying on discrete diffusion steps, ECDP models the image corruption process as a continuous-time Stochastic Differential Equation (SDE). By leveraging the corresponding Probability Flow Ordinary Differential Equation (ODE), the model can utilize efficient numerical solvers, such as Runge-Kutta methods. This transition from stochastic sampling to deterministic probability flow significantly reduces the number of iterations required to generate high-quality images.

Hybrid Parametrization

Standard diffusion models typically predict either the added noise ($\epsilon$-parametrization) or the clean image ($x_0$-parametrization). The authors observe that $\epsilon$-prediction excels at capturing fine details at low noise levels but struggles with global structure at high noise levels, leading to color shifting. ECDP employs a hybrid parametrization that dynamically interpolates between these two targets based on the current noise scale, ensuring structural consistency and natural color reproduction.

Overview of the ECDP framework, illustrating the hybrid-parametrization score predictor and the dual-loss training objective ($L_{quality}$ and $L_{score}$).

Adjoint-Method Perceptual Loss

To improve visual appeal, the framework incorporates a VGG-based perceptual loss ($L_{quality}$). Training with such a loss through an ODE solver is typically memory-prohibitive. ECDP circumvents this by using the adjoint method, which allows for gradient computation with constant memory cost, regardless of the number of sampling steps. This enables the model to be tuned for human-perceived quality directly within the diffusion training loop.

Experimental Evaluation

The model was evaluated on standard benchmarks for both general and domain-specific super-resolution:

  • General SR (4×): Evaluated on DIV2K (augmented with Flickr2K) and ImageNet.
  • Face SR (8×): Evaluated on CelebA to test the reconstruction of fine facial textures.

Key Results

  • Inference Speed: ECDP achieves a massive reduction in latency. It generates a 256×256 image in approximately 2.0 seconds, outperforming previous diffusion models like SR3 (83.1s) and IR-SDE (6.2s).
  • Perceptual Fidelity: On the DIV2K dataset, ECDP achieved an LPIPS score of 0.108, significantly better than SR3 (0.175) and SRDiff (0.136).
  • Consistency: Qualitative analysis shows that ECDP effectively eliminates common diffusion artifacts, such as grid patterns and color drifts, producing sharper architectural lines and more realistic skin and hair textures.

Discussion and Conclusion

ECDP represents a significant step forward in making diffusion-based super-resolution viable for real-world applications. By anchoring the forward process to the mean of the upscaled LR image and utilizing hybrid parametrization, the model maintains high fidelity to the source material while generating plausible high-frequency details.

While the 2-second inference time is a breakthrough for DPMs, it remains slower than feed-forward GAN or CNN architectures. Future work may focus on further reducing this latency and automating the tuning of the hybrid interpolation coefficients to generalize across a wider variety of datasets and degradation types.

Summary updated

Abstract

Image super-resolution is a fundamentally ill-posed problem because multiple valid high-resolution images exist for one low-resolution image. Super-resolution methods based on diffusion probabilistic models can deal with the ill-posed nature by learning the distribution of high-resolution images conditioned on low-resolution images, avoiding the problem of blurry images in PSNR-oriented methods. However, existing diffusion-based super-resolution methods have high time consumption with the use of iterative sampling, while the quality and consistency of generated images are less than ideal due to problems like color shifting. In this paper, we propose Efficient Conditional Diffusion Model with Probability Flow Sampling (ECDP) for image super-resolution. To reduce the time consumption, we design a continuous-time conditional diffusion model for image super-resolution, which enables the use of probability flow sampling for efficient generation. Additionally, to improve the consistency of generated images, we propose a hybrid parametrization for the denoiser network, which interpolates between the data-predicting parametrization and the noise-predicting parametrization for different noise scales. Moreover, we design an image quality loss as a complement to the score matching loss of diffusion models, further improving the consistency and quality of super-resolution. Extensive experiments on DIV2K, ImageNet, and CelebA demonstrate that our method achieves higher super-resolution quality than existing diffusion-based image super-resolution methods while having lower time consumption. Our code is available at https://github.com/Yuan-Yutao/ECDP.
BibTeX
@inproceedings{yuan2024efficient,
  author = {Yutao Yuan and Chun Yuan},
  title = {{Efficient Conditional Diffusion Model with Probability Flow Sampling for Image Super-resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=vD6ZqIyrjJ},
}
Figure