Conditional Velocity Score Estimation for Image Restoration

Ziqiang Shi, Rujie LiuWACV 2024

TL;DR

This paper introduces Conditional Velocity Score Estimation (CVSA), a novel image restoration method that models image recovery in a "position-velocity" space. By explicitly incorporating velocity into the diffusion process, CVSA achieves state-of-the-art results across various linear and non-linear, blind and non-blind image restoration tasks without requiring task-specific training.

What Problem Does This Paper Address?

Image restoration, the process of recovering a high-quality image from a degraded version, has seen significant advancements with diffusion generative models (DGMs). These models typically use the degraded image as a conditional input to guide the diffusion process from noise towards a realistic image. However, existing methods primarily operate in the "position space" of the image, neglecting the dynamics of the image's evolution during restoration. This paper argues that modeling the "velocity" (or momentum) of the image's diffusion path can lead to more effective and faster restoration. The challenge lies in accurately estimating the conditional velocity score, which is intractable, and integrating it into the restoration process.

Methods

The core innovation of CVSA is the introduction of a position-velocity space, inspired by critically-damped Langevin diffusion (CLD). The restoration process is framed as a reverse diffusion process in this coupled position-velocity space. The key technical challenge is approximating the intractable conditional velocity score (CVS). CVSA addresses this by decomposing the CVS into two parts: an unconditional velocity score (readily available from a pre-trained DGM) and an observation velocity score (OVS).

The OVS is approximated using a novel approach based on Bayesian principles and an improved Jensen's inequality. This approximation effectively transfers the information from the degraded measurement at the current time step to the initial state, allowing for the estimation of the OVS. The method then uses this approximated CVS to guide the reverse diffusion process, updating both the image position and its velocity. CVSA is designed to be versatile, handling both linear and non-linear degradation operators and even blind restoration scenarios where the degradation kernel is unknown.

Datasets

The paper evaluates CVSA on four standard datasets: CelebA-HQ, AFHQ, FFHQ, and Products-6k. These datasets are commonly used for image restoration tasks and contain diverse image content, including human faces and products.

Key Insights and Ideas

  • Position-Velocity Space: Moving beyond just image position, explicitly modeling the velocity of the diffusion process offers a new dimension for controlling and accelerating image restoration.
  • Conditional Velocity Score Approximation (CVSA): The clever decomposition of the intractable CVS into an unconditional score and an approximated observation score is the key enabler for the method.
  • Jensen's Inequality for OVS: The application of Jensen's inequality to approximate the observation velocity score is a novel mathematical tool that bridges the gap between the degraded measurement and the diffusion state.
  • Task-Agnostic Restoration: The ability of a single model to perform well across a wide range of restoration tasks (deblurring, inpainting, super-resolution, phase retrieval, blind deblurring) without task-specific training is a significant advantage.

Key Results

CVSA demonstrates state-of-the-art (SOTA) performance across multiple linear and non-linear restoration tasks.

  • Quantitative: On FFHQ, CelebA-HQ, and AFHQ-dog, CVSA shows significant improvements in PSNR, SSIM, and FID compared to existing SOTA methods like DPS and ILVR. For instance, it achieves a relative improvement of 5% in SSIM and a 10% decrease in FID compared to DPS. For blind deblurring, CVSA also outperforms the previous SOTA method BlindDPS, though the advantage is less pronounced.
  • Qualitative: Visual comparisons show that CVSA produces restored images that are more authentic and consistent with the degraded input, even in challenging cases like phase retrieval where other methods fail. The method also shows promise in real-world applications like product recognition for automatic checkout.

Limitations and Future Work

The authors acknowledge that the observation velocity score approximation provides a lower bound of the true value. They suggest that more accurate estimates could be obtained by incorporating the remainder of the Jensen's inequality estimate, potentially through Monte Carlo sampling. This is identified as a promising direction for future research. Additionally, while CVSA shows good performance in blind deblurring, further improvements are sought for this specific task. The paper also notes that the application of CVSA could be extended to other blind restoration tasks, such as imaging through turbulence.

Summary updated

BibTeX
@inproceedings{shi2024conditional,
  author = {Ziqiang Shi and Rujie Liu},
  title = {{Conditional Velocity Score Estimation for Image Restoration}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure