Consistency Models Improve Diffusion Inverse Solvers

Tongda Xu, Ziran Zhu, Dailan He, Yuanyuan Wang, Ming Sun, Ning Li, Hongwei Qin, Yan Wang, Jingjing Liu, Ya-Qin ZhangarXiv 2024 · arXiv · OpenReview

Abstract

Diffusion Inverse Solvers (DIS) are designed to sample from the conditional distribution $p_{\theta}(X_0|y)$, with a predefined diffusion model $p_{\theta}(X_0)$, an operator $f(\cdot)$, and a measurement $y=f(x'_0)$ derived from an unknown image $x'_0$. Existing DIS estimate the conditional score function by evaluating $f(\cdot)$ with an approximated posterior sample drawn from $p_{\theta}(X_0|X_t)$. However, most prior approximations rely on the posterior means, which may not lie in the support of the image distribution, thereby potentially diverging from the appearance of genuine images. Such out-of-support samples may significantly degrade the performance of the operator $f(\cdot)$, particularly when it is a neural network. In this paper, we introduce a novel approach for posterior approximation that guarantees to generate valid samples within the support of the image distribution and enhances compatibility with neural network-based operators $f(\cdot)$. We first demonstrate that the solution of the Probability Flow Ordinary Differential Equation (PF-ODE) with an initial value $x_t$ yields an effective posterior sample $p_{\theta}(X_0|X_t=x_t)$. Based on this observation, we adopt the Consistency Model (CM), which is distilled from PF-ODE, for posterior sampling. Furthermore, we design a novel family of DIS using only CM. Through extensive experiments, we show that our proposed method for posterior sample approximation substantially enhances the effectiveness of DIS for neural network operators $f(\cdot)$ (e.g., in semantic segmentation). Additionally, our experiments demonstrate the effectiveness of the new CM-based inversion techniques.

Introduction: The "Blurry Mean" Problem

Diffusion models have emerged as powerful generative priors for solving inverse problems, where the goal is to recover an image $x_0$ from a measurement $y = f(x_0)$. Diffusion Inverse Solvers (DIS) typically approximate the conditional score function by using a "peek ahead" estimate of the clean image from a noisy state $x_t$.

A common approach, exemplified by Diffusion Posterior Sampling (DPS), utilizes the posterior mean $\mathbb{E}[x_0 | x_t]$ for this approximation. However, the posterior mean often results in blurry, out-of-distribution samples. When the operator $f(\cdot)$ is a non-linear neural network (e.g., for semantic segmentation), these blurry inputs lead to unreliable gradients, causing the solver to diverge from the true image manifold.

Posterior approximation comparison: the blurry posterior mean leads to failed segmentation, whereas the sharp PF-ODE (Consistency Model) approximation enables accurate task performance.

Methodology

To overcome the limitations of the posterior mean, this work proposes using Consistency Models (CM) for posterior approximation.

CM-based Posterior Approximation (Proposed I)

The authors prove that the solution of the Probability Flow ODE (PF-ODE) starting from $x_t$ provides a sample from the posterior distribution $p(x_0 | x_t)$. By leveraging CMs, which are distilled to approximate these ODE solutions, the solver can generate sharp, realistic image estimates in a single step. This ensures that the neural network operator $f(\cdot)$ receives "in-support" inputs, providing stable and accurate guidance. A small amount of Gaussian noise is added to the CM output to act as a regularizer and prevent overfitting to the neural network operator.

Consistency Model Inversion (Proposed II)

The paper also introduces a novel inversion technique that bypasses standard diffusion sampling. By treating the CM as a generative model, the solver optimizes the latent noise vector to minimize the measurement error $\|y - f(G(z))\|^2$. This approach offers a significant speed advantage over traditional diffusion-based solvers.

Experiments and Results

The proposed methods were evaluated on the LSUN Bedroom and LSUN Cat datasets at $256 \times 256$ resolution. The study focused on challenging guidance tasks involving neural network operators, such as semantic segmentation, room layout estimation, and image captioning.

Qualitative results across various tasks. The proposed CM-based approximation (right columns) consistently produces more realistic images that better satisfy the target constraints compared to the posterior mean baseline.

Quantitative results confirm the superiority of the CM-based approach. In semantic segmentation on LSUN Bedroom, the proposed method achieved a mean Intersection over Union (mIOU) of 0.34, compared to 0.27 for the DPS baseline. Furthermore, image quality as measured by FID improved across tasks; for example, the captioning task saw FID drop from 9.49 to 8.16. The inversion-based method (Proposed II) was found to be approximately 50% faster than DPS while maintaining high reconstruction quality.

Conclusion

This research demonstrates that maintaining image support during the guidance process is critical for the success of Diffusion Inverse Solvers, especially when using neural network-based operators. By employing Consistency Models for posterior approximation, the proposed methods achieve state-of-the-art performance in complex inverse tasks while offering improvements in both image fidelity and computational efficiency.

Summary updated

Abstract

Diffusion Inverse Solvers (DIS) are designed to sample from the conditional distribution $p_{\theta}(X_0|y)$, with a predefined diffusion model $p_{\theta}(X_0)$, an operator $f(\cdot)$, and a measurement $y=f(x'_0)$ derived from an unknown image $x'_0$. Existing DIS estimate the conditional score function by evaluating $f(\cdot)$ with an approximated posterior sample drawn from $p_{\theta}(X_0|X_t)$. However, most prior approximations rely on the posterior means, which may not lie in the support of the image distribution, thereby potentially diverge from the appearance of genuine images. Such out-of-support samples may significantly degrade the performance of the operator $f(\cdot)$, particularly when it is a neural network. In this paper, we introduces a novel approach for posterior approximation that guarantees to generate valid samples within the support of the image distribution, and also enhances the compatibility with neural network-based operators $f(\cdot)$. We first demonstrate that the solution of the Probability Flow Ordinary Differential Equation (PF-ODE) with an initial value $x_t$ yields an effective posterior sample $p_{\theta}(X_0|X_t=x_t)$. Based on this observation, we adopt the Consistency Model (CM), which is distilled from PF-ODE, for posterior sampling. Furthermore, we design a novel family of DIS using only CM. Through extensive experiments, we show that our proposed method for posterior sample approximation substantially enhance the effectiveness of DIS for neural network operators $f(\cdot)$ (e.g., in semantic segmentation). Additionally, our experiments demonstrate the effectiveness of the new CM-based inversion techniques. The source code is provided in the supplementary material.
BibTeX
@misc{xu2024consistency,
  author = {Tongda Xu and Ziran Zhu and Dailan He and Yuanyuan Wang and Ming Sun and Ning Li and Hongwei Qin and Yan Wang and Jingjing Liu and Ya-Qin Zhang},
  title = {{Consistency Models Improve Diffusion Inverse Solvers}},
  year = {2024},
  eprint = {2403.12063},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.12063},
}
Figure