Abstract
Diffusion Inverse Solvers (DIS) are designed to sample from the conditional distribution $p_{\theta}(X_0|y)$, with a predefined diffusion model $p_{\theta}(X_0)$, an operator $f(\cdot)$, and a measurement $y=f(x'_0)$ derived from an unknown image $x'_0$. Existing DIS estimate the conditional score function by evaluating $f(\cdot)$ with an approximated posterior sample drawn from $p_{\theta}(X_0|X_t)$. However, most prior approximations rely on the posterior means, which may not lie in the support of the image distribution, thereby potentially diverging from the appearance of genuine images. Such out-of-support samples may significantly degrade the performance of the operator $f(\cdot)$, particularly when it is a neural network. In this paper, we introduce a novel approach for posterior approximation that guarantees to generate valid samples within the support of the image distribution and enhances compatibility with neural network-based operators $f(\cdot)$. We first demonstrate that the solution of the Probability Flow Ordinary Differential Equation (PF-ODE) with an initial value $x_t$ yields an effective posterior sample $p_{\theta}(X_0|X_t=x_t)$. Based on this observation, we adopt the Consistency Model (CM), which is distilled from PF-ODE, for posterior sampling. Furthermore, we design a novel family of DIS using only CM. Through extensive experiments, we show that our proposed method for posterior sample approximation substantially enhances the effectiveness of DIS for neural network operators $f(\cdot)$ (e.g., in semantic segmentation). Additionally, our experiments demonstrate the effectiveness of the new CM-based inversion techniques.
Introduction: The "Blurry Mean" Problem
Diffusion models have emerged as powerful generative priors for solving inverse problems, where the goal is to recover an image $x_0$ from a measurement $y = f(x_0)$. Diffusion Inverse Solvers (DIS) typically approximate the conditional score function by using a "peek ahead" estimate of the clean image from a noisy state $x_t$.
A common approach, exemplified by Diffusion Posterior Sampling (DPS), utilizes the posterior mean $\mathbb{E}[x_0 | x_t]$ for this approximation. However, the posterior mean often results in blurry, out-of-distribution samples. When the operator $f(\cdot)$ is a non-linear neural network (e.g., for semantic segmentation), these blurry inputs lead to unreliable gradients, causing the solver to diverge from the true image manifold.
Methodology
To overcome the limitations of the posterior mean, this work proposes using Consistency Models (CM) for posterior approximation.
CM-based Posterior Approximation (Proposed I)
The authors prove that the solution of the Probability Flow ODE (PF-ODE) starting from $x_t$ provides a sample from the posterior distribution $p(x_0 | x_t)$. By leveraging CMs, which are distilled to approximate these ODE solutions, the solver can generate sharp, realistic image estimates in a single step. This ensures that the neural network operator $f(\cdot)$ receives "in-support" inputs, providing stable and accurate guidance. A small amount of Gaussian noise is added to the CM output to act as a regularizer and prevent overfitting to the neural network operator.
Consistency Model Inversion (Proposed II)
The paper also introduces a novel inversion technique that bypasses standard diffusion sampling. By treating the CM as a generative model, the solver optimizes the latent noise vector to minimize the measurement error $\|y - f(G(z))\|^2$. This approach offers a significant speed advantage over traditional diffusion-based solvers.
Experiments and Results
The proposed methods were evaluated on the LSUN Bedroom and LSUN Cat datasets at $256 \times 256$ resolution. The study focused on challenging guidance tasks involving neural network operators, such as semantic segmentation, room layout estimation, and image captioning.
Quantitative results confirm the superiority of the CM-based approach. In semantic segmentation on LSUN Bedroom, the proposed method achieved a mean Intersection over Union (mIOU) of 0.34, compared to 0.27 for the DPS baseline. Furthermore, image quality as measured by FID improved across tasks; for example, the captioning task saw FID drop from 9.49 to 8.16. The inversion-based method (Proposed II) was found to be approximately 50% faster than DPS while maintaining high reconstruction quality.
Conclusion
This research demonstrates that maintaining image support during the guidance process is critical for the success of Diffusion Inverse Solvers, especially when using neural network-based operators. By employing Consistency Models for posterior approximation, the proposed methods achieve state-of-the-art performance in complex inverse tasks while offering improvements in both image fidelity and computational efficiency.
Summary updated