Zero-Shot Inverse Problem Solving with Diffusion Models

232 concepts170 papers1 proposal

The Core Dilemma of Inverse Problems

Many fundamental tasks in computer vision are inverse problems. You are given a degraded observation $y$ (a blurry image, a low-resolution photo, or a corrupted medical scan) and want to recover the original, clean image $x$. Mathematically, the forward degradation process can be written as:

$$y = \mathcal{A}(x) + \mathbf{n}$$

where $\mathcal{A}$ is a degradation operator (like blurring, downsampling, or masking) and $\mathbf{n}$ is noise.

Solving this is notoriously difficult because inverse problems are ill-posed: multiple distinct clean images $x$ could produce the exact same degraded image $y$. Traditionally, deep learning tackled this by training a specialized neural network to directly map $y \to x$. However, this approach requires a massive dataset of paired examples for every specific degradation. If the blur kernel changes slightly, or if you want to switch from super-resolution to denoising, you have to train an entirely new model from scratch.

The zero-shot paradigm completely flips this strategy. Instead of training a network to learn the mapping from dirty to clean, it leverages a frozen, pre-trained generative model that already knows what clean images look like, guiding its generation process at test time using the measurement $y$.


Decoupling Prior Knowledge from Degradation Constraints

To understand why zero-shot inverse problem solving works, we must separate the reconstruction process into two independent pieces:

  1. The Generative Prior ($p(x)$): An understanding of the natural image manifold. What textures, edges, and structures make an image look real?
  2. The Data-Fidelity Constraint ($p(y|x)$): The physical law or mathematical operator that explains how the clean image was corrupted.

As demonstrated by Kadkhodaie et al., a deep network trained simply to remove white noise implicitly learns the score function ($\nabla_x \log p(x)$) of the natural image distribution. Denoising diffusion models scale this concept up dramatically, capturing rich, large-scale structural priors.

To solve an inverse problem, we want to sample from the posterior distribution $p(x|y)$. By Bayes' rule, the score of the posterior can be broken down into:

$$\nabla_x \log p(x|y) = \nabla_x \log p(x) + \nabla_x \log p(y|x)$$

The first term is provided entirely by the frozen, pre-trained diffusion model. The second term is a guidance loss that forces the generated image to match our observation $y$. Garber et al. formalize this by incorporating a data-fidelity term exclusively during the inference stage. This allows a single unconditional model to generalize across various linear degradation operators without any task-specific retraining.

[Pure Noise] ---> ( Reverse Diffusion Process ) ---> [Clean Image x]
                         ^
                         | (Iterative Guidance)
                 [Observation y = A(x)]

The practical challenge lies in how we enforce the data-fidelity constraint $\nabla_x \log p(y|x)$ during the step-by-step reverse diffusion process.

For linear degradation operators, the measurement process destroys information in a specific subspace of the image. Cheng et al. demonstrate that the reconstruction space can be cleanly divided into a range space (the components of the image preserved by $\mathcal{A}$) and a null space (the components destroyed by $\mathcal{A}$). During each step of the reverse diffusion process, the range space is explicitly replaced or corrected using the observed data $y$, while the diffusion model is left free to generate and refine the missing details within the null space. This ensures perfect mathematical consistency with the observation while leveraging the prior to fill in the blanks.

But what happens when the degradation operator $\mathcal{A}$ is non-linear or non-differentiable, such as JPEG compression? Standard gradient descent through the operator breaks down.

To solve this, Kawar et al. introduced the concept of utilizing a "pseudo-inverse" operation within the diffusion loop, treating non-linear distortions as localized linear approximations. Expanding on this direction, Song proposed Pseudoinverse-guided Diffusion Models ($\Pi$GDM). The author proved that by steering the generative process with mathematically consistent pseudoinverse terms, general-purpose diffusion models can match or even exceed the performance of specialized, task-specific architectures on complex real-world signal degradations.


Restoring Images with Unknown Degradations

The ultimate test of this paradigm is blind image restoration, where the forward operator $\mathcal{A}$ is completely unknown or "open-set."

When a closed-form formula for the distortion does not exist, Xiao et al. show that deep diffusion priors can perform blind recovery by navigating the latent space directly toward high-probability regions of the natural image manifold. Because artifacts like weather noise or compression blockiness live outside the distribution of clean natural images, the diffusion model naturally rejects them as out-of-distribution noise, regenerating clean underlying textures.

For highly complex, real-world scenarios where the degradation shifts dynamically, Zhao introduced a test-time degradation adaptation framework. Instead of modifying the massive, frozen generative prior, a lightweight localized adapter learns the specific degradation characteristics of a single test image on the fly. This allows the system to remain robust to novel distribution shifts while preserving the foundational scene-understanding priors of the base model.

This flexibility even allows for unconventional applications like arbitrary-scale super-resolution. Li et al. demonstrated that by injecting a mathematically calibrated amount of noise into a low-resolution image, one can initiate the reverse diffusion process from an intermediate timestep. The model treats the upscaled image as a partially denoised state, using its internal generative priors to synthesize high-frequency details across non-integer scaling factors that were never explicitly seen during training.


Why It Works: The Intuition

Traditional supervised models act like specialized bridges: they memorize a direct path from a specific type of corrupted image back to a clean one. If the starting point changes even slightly, the bridge collapses.

Zero-shot diffusion solvers work because they treat the pre-trained generative model as a map of the natural image world. The reverse diffusion process is a walk along this high-probability manifold. The data-fidelity term acts as a compass, pulling the walk toward the specific slice of the manifold that aligns with the degraded observation $y$.

Because the map ($p(x)$) is completely decoupled from the compass ($p(y|x)$), you can swap out the compass at inference time to solve entirely different problems—whether it is deblurring, super-resolution, or artifact correction—without ever changing the underlying map.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_zero_shot_inverse_problem_solving_61689ad5,
  title = {zero-shot inverse problem solving},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-inverse-problem-solving/rdr_61689ad5258a4fa888a26e752fb87ed9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_zero_shot_inverse_problem_solving_61689ad5_v1,
  title = {zero-shot inverse problem solving},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-inverse-problem-solving/rdr_61689ad5258a4fa888a26e752fb87ed9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake