Zero-Shot Image Restoration with Pre-trained Generative Priors

This direction is split into other directions: zero-shot inverse problem solving. Its last version stays readable and citable here.

Version 1 of 1292 papers360 conceptsfrozen 2026-10-04

Traditional image restoration—fixing blur, noise, or low resolution—usually requires training a specific model for a specific type of damage. If you want to fix JPEG artifacts, you train on JPEG images; if you want to upscale, you train on low-resolution pairs.

A new paradigm has emerged that flips this approach: instead of training a model to fix damage, researchers use "frozen" generative models that already know what clean images look like. By treating restoration as a "zero-shot" inverse problem, a single pre-trained model can solve almost any degradation without ever being fine-tuned for it.

The Inverse Problem Framing

To understand why this works, we have to look at restoration as a Bayesian inverse problem. We have a degraded observation $y$ (the blurry or noisy image) and we want to recover the original high-quality image $x$. This is typically modeled as: $$y = \mathcal{A}(x) + n$$ where $\mathcal{A}$ is a degradation operator (like blurring or downsampling) and $n$ is noise.

As Bahjat Kawar and Michael Elad explain in Denoising Diffusion Restoration Models, the challenge is that many different clean images $x$ could have produced the same degraded image $y$. To pick the "right" one, we need a prior: a mathematical understanding of what a natural, clean image looks like.

Decoupling Content from Corruption

The breakthrough in this field is the realization that large-scale generative models, like those described by Yang Song and Stefano Ermon in Score-Based Generative Modeling through Stochastic Differential Equations, are essentially "universal priors." These models have learned the manifold of all natural images during their initial training.

Because these models already "know" textures, edges, and shapes, we don't need to teach them how to dehaze or deblur. We only need to guide their generation process so that the output remains consistent with the degraded input $y$. This decoupling allows a model trained only on clean images to restore highly compressed JPEGs without task-specific retraining.

Guidance through Data Consistency

The core mechanism for zero-shot restoration is Data Consistency (DC). During the generative process (like the reverse diffusion steps), the model proposes a cleaner version of the image. We then "force" this proposal to match the observed data $y$.

There are three primary ways researchers achieve this:

  1. Spectral Projection: In DDRM, the authors perform diffusion in the spectral space of the degradation operator. By decomposing the image into singular values, they can replace the parts of the image that are "known" (from the observation $y$) and let the diffusion model "fill in" the parts lost to degradation (the null-space).
  2. Posterior Sampling: Methods like those proposed by J. D. Young in Diffusion Posterior Sampling use the gradient of a distance function $\|y - \mathcal{A}(\hat{x})\|^2$ to nudge the generative trajectory toward the observation.
  3. Null-Space Refinement: Xinhua Cheng and Jiwen Yu demonstrate in Null-Space Diffusion Sampling that by only updating the "null-space" (the information missing from the input), the model ensures perfect consistency with the observed parts of a 3D point cloud or image.

Plug and Play Optimization

This direction also bridges the gap between modern AI and classical signal processing. Chicago Y. Park and Ulugbek S. Kamilov showed in Plug-and-Play Priors as a Score-Based Method that diffusion models can be swapped into classical optimization algorithms like ADMM.

In this view, the diffusion model acts as a modular "denoiser" within a larger iterative loop. This modularity is powerful: you can change the "physics" of the problem (the $\mathcal{A}$ operator) without ever touching the weights of the generative model. This has allowed researchers to apply the same RGB diffusion priors to specialized fields like hyperspectral satellite imaging and X-ray ptychography.

Why It Generalizes Better

Supervised models often "overfit" to the specific noise patterns they saw during training. If a supervised model was trained on Gaussian noise, it might fail on real-world sensor noise.

Zero-shot approaches avoid this because they don't model the noise; they model the signal. As Tian Ye argues in AGLLDiff, guiding a model toward fundamental attributes like "good exposure" and "natural color" is more robust than trying to simulate every possible real-world degradation. By relying on the rich, internal "texture repository" of a foundation model Xin Li, these systems can handle "blind" restoration where the exact cause of the image damage is unknown.

Papers in this version

Showing 60 of 292 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_zero_shot_image_restoration_with_pre_tra_69652c98,
  title = {Zero-Shot Image Restoration with Pre-trained Generative Priors},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-image-restoration-with-pre-trained-generative-priors/rdr_69652c982c654021804961be08c3a330},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_zero_shot_image_restoration_with_pre_tra_69652c98_v1,
  title = {Zero-Shot Image Restoration with Pre-trained Generative Priors},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-image-restoration-with-pre-trained-generative-priors/rdr_69652c982c654021804961be08c3a330/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}