In imaging science, we often face the challenge of reconstructing a high-quality image from corrupted or incomplete data. Whether it is removing blur from a photograph, reconstructing an MRI from sparse samples, or denoising a low-dose CT scan, we are solving an inverse problem.
Mathematically, we observe $y = Ax + n$, where $x$ is the clean image, $A$ is a forward operator (like blur or undersampling), and $n$ is noise. Because $A$ often loses information, there are infinitely many possible images $x$ that could explain the observation $y$. To pick the "right" one, we need a prior: a mathematical description of what a natural image looks like.
The Failure of Hand-Crafted Priors
For decades, researchers relied on hand-crafted priors like Total Variation (TV) or Sparsity. These models assume the world is simple—for instance, that images are mostly composed of flat regions with sharp edges.
As noted by Yaniv Romano, Michael Elad, and Peyman Milanfar in The Little Engine that Could: Regularization by Denoising (RED), these rigid assumptions fail to capture the intricate textures and complex statistical dependencies of the real world. The result is often an image that looks "blocky" or "unnatural."
The shift in the field, pioneered by researchers like Stefan Harmeling and Harold Christopher Burger in Image denoising with multi-layer perceptrons, was to stop designing the prior and start learning it from massive datasets using neural networks.
Denoising as a Universal Prior
The most powerful intuition in this research direction is that a good denoiser is implicitly a perfect model of image statistics. If a network can distinguish between "natural image signal" and "random noise," it effectively knows the manifold of all possible valid images.
This led to the Plug-and-Play (PnP) framework. As explained by Hong Ye Tan in From Image Denoisers to Regularizing Imaging Inverse Problems, PnP allows us to take a state-of-the-art denoiser—like the CNNs developed by Kai Zhang in Beyond a Gaussian Denoiser—and "plug" it into a classical optimization algorithm.
Instead of solving a complex equation for the prior, the algorithm simply alternates between:
- Data Fidelity: Moving the image closer to the physical measurements ($y$).
- Prior Projection: Passing the image through a deep denoiser to "clean up" artifacts and push it back toward the manifold of natural images.
Deep Unrolling: Turning Algorithms into Architectures
While PnP methods are modular, they can be slow because they require many iterations. Deep Unrolling solves this by treating the iterations of an optimization algorithm as the layers of a neural network.
In Proximal Alternating Direction Network (PADNet), Risheng Liu and colleagues demonstrate how to "unroll" the ADMM algorithm. Each "layer" of the network performs one step of the optimization, but the parameters of the prior are learned end-to-end. This combines the physical rigor of the observation model with the speed and representational power of deep learning. This approach has been successfully applied to specialized tasks like low-dose CT reconstruction by Qiaoqiao Ding and Hui Ji.
Generative and Diffusion Priors
The most recent evolution involves using generative models—like GANs or Diffusion models—as the prior. Instead of just "cleaning" an image, these models can "imagine" missing details based on the learned distribution of the data.
Rushil Anirudh and Bhavya Kailkhura showed in An Unsupervised Approach to Solving Inverse Problems using GANs that we can solve "blind" problems (where even the corruption $A$ is unknown) by searching the latent space of a pre-trained generative model.
More recently, Youzuo Lin and Lu Lu introduced RED-DiffEq, which uses diffusion models as the prior. Diffusion models provide a smoother, more stable "score field" that guides the reconstruction process, allowing for the recovery of highly complex physical structures, such as subsurface geological features, that were previously unreachable with hand-crafted math.
Why It Works
This research direction works because it respects the physics of the measurement while embracing the complexity of the data. By separating the "how the camera works" (the forward model) from "what the world looks like" (the learned prior), these methods achieve a level of fidelity and robustness that neither pure physics nor pure "black-box" deep learning can reach alone.