In computational imaging, we frequently encounter inverse problems: attempting to reconstruct a clean, high-resolution image from corrupted, blurry, or incomplete measurements. Mathematically, this is often expressed as recovering an image $x$ from observations $y = Ax + n$, where $A$ represents a physical forward operator (such as a blur kernel or a downsampling matrix) and $n$ represents noise.
Because $A$ is typically ill-conditioned or non-invertible, infinitely many images could fit the observed data. To select the most realistic reconstruction, classical frameworks introduce a regularization term $R(x)$ to solve an optimization problem:
$$\min_x \|Ax - y\|^2 + \lambda R(x)$$
For decades, $R(x)$ was hand-crafted using mathematical heuristics like Total Variation or wavelet sparsity. While these models are mathematically tractable, they rely on simplistic assumptions that often over-smooth complex textures and fail to capture the rich statistical structures of real-world scenes. As noted in the historical overview by Caroline Chaux et al. in Evolution de la résolution de problèmes inverses en imagerie, the field has undergone a paradigm shift away from these rigid formulas toward data-driven approaches that learn the very "essence" of natural images.
The Intuition: Denoisers as Manifold Projectors
The breakthrough idea behind modern learned image priors is remarkably elegant: a deep neural network trained exclusively to remove Gaussian noise implicitly learns the entire geometry of natural images.
Imagine the space of all possible pixel combinations as a high-dimensional landscape. Real-world images do not scattered randomly across this space; instead, they lie on a highly complex, low-dimensional surface or "manifold." When an image is corrupted by noise, it is pushed off this manifold into the surrounding empty space.
A deep convolutional neural network (CNN) trained on millions of examples to map noisy images back to clean images effectively learns a vector field. For any arbitrary point in pixel space, the denoiser outputs a trajectory pointing directly back toward the high-probability manifold of natural images.
By leveraging this learned trajectory, we can solve complex inverse problems without ever writing down an explicit mathematical formula for what an image looks like. As surveyed by Duoduo Xue et al. in A contemporary survey on image reconstruction with unsupervised deep learning: from denoising to generation, this insight allows pre-trained denoisers to act as universal priors, decoupled from any specific imaging hardware or forward operator $A$.
Decoupling Physics from Statistics via Variable Splitting
To use a black-box neural network inside a rigorous optimization framework, researchers employ variable splitting techniques such as Half-Quadratic Splitting (HQS) or the Alternating Direction Method of Multipliers (ADMM).
By introducing an auxiliary variable $z$, we can rewrite the optimization problem to separate the physical measurement model from the image prior:
$$\min_{x, z} \|Ax - y\|^2 + \lambda R(z) \quad \text{subject to } x = z$$
Under frameworks like Plug-and-Play (PnP) priors or Regularization by Denoising (RED), this formulation splits the reconstruction process into an alternating, two-step iterative loop:
- The Data Fidelity Step: Update $x$ to ensure the reconstruction matches the physical measurements $y$ governed by the operator $A$.
- The Prior Step: Update $z$ by passing the current estimate through a pre-trained deep denoiser $D(\cdot)$.
┌────────────────────────────────────────────────────────┐
│ │
▼ │
┌────────────────────────┐ ┌────────────────────────┐ │
│ Data Fidelity Step │ │ Prior Step │ │
│ Matches Physical Ops │─────>│ Pushes onto Manifold │─┘
│ (Uses Matrix A) │ │ (Deep Denoiser D) │
└────────────────────────┘ └────────────────────────┘
In their foundational work, Kai Zhang et al. et al. demonstrated in Learning Deep CNN Denoiser Prior for Image Restoration that this modular architecture allows the exact same deep denoiser to solve entirely different tasks—such as deblurring and super-resolution—simply by swapping out the forward operator in the data fidelity step.
This strategy has proven highly effective across non-linear and ill-posed problems. For example, Yaotian Wang et al. et al. introduced RED-ITA in When deep denoising meets iterative phase retrieval, embedding a deep denoiser within an ADMM loop to solve phase retrieval problems with unprecedented noise robustness. Similarly, Christopher Metzler et al. et al. developed prDeep in prDeep: Robust Phase Retrieval with Flexible Deep Neural Networks, demonstrating that treating a denoiser's output as an explicit regularizer enables standard splitting algorithms to handle highly structured, magnitude-only Fourier measurements.
Defining the Energy Landscape: Implicit vs. Explicit Regularizers
While PnP and RED frameworks often treat the denoiser as an implicit gradient step ($\nabla R(x) \approx x - D(x)$), understanding the exact mathematical connection between denoisers and explicit energy functions remains a focus of active research. Einav Yogev-Ofer et al. et al. explore these theoretical boundaries in An Interpretation Of Regularization By Denoising And Its Application With The Back-Projected Fidelity Term, analyzing how denoising gradients map to subgradients of underlying prior functions.
To ensure mathematical convergence, some approaches explicitly construct neural networks to output a scalar regularization energy rather than a cleaned image. Andreas Habring et al. et al. utilize these energy-based models in Energy-based models for inverse imaging problems, showing they capture sharp edges and complex textures far better than classical total variation. Taking this a step further, Erich Kobler proposed Total Deep Variation (TDV) in Total Deep Variation for Linear Inverse Problems, which parameterizes a learnable regularization energy using a multi-scale CNN. This ensures that the entire reconstruction process corresponds to minimizing a well-defined energy functional, combining variational rigor with deep learning's expressivity.
Other frameworks train networks to act as critics or classifiers within the optimization loop:
- Adversarial Critics: Sebastian Lunz et al. et al. demonstrate in Adversarial Regularizers in Inverse Problems that a network can be trained to distinguish between ground-truth images and artifacts, acting as an adversarial regularizer that penalizes un-natural reconstructions.
- Discriminative Classifiers: Jinshan Pan et al. et al. show in Learning a Discriminative Prior for Blind Image Deblurring that a CNN trained simply to classify sharp versus blurry images can serve as a powerful data-driven regularizer for blind deblurring.
Deep Unrolling: Accelerating the Loop
While iterative PnP and RED algorithms offer high modularity, running hundreds of forward and backward optimization steps can be computationally expensive. To bridge the gap between mathematical rigor and real-time execution, researchers use Deep Unrolling (or Unfolding).
In an unrolled network, a fixed number of optimization iterations (e.g., 10 or 20 steps) are mapped directly onto the layers of a deep neural network architecture. The physical forward model is embedded into specific data-fidelity layers, while the regularizer is represented by trainable convolutional blocks. The entire pipeline is then optimized end-to-end.
Shuang Xu implemented this strategy for satellite imaging in Deep Gradient Projection Networks for Pan-sharpening, replacing hand-crafted mathematical constraints with learnable proximal operators. Similarly, Wenhui Wu et al. et al. developed URetinex-Net in URetinex-Net: Retinex-based Deep Unfolding Network for Low-light Image Enhancement, unfolding traditional optimization steps into an interpretable, highly efficient network for low-light recovery.
This architecture provides significant speedups. In Learned reconstructions for practical mask-based lensless imaging, Grace Kuo demonstrated that unrolling an optimization loop for lensless cameras yields a 20x speedup, enabling real-time high-quality image previews while strictly respecting the physics of light transport.
Broad Applications Across Scientific Domains
Because learned priors separate the statistical model of the data from the physical measurement system, they have found widespread application across diverse scientific and engineering disciplines:
Medical and Biological Imaging
In medical imaging, obtaining ground-truth data is often difficult or impossible. Burhaneddin Yaman addressed this in Improved Simultaneous Multi-Slice Functional MRI Using Self-supervised Deep Learning by training image-domain regularizers directly on undersampled data, achieving a 50% boost in temporal signal-to-noise ratio for functional MRIs. For microscopy, Ghassan Hamarneh et al. et al. developed a hybrid approach in Patch-based Non-Local Bayesian Networks for Blind Confocal Microscopy Denoising, using a neural network to regress the mean and covariance parameters of a Gaussian prior for blind denoising. To standardize evaluations across these diverse medical frameworks, Maximilian B. Kiss introduced the open-source LION toolbox in Benchmarking learned algorithms for computed tomography image reconstruction tasks, enabling direct comparisons of unrolled and PnP methods on real-world computed tomography (CT) data.
Neural Decoding and 3D Scene Representation
The flexibility of deep priors extends beyond traditional camera sensors. In Maximum a posteriori natural scene reconstruction from retinal ganglion cells with deep denoiser priors, Eric Gene Wu combined a biological encoding model with a deep denoiser prior to reconstruct natural scenes directly from the electrical spikes of retinal ganglion cells, matching the performance of black-box models with a fraction of the parameters.
In 3D computer vision, Haeyun Choi et al. et al. integrated feed-forward deblurring networks into radiance field optimization in Exploiting Deblurring Networks for Radiance Fields, recovering sharp 3D scenes from blurry training inputs. For video processing, Jialong Guo et al. et al. introduced MetaNeRV in MetaNeRV: Meta Neural Representations for Videos with Spatial-Temporal Guidance, demonstrating that a meta-learned prior functions as an effective regularizer that accelerates convergence and filters noise in implicit neural video representations.
Key Foundational Work
Learning Generic Prior Models for Visual Computation (1997) — contributed 1 concept elements to this direction.
Structured Learning of Sum-of-Submodular Higher Order Energy Functions (2013) — contributed 1 concept elements to this direction.