Beyond Hand-Crafted Regularization in Inverse Problems

194 concepts179 papers1 proposal

In computer vision and medical imaging, we frequently encounter inverse problems: the task of recovering a hidden, high-quality image $x$ from degraded, noisy, or incomplete measurements $y$. This relationship is typically modeled as:

$$y = Ax + n$$

where $A$ is a forward operator (such as a blur matrix, a downsampling mask, or a Radon transform) and $n$ is noise.

Because information is lost during the degradation process, these problems are fundamentally ill-posed. Infinitely many clean images could explain the observed data $y$. To find the correct solution, classical imaging science relies on a regularizer or prior $R(x)$ to penalize unrealistic solutions, solving an optimization problem of the form:

$$\arg\min_x \|Ax - y\|^2 + \lambda R(x)$$

For decades, $R(x)$ was designed by human intuition. However, a major paradigm shift has replaced these hand-crafted mathematical formulas with data-driven, learned priors. Understanding this shift requires looking closely at why traditional priors fail and how modern architectures elegantly step into their place.

The Limitations of Hand-Crafted Priors

Traditional variational regularization relies on simple geometric or statistical assumptions about what an image "should" look like. Common examples include:

  • Tikhonov Regularization: Penalizes high frequencies, assuming images are globally smooth.
  • Total Variation (TV): Penalizes the $L_1$-norm of the image gradient, assuming images are piecewise constant.
  • Wavelet Sparsity: Assumes images can be compactly represented by a small number of localized wavelets.

While these mathematical constraints are clean and well-understood, they are far too simplistic to capture the rich, high-dimensional statistics of real-world scenes. As Sebastian Lunz et al. observe in Adversarial Regularizers in Inverse Problems, these rigid functions frequently introduce severe visual artifacts.

For instance, because Total Variation models images as collections of flat patches, it suffers from a notorious "staircasing" effect. Erich Kobler demonstrates in Total Deep Variation for Linear Inverse Problems that such piecewise-constant assumptions are fundamentally inadequate for natural visual data, leading to reconstructions that appear blocky, unnaturally smooth, or stripped of fine textures. When applied to complex biological tissues in medical imaging, these simplified assumptions can obscure critical diagnostic details, as noted by Kuo-Wei Lai et al. in Learned Proximal Networks for Quantitative Susceptibility Mapping.

The Intuition of Learned Priors

Instead of forcing natural images into a human-engineered mathematical box, why not allow a neural network to learn the underlying manifold of valid images from data?

A deep neural network can act as an implicit or explicit prior. Rather than explicitly writing down a formula for $R(x)$, we can train a network to recognize the structural nuances, high-frequency textures, and complex non-linear relationships that characterize real-world data.

The core challenge is integration: how do we combine the explicit physical knowledge of our forward operator $A$ (the physics of the camera or MRI scanner) with the implicit statistical power of a neural network?

The Plug-and-Play (PnP) Breakthrough

The most intuitive bridge between classical optimization and deep learning comes from proximal splitting algorithms, such as the Alternating Direction Method of Multipliers (ADMM).

When solving the variational problem, these algorithms decouple the physics of the problem from the image prior by introducing an auxiliary variable $z$, splitting the optimization into two alternating steps:

  1. The Data Fidelity Step: Update $x$ by enforcing consistency with the physical measurements $y$ and the current estimate of $z$. This is a standard linear algebra problem involving the operator $A$.
  2. The Proximal/Prior Step: Update $z$ by solving a subproblem that depends only on the prior $R(z)$: $$\arg\min_z R(z) + \frac{\rho}{2}\|z - x\|^2$$

Crucially, the mathematical form of this second step is identical to denoising the image $x$ using Gaussian noise reduction.

As realized by Chun-Liang Li et al. in One Network to Solve Them All - Solving Linear Inverse Problems Using Deep Projection Models and Michael Moeller et al. in Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems, we do not actually need to know the algebraic formulation of $R(z)$. We can simply "plug in" a pre-trained, highly sophisticated deep denoising network (like a U-Net) directly in place of this proximal operator.

  +-----------------------+
  |  Measurement Data (y) |
  +-----------+-----------+
              |
              v
+-------------+-------------+
|     Data Fidelity Step    | <-------+
|  (Enforces Physics: Ax=y) |         |
+-------------+-------------+         | Iterative
              |                       | Loop
              v                       |
+-------------+-------------+         |
|   Deep Denoiser Network   | --------+
|  (Learned Image Prior)    |
+---------------------------+

By separating the physics from the prior, a single deep network can be reused to solve entirely different inverse problems—such as super-resolution, inpainting, or deblurring—without ever being retrained for those specific tasks.

Priors Imprinted in Network Architecture

An even more surprising property of neural networks as priors was discovered by Andrea Vedaldi et al. in Deep Image Prior. They demonstrated that you do not even need a dataset to learn an image prior.

Instead, the inductive bias of a convolutional neural network's architecture is itself a powerful prior. When a network is tasked with reconstructing a single degraded image by optimizing its weights from scratch, it naturally learns low-frequency, spatially correlated structures (like clean image edges) much faster than it learns high-frequency random noise. The network structure itself acts as a filter that rejects unnatural artifacts.

From Denoisers to Generative Scores

The evolution of learned priors has culminated in the use of unsupervised generative models, such as diffusion models and normalizing flows.

As surveyed by Duoduo Xue et al. in A contemporary survey on image reconstruction with unsupervised deep learning: from denoising to generation, these models are unified by the mathematical concept of the score function—the gradient of the log-probability density of natural images ($\nabla_x \log p(x)$).

Instead of relying on a scalar penalty like Total Variation, modern frameworks use diffusion models to compute this score function at various noise scales. During reconstruction, the algorithm takes a step to satisfy the physical measurements (via $A$), and then takes a step along the score function provided by the generative model. This process iteratively pushes the corrupted image back onto the high-dimensional manifold of real-world data, recovering complex textures that hand-crafted mathematical rules could never explicitly define.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 174 papers
  2. Version 1 · 2026-10-04 · 293 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_hand_crafted_image_priors_84a99215,
  title = {hand-crafted image priors},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hand-crafted-image-priors/rdr_84a99215ab6c488a8d60570153de9780},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_hand_crafted_image_priors_84a99215_v2,
  title = {hand-crafted image priors},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hand-crafted-image-priors/rdr_84a99215ab6c488a8d60570153de9780/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake