Neural‐network‐based regularization methods for inverse problems in imaging

Andreas Habring, Martin Höller

TL;DR

This paper provides a comprehensive roadmap for using neural networks to solve inverse problems in imaging, such as MRI and CT scans. It categorizes the field into supervised, unsupervised, and untrained models, bridging the gap between classical variational mathematics and modern deep learning.

What Problem Does This Paper Address?

In imaging, an inverse problem involves reconstructing a high-quality image ($u$) from noisy, incomplete measurements ($y$). Mathematically, this is expressed as $y = Au + \zeta$, where $A$ is a forward operator (like a Radon transform in CT) and $\zeta$ is noise.

These problems are typically "ill-posed," meaning there isn't enough information in the measurements to find a unique, stable solution. Historically, mathematicians used "hand-crafted" regularization—like Total Variation (TV)—to penalize unrealistic solutions. However, these manual priors struggle with the complexity of real-world textures. This paper explores how neural networks (NNs) can learn much more powerful "priors" directly from data.

Methods

The authors categorize NN-based regularization into three main pillars:

  1. Unsupervised Models: These learn the structure of images without needing pairs of "noisy" and "clean" data.
  2. Learned Generators: Using GANs or VAEs to constrain the solution to a low-dimensional "manifold of interest."
    • Diffusion & Score-based Models: A hot topic where the network learns to "reverse" noise, effectively learning the gradient of the image distribution.
  3. Plug-and-Play (PnP) & RED: Using a pre-trained denoiser as a modular component inside a classical optimization loop.
  4. Supervised Models: These require pairs of ground-truth images and observations.
  5. Unrolling: Designing the network architecture to mimic the steps of a classical iterative algorithm (e.g., Gradient Descent). This makes the "black box" of deep learning more interpretable.
  6. Post-processing: Using a simple classical method first, then training a CNN to remove the resulting artifacts.
  7. Untrained Models: Perhaps the most surprising category. Methods like Deep Image Prior (DIP) show that the architecture of a convolutional network itself acts as a regularizer, allowing for reconstruction without any prior training on external datasets.

Datasets

As a review paper, it does not introduce a new benchmark but references several standard ones used in the field:

  • fastMRI: A large-scale dataset of raw k-space knee and brain data for accelerated MRI reconstruction.
  • CT Datasets: Various clinical and simulated datasets for low-dose CT reconstruction.
  • Standard Image Benchmarks: Used primarily for evaluating denoising and inpainting components.

Key Insights and Ideas

  • The Bayesian-Variational Bridge: The authors highlight that minimizing a regularization functional is mathematically equivalent to finding the Maximum A Posteriori (MAP) estimate in a Bayesian framework. This allows researchers to swap between "energy minimization" and "probability sampling" perspectives.
  • Architecture as a Prior: The success of untrained models (DIP) suggests that the "inductive bias" of convolutions—their tendency to favor spatially correlated structures over random noise—is a fundamental mathematical tool for regularization.
  • Function Space Analysis: Unlike many "black box" AI papers, this review emphasizes Function Space Theory. It discusses whether these algorithms still work as we move from discrete pixels to infinite-dimensional continuous functions, which is vital for medical safety and reliability.

Key Results

The paper summarizes the evolution of the field rather than reporting a single accuracy score. However, it highlights:

  • Superiority over Hand-crafted Priors: NN-based methods (specifically unrolled and diffusion models) consistently outperform classical TV regularization in MRI and CT, especially at high acceleration factors (see Figure 1 in the paper).
  • Convergence Guarantees: The review notes that while supervised methods are faster, unsupervised and PnP methods are often more "stable" and easier to analyze theoretically.
  • Uncertainty Quantification: The paper points to the rise of "Conformal Prediction" as a way to provide rigorous error bars for AI-generated medical images, addressing the "hallucination" problem in deep learning.

Limitations and Future Work

  • Generalization: Supervised models are often brittle; a model trained for a specific MRI machine might fail on another. Moving toward "operator-agnostic" models is a major hurdle.
  • Hallucinations: Deep networks can sometimes create realistic-looking features that aren't actually there (e.g., a fake tumor or a missing vessel).
  • Theoretical Gaps: While function space analysis is progressing, many of the most successful "unrolled" architectures still lack rigorous proofs of convergence or stability.
  • Computational Cost: Diffusion models provide high-quality samples but are currently too slow for real-time clinical use compared to end-to-end supervised networks.

Summary updated

Abstract

Abstract This review provides an introduction to—and overview of—the current state of the art in neural‐network based regularization methods for inverse problems in imaging. It aims to introduce readers with a solid knowledge in applied mathematics and a basic understanding of neural networks to different concepts of applying neural networks for regularizing inverse problems in imaging. Distinguishing features of this review are, among others, an easily accessible introduction to learned generators and learned priors, in particular diffusion models, for inverse problems, and a section focusing explicitly on existing results in function space analysis of neural‐network‐based approaches in this context.
BibTeX
@inproceedings{habring2024neuralnetworkbased,
  author = {Andreas Habring and Martin Höller},
  title = {{Neural‐network‐based regularization methods for inverse problems in imaging}},
  booktitle = {GAMM-Mitteilungen},
  year = {2024},
  doi = {10.1002/gamm.202470004},
}

Loading related content...

Figure