Provable Convergence in Learned Regularization

21 concepts16 papers

When reconstructing a clean image from noisy or incomplete measurements—a classic challenge known as an inverse problem—we must balance two competing forces: data fidelity (respecting the physical measurement process) and regularization (prioritizing plausible solutions).

Historically, this balance was struck using hand-crafted mathematical priors like Total Variation flow, analyzed by Frick et al., which offer rock-solid theoretical guarantees. Classical regularization theory ensures that as measurement noise vanishes, the reconstructed solution is mathematically guaranteed to converge to the true ground truth.

Modern deep learning has shattered empirical performance records by replacing these hand-crafted priors with neural networks. However, purely data-driven models often operate as unconstrained black boxes. When a neural network is embedded inside an iterative reconstruction loop, it can amplify minor errors, oscillate endlessly, or hallucinate realistic-looking but physically incorrect artifacts.

To solve this, a powerful research direction has emerged: provably convergent learned regularization. By embedding mathematical constraints directly into the architecture, training objectives, or optimization paths of neural networks, researchers are bridging the gap between empirical deep learning performance and rigorous classical stability.


Geometric Constraints: Convexity and Monotonicity

Why do unconstrained iterative loops fail? In many frameworks, such as Plug-and-Play (PnP) methods, a neural network denoiser is repeatedly applied to clean up intermediate steps of an optimization algorithm. If the network's behavior is arbitrary, the optimization trajectory can become chaotic.

One way to enforce stability is to constrain the geometry of the learned regularizer. If the regularizer is strictly convex, the overall optimization problem retains a unique global minimum, guaranteeing predictable behavior. Tan et al. demonstrate that incorporating convexity into neural network regularizers guarantees a unique solution and stable convergence during image reconstruction.

However, strict convexity can sometimes limit a network's capacity to represent highly complex, sharp image features. To strike a better balance between expressivity and stability, Shumaylov et al. introduce Input Weakly Convex Neural Networks (IWCNNs). By allowing a controlled degree of non-convexity, this framework creates a "convex-nonconvex" regularizer that is expressive enough to capture intricate data patterns while remaining stable enough to guarantee provable convergence without needing heuristic interventions like early stopping.

Alternatively, researchers can enforce stability through operator theory. Instead of constraining an energy landscape, one can constrain the properties of the iterative operator itself. Pesquet et al. show that learning Maximally Monotone Operators (MMOs) through their resolvents provides a theoretically sound framework for PnP regularization, ensuring mathematical convergence where unconstrained denoisers fail.


Algorithmic Constraints: Gradient Steps and Non-Expansiveness

Another strategy is to ensure that the neural network denoiser behaves exactly like a valid mathematical step in a classical optimization algorithm.

If a deep denoiser is parameterized as a "gradient-step" operator—meaning its output corresponds to the gradient of an underlying scalar cost function—the iterative loop is guaranteed to minimize an explicit global cost function. Hurault et al. and Stergiopoulou et al. leverage this formulation to prove that PnP algorithms using gradient-step denoisers reliably converge to stationary points, completely eliminating the oscillatory behavior typical of unconstrained networks.

Failure Mode Mathematical Cause Learned Solution
Oscillation / Divergence Denoiser mapping stretches distances, amplifying small errors across iterations. Jacobian Regularization: Enforces firm non-expansiveness (Benfenati).
Chaos / No Explicit Objective Denoiser outputs do not correspond to any physical energy landscape. Gradient-Step Parameterization: Forces the network to act as a proximal operator (Hurault et al.).

To prevent errors from accumulating over hundreds of iterations, the denoiser must also be contractive—it must not amplify the distance between two close images. Alessandro Benfenati addresses this by training neural network denoisers with Jacobian regularization. This constraint forces the mapping to be firmly non-expansive, providing a rigorous mathematical guarantee of long-term stability and preventing artifact accumulation in iterative restoration frameworks.


Asymptotic Convergence: The Vanishing Noise Limit

A cornerstone of classical regularization theory is asymptotic convergence: as the measurement noise level approaches zero, the reconstructed solution must converge precisely to the true physical solution.

Hauptmann et al. achieve formal convergent regularization in PnP frameworks by focusing on linear denoisers combined with a novel spectral filtering technique. This ensures that the fixed points of the iterative loop represent the correct physical solution rather than an arbitrary stopping point. Similarly, Bauermeister et al. frame regularization as a learning problem over spectral methods, maintaining rigorous convergence guarantees while outperforming classical baselines.

This philosophy translates directly to complex imaging modalities. For instance, Kabri et al. develop data-driven linear regularization methods for CT reconstruction that learn optimal filters from data while remaining provably convergent as noise vanishes.

Even when dealing with non-linear forward operators or data-dependent priors, stability can be preserved:

  • Denker et al. show that invertible residual networks function as convergent regularization schemes if they satisfy a local approximation property that mimics the forward physical process near the training data.
  • Bianchi et al. introduce a data-dependent regularization method based on the Graph Laplacian that offers theoretical guarantees of convergence and stability under weak hypotheses.
  • In the non-linear Banach space setting, Ankik Kumar Giri proves that iteratively regularized Landweber methods achieve stable convergence under noisy data by leveraging Hölder stability estimates.

Quantitative Guarantees: Variational Source Conditions

Proving that a method converges is a vital first step, but practical deployment often requires knowing how fast it converges. To derive explicit error bounds, researchers look to classical tools like the variational source condition (VSC), which measures the smoothness of the ground-truth data relative to the regularizer.

Mukherjee et al. bridge this gap by integrating a specific VSC penalty term directly into an adversarial training objective. By forcing the learned regularizer to satisfy the variational source condition on the ground-truth data distribution, the authors derive formal, quantitative convergence rate estimates. This ensures that the reconstruction error scales predictably and boundedly with decreasing noise levels.

Adversarial Training Loop
  ├── Standard Reconstruction Loss (Empirical Performance)
  └── Variational Source Condition Penalty ──> Guarantees Quantitative Convergence Rates

Finally, to make these stable regularizers practical, practitioners must be able to adjust the regularization strength dynamically depending on the noise present in a specific image. Khelifa et al. introduce Tweedie Scaling, a method derived from statistical identities that transforms a single pre-trained deep denoiser into a family of regularizers with a mathematically grounded scaling "knob." This scaling ensures that as the regularization strength is modulated, the iterative algorithm is still guaranteed to reach a stable fixed point.

Key Foundational Work

Learning Spectral Regularizations for Linear Inverse Problems (2020) — contributed 2 concept elements to this direction.
Inverse Total Variation Flow (2007) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_provable_convergence_in_learned_regulari_00621a41,
  title = {provable convergence in learned regularization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/provable-convergence-in-learned-regularization/rdr_00621a4168cb43b183b7d93bbfa6ff8b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_provable_convergence_in_learned_regulari_00621a41_v1,
  title = {provable convergence in learned regularization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/provable-convergence-in-learned-regularization/rdr_00621a4168cb43b183b7d93bbfa6ff8b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake