When solving complex inverse problems—such as reconstructing an MRI scan from sparse data, deblurring an image, or simulating physical systems—computational scientists traditionally rely on iterative optimization algorithms. Algorithms like the Alternating Direction Method of Multipliers (ADMM) or the Iterative Shrinkage-Thresholding Algorithm (ISTA) step progressively closer to a solution by optimizing a mathematically defined objective function. These classical methods are slow but come with an ironclad promise: mathematical proofs guarantee they will converge to a stable, correct solution.
Deep learning introduced a faster alternative. By "unrolling" these iterative algorithms into sequential neural network layers or plugging a deep denoiser directly into the optimization loop (Plug-and-Play priors), researchers created hybrid solvers. These learned solvers achieve remarkable accuracy in a fraction of the time.
However, this speed comes at a steep cost. Replacing well-behaved mathematical operators with black-box neural networks destroys the original convergence guarantees. Without structural constraints, a learned solver can oscillate wildly, diverge entirely, or collapse if run for more iterations at test time than it saw during training.
To bridge this gap, a major research direction focuses on embedding rigorous mathematical guarantees directly into deep iterative architectures, ensuring they remain provably stable without sacrificing their empirical power.
The Root of Instability: Why Learned Solvers Diverge
To understand why unconstrained learned solvers fail, consider a standard fixed-point iteration:
$$x_{k+1} = f(x_k)$$
In classical optimization, $f$ is carefully engineered to be a proximal operator or a gradient step derived from a convex energy function. Because of this mathematical structure, each step is guaranteed to move closer to the optimal solution.
When a neural network is embedded within $f$, this structure vanishes. As Chun-Liang Li et al. observed, integrating standard deep learning models into iterative frameworks often results in severe numerical instability because these black-box models lack a clear geometric interpretation or sharp boundaries for the target data manifold.
This instability manifests in three distinct ways:
- Inference-Time Collapse: Unrolled networks are typically trained for a fixed, small number of iterations (e.g., 10 layers). As Davis Gilton et al. demonstrated, if you attempt to run these networks for additional steps at test time to refine the results, their performance frequently collapses because they were never regularized to remain stable over an infinite horizon.
- Infinite Loops and Oscillations: In biologically inspired architectures like Predictive Coding Networks, which rely on iterative gradient-based processes to settle internal states, the assumption that iterations will always stabilize often lacks verification. Simon Frieder et al. proved that these networks can mathematically diverge, trapping the system in infinite loops.
- Long-Term Divergence: Even when a learned prior produces high-quality initial updates, Ali Joundi showed that a lack of formal projection properties frequently causes the algorithm to diverge or introduce severe artifacts if allowed to iterate past a certain threshold.
To fix this, researchers use two primary strategies: constraining the network's geometry to force convergence, or wrapping the network in a mathematical safety net.
Strategy 1: Enforcing Contraction and Lipschitz Bounds
The most direct way to guarantee that an iterative process settles into a unique, stable solution is to satisfy the Banach Fixed-Point Theorem. This theorem states that if an operator $f$ is a contraction mapping—meaning it brings any two points closer together after applying it—then iterating $f$ will always converge to a unique fixed point, regardless of the starting state.
For a neural network to act as a contraction mapping, its Lipschitz constant $L$ must be strictly less than 1:
$$\|f(x) - f(y)\| \le L \|x - y\|, \quad \text{where } L < 1$$
Real Spectral Normalization
In Plug-and-Play (PnP) frameworks, where a pre-trained deep denoiser acts as the regularization step in an optimization loop, ensuring this condition is notoriously difficult. Ernest K. Ryu et al. provided a theoretical breakthrough by proving that PnP methods provably converge if the denoiser satisfies a specific Lipschitz bound. To make this practical, the authors introduced real spectral normalization (realSN), a training technique that explicitly constrains the weights of the neural network to enforce this mathematical requirement.
Softening the Constraint: Pseudo-Contractions
While forcing a network to be strictly non-expansive ($L \le 1$) guarantees stability, it can severely limit the network's capacity, forcing a trade-off between mathematical safety and image restoration quality. To alleviate this, Deliang Wei et al. introduced pseudo-contractive denoisers. By utilizing a mathematically weaker constraint combined with the Ishikawa iteration process, they preserved global convergence guarantees while significantly boosting the network's expressive power and restoration performance.
Structural Proximity Operators
Rather than normalizing weights after designing a network, another approach builds stability directly into the network's architecture. Marzieh Hasannasab et al. introduced Parseval Proximal Neural Networks (PPNNs), where every single layer is mathematically guaranteed to be a classical proximity operator. By constraining the weight matrices to Stiefel manifolds, the authors created deep networks that are inherently 1-Lipschitz, making them robust against adversarial attacks and perfectly suited for stable integration into rigorous optimization loops.
Unconstrained Iteration: [x_0] ---> [Neural Network] ---> [x_1] ---> [Neural Network] ---> [Divergence/Oscillation]
(Lipschitz L > 1)
Constrained Iteration: [x_0] ---> [Normalized Net] ---> [x_1] ---> [Normalized Net] ---> [Stable Fixed Point]
(Lipschitz L < 1)
Strategy 2: Embedding Networks in Mathematical Safety Nets
An alternative strategy leaves the neural network unconstrained, allowing it to be completely "free-form" and highly expressive. Instead of modifying the network, researchers embed it inside a classical optimization framework that acts as a supervisor, monitoring the network's outputs and correcting them if they threaten to cause divergence.
Learnable Operator Splitting
Risheng Liu et al. pioneered this approach with Learnable Bregman Splitting (LBS). They embedded a deep neural network into a "triple splitting" operator scheme. The neural network proposes fast, data-driven updates, but the overarching mathematical splitting scheme ensures that the sequence of iterations is rigorously bounded, proving that the algorithm will always converge to a stable solution even in nonconvex scenarios.
This philosophy was expanded by Risheng Liu et al. through the Propagation and Optimization based Deep Model (PODM), which explicitly bridges classical mathematical optimization and heuristic deep learning to provide a deeply trained solver with strict mathematical guarantees.
Task-Driven Feasibility and Coordinate Updates
When a neural network operates as a modular component in a global solver, it can occasionally act as an unreliable component, producing outputs that fall outside expected physical or mathematical bounds. To counteract this, Risheng Liu et al. developed the Task-driven Latent Feasibility (TLF) framework. TLF treats learned image priors as latent constraints that narrow down the search space, utilizing a control mechanism to monitor and correct the network's proposals on the fly.
Similarly, Risheng introduced the Task Embedded Coordinate Update (TECU) framework, which embeds arbitrary black-box operators directly into a coordinate descent loop. If the neural network proposes a step that violates the descent condition, the coordinate update wrapper adjusts the step, preventing algorithmic divergence while maintaining state-of-the-art performance.
Applications Beyond Images: Graphs and Equilibrium Models
The necessity of convergence guarantees extends far beyond image processing. It is vital anywhere neural networks model systems that naturally seek an equilibrium state.
Implicit Graph Neural Networks
Standard Graph Neural Networks (GNNs) capture structural information by stacking a fixed number of message-passing layers. However, this fixed depth prevents them from solving long-range dependency problems. To solve this, Fangda Gu et al. introduced Implicit Graph Neural Networks (IGNNs). Instead of stacking layers, IGNNs define node representations implicitly as the solution to a fixed-point equilibrium equation. To ensure this infinite-depth process converges to a unique solution, the authors leveraged Perron-Frobenius theory to derive strict stability criteria for the network's internal weight matrices.
Convergent Graph Solvers
When predicting the stationary states of physical networks (like power grids or fluid dynamics), standard GNNs lack the guarantees required for physical simulations. Junyoung Park et al. developed Convergent Graph Solvers (CGS), a framework that constructs input-dependent contracting maps. By ensuring the message-passing architecture behaves as a strict contraction, CGS guarantees convergence to the true stationary state while utilizing the implicit function theorem for memory-efficient training.
Summary of the Paradigm Shift
The evolution of learned iterative solvers highlights a clear trajectory toward provable stability:
| Phase | Architecture | Performance | Convergence Guarantee |
|---|---|---|---|
| Classical Optimization | Purely mathematical (ISTA, ADMM) | Slow; requires manual tuning | Guaranteed via convex analysis and descent conditions |
| Early Deep Unrolling | Black-box layers mimicking iterations | Fast; highly adaptive | None; prone to inference collapse and divergence |
| Modern Convergent Solvers | Constrained networks (realSN, PPNNs) or Optimization Wrappers (LBS, TECU) | Fast; state-of-the-art accuracy | Guaranteed via contraction mappings and safety-net frameworks |
By enforcing Lipschitz bounds or wrapping free-form networks in classical optimization loops, researchers have shown that we do not have to choose between the speed of deep learning and the safety of traditional mathematics. The resulting convergent deep solvers provide a reliable, provably stable foundation for mission-critical applications.
Key Foundational Work
Toward Designing Convergent Deep Operator Splitting Methods for Task-specific Nonconvex Optimization (2018) — contributed 1 concept elements to this direction.
An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks (2018) — contributed 1 concept elements to this direction.
On the Convergence of Learning-based Iterative Methods for Nonconvex Inverse Problems (2018) — contributed 1 concept elements to this direction.