Deep Equilibrium Learning of Explicit Regularization Functionals for Imaging Inverse Problems

Zihao Zou, Jiaming Liu, Brendt Wohlberg, Ulugbek S. Kamilov

TL;DR

ELDER (Explicit Learned Deep Equilibrium Regularizer) is a new framework that learns explicit mathematical functions to regularize imaging inverse problems. By combining Deep Equilibrium (DEQ) training with classical optimization theory, it achieves state-of-the-art reconstruction performance while providing the convergence guarantees that "black-box" neural networks often lack.

What Problem Does This Paper Address?

In computational imaging, we often try to solve "inverse problems"—recovering a clean image from data that is blurry, noisy, or incomplete (like a sparse MRI scan). Traditionally, this was done using hand-crafted mathematical "regularizers" that penalize unlikely images.

Recently, Deep Learning (DL) has taken over, but most DL methods use implicit regularization. In these models, a Convolutional Neural Network (CNN) acts as a denoiser within an iterative loop. While powerful, these models are hard to analyze; it’s difficult to prove they will always converge to a stable solution. On the other hand, explicit regularizers—where the CNN defines an actual energy function to be minimized—are theoretically safer but have historically struggled to match the performance of their implicit cousins.

Furthermore, training deep iterative models usually requires "unrolling" the iterations, which consumes massive amounts of GPU memory, limiting how "deep" or complex the model can be.

Methods

The authors introduce ELDER, which solves these issues using two key innovations:

  1. Deep Equilibrium (DEQ) Training: Instead of training a network by unrolling a fixed number of steps, ELDER treats the reconstruction as finding a "fixed point" (an equilibrium) of an algorithm. Using implicit differentiation, it can backpropagate through this equilibrium point with constant memory complexity, regardless of how many iterations are needed during inference.
  2. Explicit Parameterization: ELDER parameterizes the regularization function $h(x)$ using a CNN. The authors tested three specific architectures:
  3. LSR (Least-Squares Residual): Penalizes the distance between the image and the output of a CNN denoiser.
  4. RED (Regularization by Denoising): A functional based on the cross-product of the image and the denoiser residual.
  5. DSV (Direct Scalar-Valued): A network that directly outputs a single scalar "score" for an image.

By having an explicit objective function, the authors can use standard optimization tools like backtracking line-search to automatically select the best step size during reconstruction, ensuring the algorithm always moves toward a better solution.

Datasets

The models were trained and evaluated on several standard computer vision and medical imaging benchmarks:

  • General Imaging: CBSD68, BSD68, and Kodak24 datasets were used for denoising, super-resolution, and inpainting tasks.
  • Medical Imaging: A brain MRI dataset (800 training slices, 50 testing) was used for Compressed Sensing MRI (CS-MRI) evaluations.
  • Training Data: A large composite dataset including CBSD, Waterloo Exploration Database, DIV2K, and Flickr2K.

Key Insights and Ideas

  • Explicit is not a Compromise: The most significant "aha moment" is that you don't have to sacrifice performance to get theoretical stability. ELDER matches or beats implicit methods while remaining firmly rooted in optimization theory.
  • LSR is the Strongest Prior: Among the three explicit forms tested, the Least-Squares Residual (LSR) parameterization consistently performed best, suggesting that "closeness to a denoised version" is a highly effective mathematical prior for deep learning.
  • Stability Without Constraints: Most DEQ models for imaging require strict "Lipschitz constraints" (mathematical limits on how much the network can change its output) to stay stable. Because ELDER minimizes an explicit energy function, it inherits stability naturally from optimization theory without needing these restrictive constraints that often hamper performance.

Key Results

  • Super-Resolution: ELDER significantly outperformed GSPnP (a recent explicit method) and matched the performance of state-of-the-art implicit methods like DPIR and standard DEQ.
  • CS-MRI: At a 10% sampling ratio, ELDER achieved a PSNR of 35.14 dB, essentially matching the performance of the best implicit DEQ models (35.16 dB) while offering better convergence properties.
  • Inpainting: ELDER showed a substantial lead over traditional Plug-and-Play (PnP) methods, recovering fine details and sharper edges in regions with 70% missing pixels.
  • Memory Efficiency: Thanks to the DEQ approach, ELDER can be trained on a single GPU even for complex, high-iteration tasks that would crash standard "unrolled" models.

Limitations and Future Work

While ELDER is a major step forward, the authors note a few areas for growth:

  • Jacobian-Free Approximation: To speed up training, the authors use a "Jacobian-free" approximation in the backward pass. While effective, exploring more exact or efficient ways to compute these gradients could further improve accuracy.
  • Architectural Exploration: The study focused on the form of the regularizer (LSR vs RED vs DSV). Future work could explore how different CNN backbone architectures (like Transformers or newer U-Nets) interact with the ELDER framework.
  • Non-Linear Problems: Currently, the focus is on linear inverse problems; extending this to non-linear forward models (like phase retrieval) is a logical next step.

Summary updated

Abstract

There has been significant recent interest in the use of deep learning for regularizing imaging inverse problems. Most work in the area has focused on regularization imposed implicitly by convolutional neural networks (CNNs) pre-trained for image reconstruction. In this work, we follow an alternative line of work based on learning explicit regularization functionals that promote preferred solutions. We develop the Explicit Learned Deep Equilibrium Regularizer (ELDER) method for learning explicit regularization functionals that minimize a mean-squared error (MSE) metric. ELDER is based on a regularization functional parameterized by a CNN and a deep equilibrium learning (DEQ) method for training the functional to be MSE-optimal at the fixed points of the reconstruction algorithm. The explicit regularizer enables ELDER to directly inherit fundamental convergence results from optimization theory. On the other hand, DEQ training enables ELDER to improve over existing explicit regularizers without prohibitive memory complexity during training. We use ELDER to train several approaches to parameterizing explicit regularizers and test their performance on three distinct imaging inverse problems. Our results show that ELDER can greatly improve the quality of explicit regularizers compared to existing methods, and show that learning explicit regularizers does not compromise performance relative to methods based on implicit regularization.
BibTeX
@article{zou2023deep,
  author = {Zihao Zou and Jiaming Liu and Brendt Wohlberg and Ulugbek S. Kamilov},
  title = {{Deep Equilibrium Learning of Explicit Regularization Functionals for Imaging Inverse Problems}},
  journal = {IEEE Open Journal of Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=XsANeD3a3e},
}
Figure