Deep Equilibrium Learning of Explicit Regularizers for Imaging Inverse Problems

Zihao Zou, Jiaming Liu, Brendt Wohlberg, Ulugbek S. Kamilov

Abstract

There has been significant recent interest in the use of deep learning for regularizing imaging inverse problems. Most work in the area has focused on regularization imposed implicitly by convolutional neural networks (CNNs) pre-trained for image reconstruction. In this work, we follow an alternative line of work based on learning explicit regularization functionals that promote preferred solutions. We develop the Explicit Learned Deep Equilibrium Regularizer (ELDER) method for learning explicit regularizers that minimize a mean-squared error (MSE) metric. ELDER is based on a regularization functional parameterized by a CNN and a deep equilibrium learning (DEQ) method for training the functional to be MSE-optimal at the fixed points of the reconstruction algorithm. The explicit regularizer enables ELDER to directly inherit fundamental convergence results from optimization theory. On the other hand, DEQ training enables ELDER to improve over existing explicit regularizers without prohibitive memory complexity during training. We use ELDER to train several approaches to parameterizing explicit regularizers and test their performance on three distinct imaging inverse problems. Our results show that ELDER can greatly improve the quality of explicit regularizers compared to existing methods, and show that learning explicit regularizers does not compromise performance relative to methods based on implicit regularization.

Introduction

Computational imaging frequently involves solving inverse problems—recovering an underlying clean image from degraded observations that may be noisy, blurred, or undersampled. Traditionally, these problems were addressed using hand-crafted explicit regularizers, such as Total Variation, which penalize undesirable image features. While theoretically sound and providing convergence guarantees, these classical methods often fall short of the performance achieved by modern deep learning approaches.

Recent deep learning methods predominantly rely on implicit regularization, where a CNN acts as a denoiser within an iterative reconstruction loop (e.g., Plug-and-Play or Unrolled networks). Although powerful, these "black-box" models often lack the theoretical stability and convergence proofs of explicit methods. Furthermore, training unrolled iterative models is memory-intensive, limiting the depth and complexity of the learned priors.

Methodology

The Explicit Learned Deep Equilibrium Regularizer (ELDER) framework bridges the gap between the theoretical rigor of explicit regularization and the high performance of deep learning.

Deep Equilibrium (DEQ) Training

ELDER leverages Deep Equilibrium (DEQ) models to overcome the memory constraints of traditional unrolling. Instead of backpropagating through a fixed number of iterations, ELDER treats the reconstruction as finding a fixed point (equilibrium) of an optimization algorithm. By using implicit differentiation, ELDER can compute gradients at the equilibrium point with constant memory complexity, regardless of the number of iterations required during inference.

Explicit Regularization Architectures

ELDER parameterizes the regularization function $h(x)$ using a CNN. The study evaluates three distinct parameterization strategies:

  • Least-Squares Residual (LSR): Penalizes the $L_2$ distance between the image and the output of a CNN-based denoiser.
  • Regularization by Denoising (RED): Utilizes a functional based on the inner product of the image and the denoiser's residual.
  • Direct Scalar-Valued (DSV): Employs a network that directly maps an image to a scalar regularization score.

By maintaining an explicit objective function, ELDER can utilize standard optimization techniques, such as backtracking line-search, to ensure stable convergence to the optimal solution.

Experimental Results

The ELDER framework was evaluated across several imaging tasks, including super-resolution, inpainting, and compressed sensing MRI (CS-MRI).

Datasets and Tasks

  • General Imaging: Evaluations were conducted on CBSD68, BSD68, and Kodak24 for denoising and super-resolution.
  • Medical Imaging: A brain MRI dataset (800 training, 50 testing slices) was used for CS-MRI at various sampling ratios.
  • Training: Models were trained on a large-scale composite dataset including DIV2K and Flickr2K.

Performance Comparison

ELDER demonstrates significant improvements over existing explicit regularization methods like GSPnP. Notably, it matches the performance of state-of-the-art implicit methods such as DPIR and standard DEQ models, proving that explicit regularization does not necessitate a performance trade-off.

Ground truth image for the statue reconstruction task.

In CS-MRI tasks at a 10% sampling ratio, ELDER achieved a PSNR of 35.14 dB, nearly identical to the 35.16 dB achieved by implicit DEQ models, while providing superior convergence stability. In inpainting tasks with 70% missing pixels, ELDER effectively recovered fine textures and sharp edges where traditional methods failed.

Discussion and Conclusion

The success of ELDER highlights that explicit regularization can be as effective as implicit methods when trained using modern DEQ techniques. Among the tested architectures, the LSR parameterization emerged as the most effective prior, suggesting that proximity to a denoised manifold is a powerful inductive bias.

Because ELDER minimizes an explicit energy function, it inherits stability naturally from optimization theory without requiring the restrictive Lipschitz constraints often needed for DEQ stability. Future work will explore more complex backbone architectures and the extension of ELDER to non-linear inverse problems such as phase retrieval.

Summary updated

Abstract

There has been significant recent interest in the use of deep learning for regularizing imaging inverse problems. Most work in the area has focused on regularization imposed implicitly by convolutional neural networks (CNNs) pre-trained for image reconstruction. In this work, we follow an alternative line of work based on learning explicit regularization functionals that promote preferred solutions. We develop the Explicit Learned Deep Equilibrium Regularizer (ELDER) method for learning explicit regularizers that minimize a mean-squared error (MSE) metric. ELDER is based on a regularization functional parameterized by a CNN and a deep equilibrium learning (DEQ) method for training the functional to be MSE-optimal at the fixed points of the reconstruction algorithm. The explicit regularizer enables ELDER to directly inherit fundamental convergence results from optimization theory. On the other hand, DEQ training enables ELDER to improve over existing explicit regularizers without prohibitive memory complexity during training. We use ELDER to train several approaches to parameterizing explicit regularizers and test their performance on three distinct imaging inverse problems. Our results show that ELDER can greatly improve the quality of explicit regularizers compared to existing methods, and show that learning explicit regularizers does not compromise performance relative to methods based on implicit regularization.
BibTeX
@article{zou2023deep,
  author = {Zihao Zou and Jiaming Liu and Brendt Wohlberg and Ulugbek S. Kamilov},
  title = {{Deep Equilibrium Learning of Explicit Regularizers for Imaging Inverse Problems}},
  journal = {IEEE Open Journal of Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=XsANeD3a3e},
}
Figure