Gradient Step Denoiser for convergent Plug-and-Play

Samuel Hurault, Arthur Leclaire, Nicolas Papadakis

Abstract

Plug-and-Play (PnP) methods are powerful tools for image restoration that use deep learning denoisers as regularizers, but they often lack mathematical convergence guarantees. This paper introduces the Gradient Step (GS) Denoiser, a novel architecture that forces a deep denoiser to act as a gradient descent step on an explicit scalar functional. This simple yet clever constraint allows the authors to prove that their PnP algorithm converges to stationary points of a global objective while maintaining state-of-the-art performance in deblurring, super-resolution, and inpainting.

The Challenge of Unstable Image Restoration

Image restoration is typically framed as an inverse problem: finding a clean image $x$ from a degraded observation $y$. Mathematically, we minimize an objective function $f(x) + \lambda g(x)$, where $f$ ensures data fidelity (how well $x$ matches $y$) and $g$ is a "prior" that enforces image quality.

Plug-and-Play (PnP) methods revolutionized this field by replacing the complex math of the prior $g$ with a pre-trained, off-the-shelf deep denoiser. While PnP yields stunning visual results, it suffers from a "black box" problem: deep neural networks aren't naturally the "gradients" or "proximal operators" of any real function $g$. Consequently, most PnP algorithms lack convergence guarantees. They might produce a great image in 10 iterations but diverge into noise if run for 100. Previous attempts to fix this required unrealistic constraints, like forcing the network to be a "contraction" (which hurts image quality) or assuming "Jacobian symmetry" that deep networks simply don't possess.

The Gradient Step (GS) Denoiser Framework

The authors propose a new PnP framework called GS-PnP. The core innovation is the design of the Gradient Step (GS) Denoiser.

  1. The Architecture: Instead of a standard black-box network $D(x)$, they define the denoiser as: $$D_\sigma = \text{Id} - \nabla g_\sigma$$ where $g_\sigma$ is a scalar energy function. To make this practical, they parameterize $g_\sigma$ using a deep network $N_\sigma$ (specifically a modified DRUNet): $$g_\sigma(x) = \frac{1}{2} \|x - N_\sigma(x)\|^2$$ By calculating the gradient of this specific form, the denoiser $D_\sigma$ is mathematically guaranteed to be a "conservative vector field"—essentially, it behaves like a proper gradient step.

  2. The Algorithm: They plug this GS-denoiser into a Proximal Gradient Descent (PGD) scheme. Because the denoiser is now a gradient step, the entire PnP algorithm becomes a standard optimization of an explicit global functional $F(x) = f(x) + \lambda g_\sigma(x)$.

  3. Backtracking: To ensure stability without knowing the exact "smoothness" (Lipschitz constant) of their deep prior, they implement a backtracking line search that automatically adjusts the step size at each iteration.

Key Insights

  • Constraint without Compromise: The most surprising insight is that forcing a denoiser to be a gradient step doesn't hurt its performance. The GS-DRUNet actually matches or slightly outperforms its unconstrained counterparts.
  • Jacobian Symmetry by Design: Previous methods like RED (Regularization by Denoising) failed to converge because they assumed the denoiser's Jacobian was symmetric. This paper constructs a denoiser that is symmetric by definition, solving the theoretical bottleneck.
  • Tweedie’s Identity Connection: The authors link their approach to Tweedie’s Identity from statistics, which relates the Minimum Mean Square Error (MMSE) denoiser to the gradient of the log-prior of the data. This provides a solid statistical foundation for why their "gradient step" approach works so well.

Experimental Results & Convergence

The GS-denoiser was trained on a massive composite dataset (CBSD, Waterloo, DIV2K, Flick2K) and evaluated on standard benchmarks for deblurring, super-resolution, and inpainting.

Quantitative deblurring results comparing GS-PnP against state-of-the-art methods across different noise levels.
  • State-of-the-Art Performance: In deblurring and super-resolution, GS-PnP consistently matches or beats DPIR, the previous gold standard for PnP. For example, in deblurring on CBSD10, GS-PnP often achieved the highest PSNR across various noise levels.
  • Empirical Convergence: Unlike other PnP methods that can fluctuate or diverge, the authors show that GS-PnP's objective function decreases monotonically. In their tests, the residual $\|x_{k+1} - x_k\|$ dropped to near-zero ($10^{-13}$), proving the algorithm reaches a stable stationary point.
Visual comparison on the starfish image and empirical convergence plots demonstrating monotonic decrease of the objective function.
  • Visual Fidelity: Qualitatively, the method recovers sharper edges and finer textures (like the scales on a starfish or the veins in a leaf) compared to older methods like RED or IRCNN.

Limitations and Future Work

While GS-PnP represents a major theoretical leap, it comes with trade-offs. Because the denoiser involves calculating a gradient (essentially a backpropagation pass through the network), it is roughly 2-7 times slower than a standard forward-pass denoiser. The authors mitigated this by using a "light" version of DRUNet, but speed remains a consideration.

Additionally, while we know $g_\sigma$ acts as a regularizer, it remains a "black box" in terms of what specific image features it prioritizes. Future work could explore what information is actually encoded in this learned energy functional. Finally, while the algorithm converges to a stationary point, the objective function is non-convex, meaning it could theoretically get stuck in a local minimum. However, in practice, the results remain robust and high-quality regardless of initialization.

Summary updated

Abstract

Plug-and-Play methods constitute a class of iterative algorithms for imaging problems where regularization is performed by an off-the-shelf denoiser. Although Plug-and-Play methods can lead to tremendous visual performance for various image problems, the few existing convergence guarantees are based on unrealistic (or suboptimal) hypotheses on the denoiser, or limited to strongly convex data terms. In this work, we propose a new type of Plug-and-Play methods, based on half-quadratic splitting, for which the denoiser is realized as a gradient descent step on a functional parameterized by a deep neural network. Exploiting convergence results for proximal gradient descent algorithms in the non-convex setting, we show that the proposed Plug-and-Play algorithm is a convergent iterative scheme that targets stationary points of an explicit global functional. Besides, experiments show that it is possible to learn such a deep denoiser while not compromising the performance in comparison to other state-of-the-art deep denoisers used in Plug-and-Play schemes. We apply our proximal gradient algorithm to various ill-posed inverse problems, e.g. deblurring, super-resolution and inpainting. For all these applications, numerical results empirically confirm the convergence results. Experiments also show that this new algorithm reaches state-of-the-art performance, both quantitatively and qualitatively.
BibTeX
@inproceedings{hurault2022gradient,
  author = {Samuel Hurault and Arthur Leclaire and Nicolas Papadakis},
  title = {{Gradient Step Denoiser for convergent Plug-and-Play}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=fPhKeld3Okz},
}
Figure