Plug-and-Play image restoration with Stochastic deNOising REgularization

Marien Renaud, Jean Prost, Arthur Leclaire, Nicolas Papadakis

TL;DR

The paper introduces SNORE (Stochastic deNOising REgularization), a novel Plug-and-Play (PnP) framework that bridges the gap between classical PnP optimization and modern Denoising Diffusion Models (DDMs). By explicitly injecting noise into the denoiser at each iteration, SNORE ensures the deep denoiser always operates within its training domain, yielding state-of-the-art image restoration quality backed by rigorous convergence guarantees.

What Problem Does This Paper Address?

Image restoration often involves solving "inverse problems"—reconstructing a clean, high-quality image from a degraded (blurry, noisy, or downsampled) observation. Traditional Plug-and-Play (PnP) frameworks handle this by decoupling the physical degradation model from the image prior, replacing the regularization step with a forward pass through a pre-trained deep neural network denoiser.

However, PnP algorithms suffer from a fundamental flaw: as the algorithm iterates, the image becomes progressively cleaner. Deep denoisers, having been trained explicitly on noisy data, experience a domain shift when fed these cleaner intermediate images, leading to suboptimal performance or artifacts.

Conversely, Denoising Diffusion Models (DDMs) excel at generating highly realistic textures because they always apply the denoiser to re-noised images. Unfortunately, guiding a diffusion model to solve a specific inverse problem typically relies on heuristics, leaving them without solid theoretical convergence guarantees. This paper introduces a framework that captures the best of both worlds: the perceptual quality of diffusion models and the mathematical rigor of PnP optimization.

Methods

The authors propose SNORE, which introduces an explicit stochastic regularization term. Instead of evaluating the image prior directly on the clean estimate, SNORE minimizes the average value of the smoothed log-prior over a noisy version of the image.

The core algorithmic contributions include:

  1. Stochastic Gradient Descent Optimization: SNORE optimizes its objective by adding Gaussian noise to the current image iterate before passing it to the denoiser. This forces the denoiser to stay within its training distribution.
  2. Annealed SNORE: To improve practical performance, the authors introduce an annealing schedule (Algorithm 3) that gradually decreases the injected noise level ($\sigma$) and adjusts the regularization weight ($\alpha$) over multiple stages, mimicking the coarse-to-fine refinement seen in diffusion models.
  3. Theoretical Convergence Proofs: Unlike many diffusion-based heuristics, the authors provide a rigorous convergence analysis. They prove that under mild assumptions, SNORE converges to a critical point of the objective function when using an exact Minimum Mean Square Error (MMSE) denoiser, and they successfully bound the error when a learned, inexact denoiser is used.

Datasets

  • Denoiser Training: The underlying deep denoiser (DRUNet architecture) was trained on a massive composite dataset of natural images, including the Berkeley Segmentation Dataset (CBSD), Waterloo Exploration Dataset, DIV2K, and Flickr2K.
  • Evaluation Benchmarks:
  • Deblurring: Evaluated on CBSD10 (a 10-image subset of CBSD68) using 10 different real-world and synthetic blur kernels.
  • Inpainting: Evaluated on the full CBSD68 dataset using random masks removing 50% of the pixels.
  • SAR Despeckling & Super-Resolution: Additional experiments were conducted on Synthetic Aperture Radar (SAR) datasets and standard image patches.

Key Insights and Ideas

  • The "Noisy Input" Advantage: The key insight is that deep denoisers perform best when they are allowed to do exactly what they were trained to do: remove noise. Injecting noise at each step prevents the out-of-distribution drift that plagues traditional PnP methods.
  • A Smoother Landscape: Mathematical and 1D simulations reveal that SNORE acts as a relaxed, smoother version of the traditional negative log-prior. This makes the optimization landscape less rugged and easier for gradient descent to navigate without getting trapped in poor local minima.
  • Bridging Optimization and Diffusion: SNORE demonstrates that the empirical success of reverse-time diffusion processes can be framed and analyzed as a rigorous, mathematically sound stochastic optimization problem.

Key Results

SNORE achieves highly competitive quantitative results and visibly superior qualitative performance compared to state-of-the-art baselines like RED, RED Prox, DiffPIR, and PnP SGD.

  • Inpainting Performance: On the CBSD68 dataset (50% pixel mask), SNORE significantly outperformed competitors in both distortion and perceptual metrics. It achieved a PSNR of 31.94 and an LPIPS of 0.04, compared to RED Prox's 30.31 PSNR and 0.12 LPIPS.
  • Perceptual Realism: Qualitatively, SNORE excels at recovering sharp structures and fine textures (such as fabric patterns or plant details) where traditional methods produce overly smoothed or flat regions.
Figure 2. Deblurring with various methods of a motion blur kernel with input noise level σy = 10/255 with a GS-denoiser trained on natural images. Note that Ann-SNORE produces a better perceptual reconstruction (BRISQUE). Bottom-Right: Decrease of the optimized function J (Equation (16)) along the stochastic gradient descent with the last parameters (αm−1, σm−1) of the annealing procedure.

Limitations and Future Work

The primary limitation of SNORE is its computational cost for deblurring tasks. Because the annealing schedule requires the algorithm to reach convergence across 16 different noise levels (totaling roughly 1500 iterations), it is significantly slower than standard PnP methods (e.g., taking 67 seconds per image compared to RED Prox's 2 seconds). Interestingly, this overhead disappears in inpainting tasks, where SNORE requires no extra computational load relative to RED.

For future work, the authors suggest exploring faster stochastic optimization variants (such as ADAM) to mitigate the computational bottleneck, alongside proving convergence without relying on the standard "boundedness assumption" of image iterates.

Summary updated

Abstract

Plug-and-Play (PnP) algorithms are a class of iterative algorithms that address image inverse problems by combining a physical model and a deep neural network for regularization. Even if they produce impressive image restoration results, these algorithms rely on a non-standard use of a denoiser on images that are less and less noisy along the iterations, which contrasts with recent algorithms based on Diffusion Models (DM), where the denoiser is applied only on re-noised images. We propose a new PnP framework, called Stochastic deNOising REgularization (SNORE), which applies the denoiser only on images with noise of the adequate level. It is based on an explicit stochastic regularization, which leads to a stochastic gradient descent algorithm to solve ill-posed inverse problems. A convergence analysis of this algorithm and its annealing extension is provided. Experimentally, we prove that SNORE is competitive with respect to state-of-the-art methods on deblurring and inpainting tasks, both quantitatively and qualitatively.
BibTeX
@inproceedings{renaud2024plug,
  author = {Marien Renaud and Jean Prost and Arthur Leclaire and Nicolas Papadakis},
  title = {{Plug-and-Play image restoration with Stochastic deNOising REgularization}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=byAXJTk0LH},
}
Figure