Stacking Networks Dynamically for Image Restoration Based on the Plug-and-Play Framework

Haixin Wang, Tianhao Zhang, Muzhi Yu, Jinan Sun, Wei Ye, Chen Wang, Shikun Zhang

Abstract

Image restoration (IR) tasks, such as denoising and deblurring, have increasingly relied on "stacked" neural network architectures to achieve high performance. However, these static designs often require manual hyper-parameter tuning and lack theoretical flexibility. This paper introduces a dynamic stacking framework that embeds pre-trained sub-networks into a mathematical Plug-and-Play (PnP) optimization scheme. By iteratively updating an adaptive degradation model, the framework focuses on heavily corrupted spatial regions, achieving state-of-the-art results on standard benchmarks like BSD68 and GoPro without additional training.

Introduction

In the field of Image Restoration (IR), "stacked" networks have become a dominant architectural trend. By layering multiple sub-networks, models can refine an image step-by-step to remove noise or motion blur. Despite their success, these architectures are typically static; the number of stacking levels is a hyper-parameter that researchers must determine through trial and error. This leads to two major issues: inflexibility, where a model may be overkill for light corruption but insufficient for heavy corruption, and a lack of theoretical grounding for the specific depth chosen.

This research addresses these limitations by proposing a dynamic approach where the network effectively "stacks itself" through iterations. Guided by a rigorous mathematical Plug-and-Play (PnP) framework, this method allows for a flexible restoration process that adapts to the specific needs of the input image.

Methodology

The core of the proposed method is the integration of deep learning priors into a classical iterative optimization algorithm. Instead of building a massive, fixed-depth network, the researchers use a single pre-trained network and pass the image through it multiple times, with each iteration acting as a new "level" in a stacked network.

The proposed dynamic Plug-and-Play framework, illustrating the iterative process between the forward model, deep prior, and adaptive degradation update.

As shown in the framework overview, the process is divided into two main stages:

  1. Initialization (Stage 1): Parameters such as the initial image estimate $x^{(0)}$ and the degradation matrix $A$ are established.
  2. Deep Plug-and-Play (Stage 2): The model enters an iterative loop comprising four key components:
  3. Forward Model (2.1): An updated estimate $x^{(t+1)}$ is generated using a gradient descent step.
  4. Dynamic Stacked Networks (2.2): The image passes through the deep prior, simulating a stacked architecture.
  5. Deep Prior (2.3): A pre-trained network $f(x^{(t+1)}; \theta)$ is "plugged in" to regularize the restoration.
  6. Adaptive Degradation Model (2.4): A critical innovation where the degradation matrix $A^{(t+1)}$ is updated in every iteration. This allows the model to adaptively "shift its focus" to the most corrupted spatial areas as the restoration progresses.

To maintain computational efficiency, the authors replace complex closed-form solutions with a single step of gradient descent per iteration, providing mathematical proof that this approach still converges to an optimal solution.

Experimental Results

The framework was evaluated on standard benchmarks for two primary tasks: image denoising and motion deblurring.

  • Image Denoising: Tested on the BSD68 and Set12 datasets. On BSD68 (at noise level $\sigma=50$), the proposed method achieved a PSNR of 27.64 dB, outperforming state-of-the-art models like VDN (27.18 dB) and DnCNN (26.23 dB).
  • Motion Deblurring: Tested on the GoPro dataset. The framework improved the performance of the DMPHN (a leading deblurring model) by 0.15 to 0.24 dB across various configurations without any additional training of the underlying model weights.

Key Insights

The most striking innovation is the "shift of focus" enabled by the adaptive degradation model. By updating the matrix $A$ during the process, the framework identifies which parts of the image remain "messy" and forces the deep prior to work harder on those specific regions in subsequent iterations. This results in sharper edges and fewer artifacts compared to traditional methods like BM3D.

Furthermore, the method is highly parameter-efficient. Because it reuses the same pre-trained weights across iterations, it achieves the performance of a much deeper stacked network without increasing the number of trainable parameters. Convergence is also remarkably fast, typically reaching peak performance in just 2 to 4 iterations.

Conclusion and Future Work

The dynamic PnP framework provides a mathematically grounded alternative to static network stacking. While the authors noted a performance "peak" after 4 or 5 iterations—attributing a subsequent decrease in PSNR to over-fitting caused by gradient descent—the method consistently outperforms static baselines. Future research will explore internal architecture adjustments to mitigate over-fitting and automate the tuning of step-size parameters for different datasets.

Summary updated

BibTeX
@inproceedings{wang2020stacking,
  author = {Haixin Wang and Tianhao Zhang and Muzhi Yu and Jinan Sun and Wei Ye and Chen Wang and Shikun Zhang},
  title = {{Stacking Networks Dynamically for Image Restoration Based on the Plug-and-Play Framework}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58601-0\_27},
}
Figure