Dynamically Unfolding Recurrent Restorer: A Moving Endpoint Control Method for Image Restoration

Xiaoshuai Zhang, Yiping Lu, Jiaying Liu, Bin Dong

Introduction

Image restoration tasks, such as denoising and JPEG artifact removal, face a significant challenge: the variability of degradation levels. In real-world scenarios, the intensity of noise or compression is often unknown and non-uniform. Traditional deep learning approaches typically employ fixed-depth architectures, applying a constant amount of computation regardless of the image's corruption level. This lack of adaptivity often leads to poor generalization, where models trained on specific noise levels fail when encountering more severe degradation.

To address this, we introduce the Dynamically Unfolding Recurrent Restorer (DURR). By framing image restoration as a "moving endpoint" control problem, DURR adaptively determines the optimal number of iterations required for each specific image, allowing for a flexible and efficient restoration process.

Moving Endpoint Control Framework

The DURR framework treats image restoration as a dynamic system where the "terminal time" (the number of processing steps) is a variable to be optimized. The system consists of two primary modules:

  1. Restoration Unit (RU): This unit is a convolutional Recurrent Neural Network (RNN) with a U-Net-like architecture. It is designed to mimic nonlinear diffusion processes, which are effective at smoothing noise while preserving critical edge information. As an RNN, the RU can be iteratively applied to the image, progressively refining the result.
  2. Policy Unit (PU): Acting as the controller, the PU decides when to stop the restoration process. It utilizes ResNet blocks and an LSTM cell to analyze the current state of the image at each iteration. The PU outputs a binary decision: "Again" (continue processing) or "Done" (terminate and output the result).

Training Strategy

The model is trained in a two-stage process to ensure both restoration quality and decision-making accuracy:

  • Stage 1: Restoration Unit Optimization. The RU is trained using a "refined policy" where optimal iteration counts are pre-determined for various noise levels. This ensures the RU learns an effective iterative mapping.
  • Stage 2: Policy Unit Reinforcement Learning. The PU is trained using Deep Q-learning. The reward function is tied to the improvement in Peak Signal-to-Noise Ratio (PSNR). The agent receives a positive reward for increasing image quality and a penalty for each additional iteration, encouraging efficiency.

Experimental Results

DURR was evaluated on standard benchmarks including BSD68 for denoising and LIVE1 for JPEG deblocking. The results demonstrate significant advantages over state-of-the-art fixed-depth models like DnCNN.

Generalization to Extreme Degradation

One of the most striking features of DURR is its ability to generalize to degradation levels far beyond those seen during training. For instance, when faced with extreme Gaussian noise ($\sigma = 95$), DURR maintains high visual fidelity and a PSNR of 24.94 dB. In contrast, fixed-depth models like DnCNN often collapse, producing results with significantly lower PSNR (e.g., 14.46 dB) and severe artifacts.

Comparison of restoration performance between DnCNN and DURR on extreme noise levels. DURR shows superior visual quality and significantly higher PSNR values.

Efficiency and Parameter Count

Despite its iterative nature, DURR is highly parameter-efficient. It achieves superior performance compared to DnCNN-B while using fewer parameters. Furthermore, the "Doppio-DURR" variant demonstrates that reducing network width while allowing for more iterations can lead to faster inference times without sacrificing quality.

Conclusion

The Dynamically Unfolding Recurrent Restorer represents a paradigm shift from fixed-depth to dynamic-depth architectures in image restoration. By leveraging moving endpoint control and reinforcement learning, DURR provides a robust solution that adapts to unknown degradation levels. Its exceptional generalization capabilities and parameter efficiency make it a promising framework for various ill-posed inverse problems in computer vision, such as deraining and deblurring.

Summary updated

Abstract

In this paper, we propose a new control framework called the moving endpoint control to restore images corrupted by different degradation levels in one model. The proposed control problem contains a restoration dynamics which is modeled by an RNN. The moving endpoint, which is essentially the terminal time of the associated dynamics, is determined by a policy network. We call the proposed model the dynamically unfolding recurrent restorer (DURR). Numerical experiments show that DURR is able to achieve state-of-the-art performances on blind image denoising and JPEG image deblocking. Furthermore, DURR can well generalize to images with higher degradation levels that are not included in the training stage.
BibTeX
@inproceedings{zhang2019dynamically,
  author = {Xiaoshuai Zhang and Yiping Lu and Jiaying Liu and Bin Dong},
  title = {{Dynamically Unfolding Recurrent Restorer: A Moving Endpoint Control Method for Image Restoration}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=jUkwZ_nxKY},
}
Figure