Dynamically Unfolding Recurrent Restorer: A Moving Endpoint Control Method for Image Restoration

Xiaoshuai Zhang, Yiping Lu, Jiaying Liu, Bin Dong

Introduction

Image restoration—such as denoising or removing JPEG compression artifacts—is a delicate balancing act. Apply too little restoration, and the image remains grainy; apply too much, and fine details are lost, resulting in an artificial, "plastic" appearance.

Traditionally, achieving this balance required manual parameter tuning based on specific degradation levels (e.g., the noise variance $\sigma$). While deep learning has significantly advanced the state-of-the-art, most current models remain "brittle." They are typically trained for specific noise levels and struggle when faced with unknown or varying corruption. To handle "blind" restoration, researchers often resort to massive, parameter-heavy networks.

To address this, researchers have proposed the Dynamically Unfolding Recurrent Restorer (DURR). DURR is a novel framework that treats image restoration as an optimal control problem with a "moving endpoint." By pairing a convolutional Recurrent Neural Network (RNN) with a reinforcement learning-based policy unit, the model adaptively decides how many iterations are needed to clean an image. This allows a single, lightweight model to outperform specialized, state-of-the-art networks across varying noise and compression levels.

The Moving Endpoint Framework

The DURR architecture consists of two primary components that work in tandem:

1. The Restoration Unit (The "Muscle") This unit is a convolutional RNN. Drawing on the mathematical link between Residual Networks (ResNets) and Ordinary Differential Equations (ODEs), the authors treat the restoration process as a dynamic system. Each "unfolding" (iteration) of the RNN acts as a step in a diffusion process that gradually smooths out noise.

Restoration Dynamics Diagram

2. The Policy Unit (The "Brain") This unit determines the "moving endpoint"—essentially, when to stop the iterative restoration process. It evaluates the current intermediate image and decides whether to pass it back to the restoration unit for another round or output it as the final, clean result.

Training Strategy The model is trained in two stages. First, the restoration unit is trained using a "refined policy" where the number of iterations is manually scaled with the noise level. Second, the policy unit is trained using Deep Q-Learning (RL). The reward function for the RL agent is based on the improvement in image quality at each step, incentivizing the model to halt exactly when the image reaches its peak quality.

Experimental Setup and Key Insights

The researchers evaluated DURR using standard imaging benchmarks:

  • BSD500: 400 images used for training.
  • BSD68: Used for evaluating image denoising performance.
  • LIVE1: Used for evaluating JPEG deblocking.
  • Real-world noise: A dataset of real-world noisy images (Lebrun et al., 2015) was used to test practical utility.

Key Insights

  • Depth as a Variable: Instead of building a deeper network to handle more noise, DURR simply runs the same small network for more iterations. This makes the model significantly more parameter-efficient than static competitors like DnCNN.
  • Generalization via Time: DURR generalizes to degradation levels it never saw during training. Because it learns a "restoration dynamic" rather than a fixed mapping, it can handle extreme noise (e.g., $\sigma=75$) simply by running for more steps, even if it was only trained on noise up to $\sigma=55$.
  • Optimal Control: The paper successfully bridges the gap between classical PDE-based imaging (like nonlinear diffusion) and modern deep reinforcement learning.

Results

DURR achieves state-of-the-art results on blind denoising and JPEG deblocking while remaining highly efficient:

  • Superior Performance: On the BSD68 dataset with a noise level of $\sigma=35$, DURR achieved a PSNR of 27.72 dB, outperforming the popular DnCNN-B model (27.66 dB).
  • Robustness to Unseen Data: In extreme cases ($\sigma=95$, far outside the training range), DURR maintained structural integrity and achieved significantly higher PSNR (up to 10 dB higher) than DnCNN, which produced garbled results.
  • Parameter Efficiency: The total parameter count for DURR (Restoration + Policy units) is roughly $2.8 \times 10^5$, which is less than half the size of DnCNN ($7.0 \times 10^5$).
  • Real-World Utility: The model performed exceptionally well on real-world noisy images without any additional fine-tuning, producing sharper details and fewer artifacts than competing methods.

Limitations and Future Work

While the results are impressive, the authors note a few areas for future exploration:

  • Task Expansion: The current framework focuses on denoising and deblocking, but it could be extended to deraining, dehazing, and deblurring.
  • Policy Complexity: The policy unit currently uses a combination of ResUnits and an LSTM. Future work could explore simpler architectures for the "stopping" decision to reduce computational overhead.
  • Inference Time: While the model has fewer parameters, running many iterations on severely degraded images may increase inference time compared to a single-pass feed-forward network.

Summary updated

Abstract

In this paper, we propose a new control framework called the moving endpoint control to restore images corrupted by different degradation levels in one model. The proposed control problem contains a restoration dynamics which is modeled by an RNN. The moving endpoint, which is essentially the terminal time of the associated dynamics, is determined by a policy network. We call the proposed model the dynamically unfolding recurrent restorer (DURR). Numerical experiments show that DURR is able to achieve state-of-the-art performances on blind image denoising and JPEG image deblocking. Furthermore, DURR can well generalize to images with higher degradation levels that are not included in the training stage.
BibTeX
@inproceedings{zhang2019dynamically,
  author = {Xiaoshuai Zhang and Yiping Lu and Jiaying Liu and Bin Dong},
  title = {{Dynamically Unfolding Recurrent Restorer: A Moving Endpoint Control Method for Image Restoration}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=jUkwZ_nxKY},
}
Figure