Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution

Wenhan Yang, Jiashi Feng, Jianchao Yang, Fang Zhao, Jiaying Liu, Zongming Guo, Shuicheng Yan

Abstract

In this work, we consider the image super-resolution (SR) problem. The main challenge of image SR is to recover high-frequency details of a low-resolution (LR) image that are important for human perception. To address this essentially ill-posed problem, we introduce a Deep Edge Guided REcurrent rEsidual (DEGREE) network to progressively recover the high-frequency details. Different from most of existing methods that aim at predicting high-resolution (HR) images directly, DEGREE investigates an alternative route to recover the difference between a pair of LR and HR images by recurrent residual learning. DEGREE further augments the SR process with edge-preserving capability, namely the LR image and its edge map can jointly infer the sharp edge details of the HR image during the recurrent recovery process. To speed up its training convergence rate, by-pass connections across multiple layers of DEGREE are constructed. In addition, we offer an understanding on DEGREE from the view-point of sub-band frequency decomposition on image signal and experimentally demonstrate how DEGREE can recover different frequency bands separately. Extensive experiments on three benchmark datasets clearly demonstrate the superiority of DEGREE over well-established baselines and DEGREE also provides new state-of-the-arts on these datasets.

Introduction

Image Super-Resolution (SR) is the task of recovering a high-resolution (HR) image from a low-resolution (LR) counterpart. This task is fundamentally "ill-posed" because a single LR image can correspond to multiple possible HR images. Traditional deep learning approaches that minimize Mean Squared Error (MSE) often result in blurry outputs, as MSE tends to average out high-frequency details like textures and sharp edges. While previous methods utilized image priors or sub-band decomposition, they often relied on shallow architectures or hand-crafted filters that limited their ability to reconstruct complex, sharp details.

The DEGREE Network

The proposed DEGREE (Deep Edge Guided REcurrent rEsidual) network addresses these limitations through a novel architecture that emphasizes edge information and progressive refinement.

The high-level architecture of the DEGREE network, showing how LR images and edge maps are jointly processed to produce HR outputs.

The architecture is defined by three core components:

  1. Recurrent Residual Learning: Instead of a single-step prediction, DEGREE uses a recurrent structure to predict the residue (the difference between LR and HR) across multiple stages. This iterative process allows the network to recover different frequency bands step-by-step.
  2. Deep Edge Guidance: The model explicitly incorporates edge information. It takes an LR edge map (extracted via a Sobel operator) as an additional input and is trained with a joint loss function to predict both the HR image and its corresponding edge map. This forces the model to prioritize sharp boundaries.
  3. Bypass Connections: To enable deeper networks (up to 20 layers) without suffering from vanishing gradients, bypass connections are integrated across layers, significantly speeding up training convergence.

Frequency Decomposition and Insights

A key insight of this work is the interpretation of the recurrent process as sub-band frequency decomposition. Edges are generally more robust than fine textures during downsampling; by explicitly modeling these edges, the network establishes a structural "skeleton" for reconstruction.

Detailed breakdown of the DEGREE architecture illustrating sub-band frequency decomposition and the joint loss mechanism.

Visualization of the recurrent steps confirms that earlier stages focus on recovering coarse structural edges, while later stages refine fine details and textures. This "iterative refinement" mimics classical signal processing concepts like iterative shrinkage but within an end-to-end deep learning framework.

Experimental Results

The DEGREE model was evaluated on standard benchmarks: Set5, Set14, and BSD100. Training was conducted on a standard set of 91 images, augmented to approximately 240,000 sub-images.

  • Quantitative Performance: The 20-layer version (DEGREE-2) consistently outperformed state-of-the-art baselines. For instance, on Set5 (x2 scale), it achieved a PSNR of 37.40 dB, a 0.52 dB improvement over previous top models.
  • Qualitative Quality: Visually, DEGREE produces significantly sharper results with fewer artifacts. It excels at reconstructing complex patterns, such as butterfly wings or window frames, where other methods often introduce "ringing" or blur.
  • Efficiency: Despite its depth, the implementation is highly efficient, processing the entire Set5 dataset in 1.81 seconds on a GPU.

Conclusion

The DEGREE network demonstrates that explicit edge guidance and recurrent residual learning are powerful tools for image super-resolution. By treating the SR problem as a progressive sub-band recovery task, the model achieves superior sharpness and perceptual quality. Future work may involve integrating learned edge extraction layers to further enhance the end-to-end performance and applying the framework to other restoration tasks like JPEG artifact reduction.

Summary updated

Abstract

In this work, we consider the image super-resolution (SR) problem. The main challenge of image SR is to recover high-frequency details of a low-resolution (LR) image that are important for human perception. To address this essentially ill-posed problem, we introduce a Deep Edge Guided REcurrent rEsidual~(DEGREE) network to progressively recover the high-frequency details. Different from most of existing methods that aim at predicting high-resolution (HR) images directly, DEGREE investigates an alternative route to recover the difference between a pair of LR and HR images by recurrent residual learning. DEGREE further augments the SR process with edge-preserving capability, namely the LR image and its edge map can jointly infer the sharp edge details of the HR image during the recurrent recovery process. To speed up its training convergence rate, by-pass connections across multiple layers of DEGREE are constructed. In addition, we offer an understanding on DEGREE from the view-point of sub-band frequency decomposition on image signal and experimentally demonstrate how DEGREE can recover different frequency bands separately. Extensive experiments on three benchmark datasets clearly demonstrate the superiority of DEGREE over well-established baselines and DEGREE also provides new state-of-the-arts on these datasets.
BibTeX
@article{yang2017deep,
  author = {Wenhan Yang and Jiashi Feng and Jianchao Yang and Fang Zhao and Jiaying Liu and Zongming Guo and Shuicheng YAN},
  title = {{Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution}},
  journal = {IEEE Trans. Image Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=_tfjjXytUpC},
}
Figure