Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution

Wenhan Yang, Jiashi Feng, Jianchao Yang, Fang Zhao, Jiaying Liu, Zongming Guo, Shuicheng Yan

TL;DR

The DEGREE (Deep Edge Guided REcurrent rEsidual) network tackles image super-resolution by progressively recovering high-frequency details through a recurrent residual architecture. By explicitly incorporating edge maps as guidance and using bypass connections to speed up training, it achieves sharper, more perceptually pleasing results than standard MSE-based deep learning models.

What Problem Does This Paper Address?

Image Super-Resolution (SR)—the task of recovering a high-resolution (HR) image from a low-resolution (LR) one—is an "ill-posed" problem. Because multiple HR images can correspond to the same LR image, simply minimizing the Mean Squared Error (MSE) often leads to blurry results. MSE tends to average out possible high-frequency details like textures and sharp edges.

While previous methods tried to use "priors" (assumptions about how natural images look) or sub-band decomposition (breaking the image into different frequency layers), they often relied on hand-crafted filters or shallow architectures that couldn't capture the complex mapping required for truly sharp images.

Methods

The authors introduce the DEGREE network, which stands out through three main architectural choices:

  1. Recurrent Residual Learning: Instead of predicting the HR image in one giant leap, the network uses a recurrent structure. It predicts the residue (the difference) between the LR and HR images across multiple stages. This mimics sub-band decomposition, where the network learns to recover different frequency bands step-by-step.
  2. Deep Edge Guidance: The model doesn't just look at the raw pixels. It takes the LR image's edge map (extracted via a Sobel operator) as an additional input. Furthermore, the network is trained with a joint loss function: it must simultaneously predict the HR image and the HR edge map. This forces the model to prioritize sharp boundaries.
  3. Bypass Connections: To prevent the vanishing gradient problem common in deep networks and to speed up convergence, the authors added bypass connections across layers. This allows the network to be deeper (up to 20 layers) while remaining trainable.

Datasets

The model was evaluated on three standard benchmarks in the SR community:

  • Set5: A small dataset of 5 images often used for quick testing.
  • Set14: 14 images with more varied content.
  • BSD100: 100 images from the Berkeley Segmentation Dataset, representing diverse natural scenes.

For training, they used a standard set of 91 images, augmented with flips and rotations to create a final training set of approximately 240,000 sub-images.

Key Insights and Ideas

  • Edges are Easier than Textures: The authors argue that while fine textures are often destroyed during downsampling, edge information is more robust. By explicitly modeling these edges, the network has a "skeleton" to build upon when reconstructing high-frequency details.
  • Recurrence as Refinement: By using a recurrent structure, the network effectively performs "iterative shrinkage," a concept from classical signal processing, but within an end-to-end trainable deep learning framework.
  • Sub-band Visualization: One of the most interesting parts of the paper is the visualization of the recurrent steps. The authors show that earlier stages of the recurrence focus on coarse edges, while later stages "fix" fine details and textures, validating their sub-band decomposition intuition.

Key Results

  • Quantitative Gains: DEGREE-2 (the 20-layer version) consistently outperformed state-of-the-art methods like SRCNN and CSCN. On the Set5 dataset (x2 scale), it achieved a PSNR of 37.40 dB, a significant 0.52 dB improvement over the best baseline at the time.
  • Qualitative Sharpness: Visually, the model excels at removing artifacts around window boundaries and butterfly wing patterns. Unlike other methods that produce "ringing" artifacts or blurry edges, DEGREE produces clean, sharp transitions.
  • Efficiency: Despite the recurrent nature, the GPU implementation is fast, taking only 1.81 seconds to process all images in the Set5 dataset.

Limitations and Future Work

  • Scaling Factors: While the model performs exceptionally well at x2 scaling, the performance gain narrows slightly at x4 scaling. The authors note that edge features are harder to predict as the scaling factor increases.
  • Fixed Edge Detector: The current model relies on a hand-crafted Sobel operator for initial edge extraction. A potential future direction would be to learn the edge extraction layer itself as part of the end-to-end process.
  • Generalization: The authors briefly mention that the framework could be applied to other tasks like JPEG artifact reduction, suggesting the "edge-guided" philosophy is broadly applicable to low-level vision problems.

Summary updated

Abstract

In this paper, we consider the image super-resolution (SR) problem. The main challenge of image SR is to recover high-frequency details of a low-resolution (LR) image that are important for human perception. To address this essentially ill-posed problem, we introduce a Deep Edge Guided REcurrent rEsidual (DEGREE) network to progressively recover the high-frequency details. Different from most of the existing methods that aim at predicting high-resolution (HR) images directly, the DEGREE investigates an alternative route to recover the difference between a pair of LR and HR images by recurrent residual learning. DEGREE further augments the SR process with edge-preserving capability, namely the LR image and its edge map can jointly infer the sharp edge details of the HR image during the recurrent recovery process. To speed up its training convergence rate, by-pass connections across the multiple layers of DEGREE are constructed. In addition, we offer an understanding on DEGREE from the view-point of sub-band frequency decomposition on image signal and experimentally demonstrate how the DEGREE can recover different frequency bands separately. Extensive experiments on three benchmark data sets clearly demonstrate the superiority of DEGREE over the well-established baselines and DEGREE also provides new state-of-the-arts on these data sets. We also present addition experiments for JPEG artifacts reduction to demonstrate the good generality and flexibility of our proposed DEGREE network to handle other image processing tasks.
BibTeX
@article{yang2017deep,
  author = {Wenhan Yang and Jiashi Feng and Jianchao Yang and Fang Zhao and Jiaying Liu and Zongming Guo and Shuicheng YAN},
  title = {{Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution}},
  journal = {IEEE Trans. Image Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=_tfjjXytUpC},
}

Loading related content...

Figure