FBRNN: feedback recurrent neural network for extreme image super-resolution

Junyeop Lee, Jaihyun Park, Kanghyu Lee, Jeongki Min, Gwantae Kim, Bokyeung Lee, Bonhwa Ku, David K. Han, Hanseok KoWorkshops @ CVPR 2020 · DOI

Introduction

Single Image Super-Resolution (SISR) has traditionally focused on modest upscaling factors like 2x or 4x. However, "extreme" super-resolution—scaling factors of 10x or greater—presents a massive information gap. In a 16x upscale, a single pixel in a low-resolution (LR) image must be expanded into a 256-pixel (16x16) patch.

Standard feed-forward networks often struggle with this task, producing "fuzzy" textures or losing fine semantic details because they attempt to map the transformation in a single pass. FBRNN (Feedback Recurrent Neural Network) addresses this by treating super-resolution as an iterative refinement process rather than a one-shot estimation.

Architecture and Methodology

The FBRNN architecture is built on the principle of iterative refinement through a feedback loop. Unlike standard deep networks that process data linearly, FBRNN uses a series of connected blocks that process the image over multiple iterations ($T$). This allows the network to refine the image from "coarse to fine" over time.

The FBRNN architecture showing the recurrent structure and feedback mechanism across iterations.

Core Components

  1. Recurrent Structure: The model shares weights across iterations, making it highly parameter-efficient.
  2. Feedback Mechanism: At each step $t$, the high-resolution (SR) output is passed through a Back Projection block. This block projects the SR image back down to the LR space, allowing the network to compare its current estimate with the original input. This acts as a self-correcting loop to identify missing details.
  3. Sub-network Blocks:
    • LR Feature Extraction (LFE): Captures initial features from the input.
  4. Feedback (FB) Block: Utilizes Residual Dense Blocks (RDB) to exploit hierarchical features.
  5. Gated Feedback Module (GFM): Manages the flow of information between iterations, deciding what to preserve or update.
  6. Reconstruction Block (RB): Uses deconvolution layers to upscale features back to the target resolution.

Key Results and Performance

FBRNN was evaluated on high-resolution benchmarks including DIV8K and DIV2K. In the NTIRE 2020 Challenge, the FBRNN-based team (KU_ISPLB) secured 6th place in PSNR and 7th place in SSIM globally.

Qualitative comparison of 16x super-resolution results. FBRNN shows significantly higher PSNR/SSIM and sharper details compared to the SRFBN baseline.

Quantitative and Qualitative Gains

On the DIV2K validation set at a 16x scale, FBRNN achieved a PSNR of 28.58 dB, significantly outperforming established baselines like D-DBPN (27.16 dB) and SRFBN+ (28.02 dB). As shown in the visual comparison above, FBRNN produces much sharper edges and fewer "halo" artifacts in complex patterns, such as the stained glass window, compared to previous state-of-the-art methods.

Efficiency

One of the most notable aspects of FBRNN is its efficiency. By tying weights across iterations, the 3-iteration version of the model uses only 1.9M parameters. This allows it to outperform much larger models while maintaining a smaller memory footprint.

Conclusion

FBRNN demonstrates that feedback mechanisms and recurrent structures are highly effective for extreme super-resolution tasks. While there remains a trade-off between mathematical fidelity (PSNR) and perceptual "realness" (often achieved by GANs), FBRNN sets a high bar for pixel-perfect reconstruction. Future work may involve integrating GAN-based losses to further enhance the visual texture of the upscaled images.

Summary updated

BibTeX
@inproceedings{lee2020fbrnn,
  author = {Junyeop Lee and Jaihyun Park and Kanghyu Lee and Jeongki Min and Gwantae Kim and Bokyeung Lee and Bonhwa Ku and David K. Han and Hanseok Ko},
  title = {{FBRNN: feedback recurrent neural network for extreme image super-resolution}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  doi = {10.1109/cvprw50498.2020.00252},
}
Figure