A Deep Journey into Super-resolution: A survey

Saeed Anwar, Salman Khan, Nick Barnes

This survey paper, "A Deep Journey into Super-resolution," provides a comprehensive roadmap through the rapidly evolving landscape of deep learning-based Single Image Super-Resolution (SISR). By benchmarking over 30 state-of-the-art architectures across six datasets, the authors establish a clear taxonomy and highlight the shift from simple convolutional stacks to complex, attention-driven, and perceptually-motivated models.

TL;DR

The authors provide a rigorous taxonomy and comparative analysis of over 30 deep learning models for single-image super-resolution, evaluating them on both classical and modern datasets. The survey highlights how architectural innovations—like residual learning and attention mechanisms—have drastically improved performance, while also identifying the persistent gap between quantitative metrics (PSNR) and human visual perception.

What Problem Does This Paper Address?

Image Super-Resolution (SR) is the process of recovering a high-resolution (HR) image from a low-resolution (LR) input. It is a classic "ill-posed" inverse problem: for any given LR image, there are infinitely many possible HR versions that could have produced it.

While traditional interpolation methods (like bicubic) often result in blurry outputs, deep learning has revolutionized the field. However, the sheer volume of new architectures makes it difficult for researchers to track which innovations actually drive performance. This paper addresses the need for a systematic categorization of these methods, a standardized comparison of their computational complexity (parameters and FLOPs), and an honest assessment of their performance on challenging, modern datasets.

Methods

The authors introduce a new taxonomy that groups SISR methods into nine distinct categories based on their structural design:

  1. Linear Networks: Simple stacks like the pioneering SRCNN.
  2. Residual Networks: Utilizing skip connections to train much deeper models (e.g., EDSR).
  3. Recursive Networks: Sharing parameters across layers to reduce memory footprint (e.g., DRRN).
  4. Progressive Reconstruction: Predicting HR images in stages (e.g., LapSRN).
  5. Densely Connected Networks: Maximizing feature reuse (e.g., RDN).
  6. Multi-branch Designs: Capturing features at multiple scales (e.g., CMSC).
  7. Attention-based Networks: Focusing on the most informative features or channels (e.g., RCAN).
  8. Multiple Degradation Handling: Moving beyond simple bicubic downsampling (e.g., SRMD).
  9. GAN Models: Using adversarial training to generate realistic textures (e.g., ESRGAN).

![Fig.

  1. The taxonomy of the existing single-image super-resolution techniques based on the most distinguishing features.](/figures/art_440aa6eb2c17413b9a2aeb0e85831a02)

The paper also details the evolution of "upsampling" strategies, distinguishing between early upsampling (which is computationally expensive) and late upsampling (like sub-pixel convolution), which processes features in the LR space for better efficiency.

![Fig.

  1. A glimpse of the diverse range of network architectures used for single-image super-resolution using deep networks.](/figures/art_5fbc38e393d840f8967a8416321d983b)

Datasets

The survey evaluates models on a mix of benchmark datasets:

  • Classical: Set5, Set14, and BSD100 (small, standard sets).
  • Challenging/Modern:
    • Urban100: Focuses on man-made structures with repetitive patterns.
  • DIV2K: A high-quality 2K resolution dataset used in NTIRE competitions.
  • Manga109: A collection of Japanese manga, testing the ability to reconstruct fine lines and artistic textures.

Key Insights and Ideas

  • The Residual Revolution: Residual learning (skip connections) is identified as the single most important factor in enabling deeper, more accurate networks. It allows the model to focus on learning the "residue" (high-frequency details) rather than the entire image.
  • Efficiency via Late Upsampling: Moving the upsampling layer to the end of the network (as seen in FSRCNN and ESPCN) significantly reduces computational cost and memory footprint without sacrificing quality.
  • The PSNR-Perception Trade-off: Models optimized for Peak Signal-to-Noise Ratio (PSNR) often produce "safe," blurry results. Conversely, Generative Adversarial Networks (GANs) produce much sharper, visually pleasing textures but often score lower on PSNR because the generated pixels don't perfectly align with the ground truth.
  • Attention is All You Need (for SR too): Recent top-performers like RCAN demonstrate that channel attention—allowing the network to weigh different feature maps differently—leads to significant gains in reconstructing complex textures.

Key Results

  • Depth and Complexity: The field has moved from SRCNN (3 layers, 57k parameters) to models like RCAN (500+ layers, 16M parameters).
  • Performance Leaders: At the time of publication, DRLN and RCAN provided the best quantitative results (PSNR/SSIM) across most datasets. For visual quality, ESRGAN was highlighted as a leader in generating realistic textures.
  • Scaling Challenges: While most models perform well at 2x and 4x scales, performance drops significantly at 8x, where artifacts become highly visible and textures are difficult to hallucinate accurately.

Limitations and Future Work

The authors identify several "open problems" that the community still needs to solve:

  • Real-world Degradation: Most current models are trained on synthetic "bicubic" downsampling. They often fail when applied to real-world images with unknown noise, sensor artifacts, or motion blur.
  • Evaluation Metrics: PSNR and SSIM are poor proxies for human vision. There is a critical need for better perceptual metrics that can quantify "naturalness."
  • Unified Models: Most current networks are specialized for a single scale (e.g., 4x). Developing unified models that handle arbitrary scaling factors and multiple types of degradation simultaneously remains a challenge.
  • Extreme SR: Upsampling beyond 8x remains largely unsolved, as it requires the model to incorporate strong semantic priors (knowing what an object is) to reconstruct missing details.

Summary updated

Abstract

Deep convolutional networks based super-resolution is a fast-growing field with numerous practical applications. In this exposition, we extensively compare 30+ state-of-the-art super-resolution Convolutional Neural Networks (CNNs) over three classical and three recently introduced challenging datasets to benchmark single image super-resolution. We introduce a taxonomy for deep-learning based super-resolution networks that groups existing methods into nine categories including linear, residual, multi-branch, recursive, progressive, attention-based and adversarial designs. We also provide comparisons between the models in terms of network complexity, memory footprint, model input and output, learning details, the type of network losses and important architectural differences (e.g., depth, skip-connections, filters). The extensive evaluation performed, shows the consistent and rapid growth in the accuracy in the past few years along with a corresponding boost in model complexity and the availability of large-scale datasets. It is also observed that the pioneering methods identified as the benchmark have been significantly outperformed by the current contenders. Despite the progress in recent years, we identify several shortcomings of existing techniques and provide future research directions towards the solution of these open problems.
BibTeX
@article{anwar2019deep,
  author = {Saeed Anwar and Salman Khan and Nick Barnes},
  title = {{A Deep Journey into Super-resolution: A survey}},
  journal = {ACM Computing Surveys},
  year = {2019},
  doi = {10.1145/3390462},
}
Figure