Abstract
Image Super-Resolution (SR) is an important class of image processing techniques to enhance the resolution of images and videos in computer vision. Recent years have witnessed remarkable progress of image super-resolution using deep learning techniques. This article aims to provide a comprehensive survey on recent advances of image super-resolution using deep learning approaches. In general, we can roughly group the existing studies of SR techniques into three major categories: supervised SR, unsupervised SR, and domain-specific SR. In addition, we also cover some other important issues, such as publicly available benchmark datasets and performance evaluation metrics. Finally, we conclude this survey by highlighting several future directions and open issues which should be further addressed by the community in the future.
Introduction
Image Super-Resolution (SR) is the process of recovering a high-resolution (HR) image from a single low-resolution (LR) input. This is an inherently ill-posed problem: for any given LR image, there are infinitely many possible HR versions that could have produced it. Traditional methods like bicubic interpolation often result in blurry edges and lost textures. While deep learning has significantly pushed the state-of-the-art, the field has become fragmented with a dizzying array of architectures. This survey provides a unified framework to understand how different components, such as upsampling layers and attention mechanisms, contribute to the final result.
A Taxonomy of Super-resolution Methods
The survey systematizes hundreds of methods into a coherent taxonomy, covering everything from basic architectural building blocks to advanced learning strategies and domain-specific applications.
Model Frameworks
The authors categorize SR models based on four primary architectural frameworks, each with distinct computational trade-offs:
- Pre-upsampling SR: Upscales the image first (e.g., via bicubic interpolation), then refines it with a CNN. While simple, it is computationally expensive as most operations occur in the high-resolution space.
- Post-upsampling SR: Performs most computations in the LR space and upscales at the very end. This is the current "gold standard" for efficiency.
- Progressive Upsampling SR: Upscales in stages (e.g., 2x → 4x → 8x) to handle large scaling factors more stably.
- Iterative Up-and-down Sampling SR: Uses back-projection to iteratively refine the HR estimate by alternating between upsampling and downsampling.
Upsampling and Network Design
Beyond the general framework, models differ in their Upsampling Methods (e.g., Transposed Convolutions, Sub-pixel Layers, or Meta-Upscale Modules) and Network Design. Modern architectures often incorporate Residual Learning to train deeper networks, Dense Connections, and Channel Attention to focus on high-frequency features.
Learning Strategies
There has been a significant shift from simple $L_1/L_2$ pixel losses toward more complex strategies:
- Perceptual/Content Loss: Uses VGG features to match high-level semantics.
- Adversarial Loss: Employs GANs to generate "hallucinated" but realistic details.
- Texture Loss: Uses Gram matrices to produce visually convincing patterns.
Datasets and Evaluation
Benchmarking is critical for progress in SR. The paper identifies several foundational datasets:
- Small Benchmarks: Set5, Set14, BSDS300, and Urban100.
- Large-scale Training: DIV2K (1,000 high-quality 2K images) and Flickr2K.
- Domain-Specific: Manga109 (illustrations) and CelebA (faces).
Performance is typically measured via objective metrics like PSNR and SSIM, though subjective measures like Mean Opinion Score (MOS) are increasingly important for evaluating perceptual quality.
Key Insights
- The Perception-Distortion Trade-off: There is a mathematical limit to how well a model can minimize pixel error (PSNR) while maximizing perceptual "realness." GAN-based models often look better to humans but score poorly on PSNR.
- Efficiency via Post-upsampling: Moving upsampling to the end of the network was a breakthrough that allowed for much deeper networks by reducing the spatial resolution of intermediate feature maps.
- Zero-Shot SR (ZSSR): Some models can be trained on a single test image by exploiting internal recurrence, where small patches repeat across different scales within the same image.
Results and Future Directions
State-of-the-art models like RCAN (Residual Channel Attention Network) and EDSR (Enhanced Deep SR) lead in PSNR-based benchmarks, while ESRGAN (Enhanced SRGAN) is noted for producing the most visually convincing results.
Looking forward, the field faces several challenges:
- Metric Mismatch: PSNR and SSIM are poor proxies for human vision; better automated metrics are needed.
- Real-world Robustness: Most models fail when faced with real-world sensor noise or unknown degradation kernels.
- Arbitrary Scaling: Developing "one-size-fits-all" models for continuous zooming remains an open problem.
- Video Consistency: Maintaining temporal consistency (preventing flickering) in video SR is a major area for future research.
Summary updated