TL;DR
This paper introduces a novel, fully convolutional module for arbitrary scale image super-resolution (SR) that is computationally efficient and flexible. By learning scale-informative features, their approach achieves state-of-the-art results across various scale factors and even extends to handling homographic transformations, offering a practical solution for real-world SR applications.
What Problem Does This Paper Address?
Single Image Super-Resolution (SISR) aims to enhance the resolution of an image, which is crucial for applications ranging from medical imaging to satellite analysis. While deep learning has significantly advanced SISR, most existing models are designed for specific, integer scale factors (e.g., 2x, 4x). This leads to a need for separate models for each scale, demanding substantial memory and computational resources for deployment. The demand for a single model capable of arbitrary scale SR (any scale factor) has grown, but current solutions often rely on complex implicit neural representations or meta-learning, which can be computationally expensive and memory-intensive. Furthermore, real-world scenarios often involve not just arbitrary scaling but also geometric distortions like homography, which existing arbitrary scale SR methods may not adequately address.
Methods
The core contribution is a Scale Informative Feature Transform (FT) module. This module is designed to be fully convolutional, making it computationally efficient. The key idea is to transform scale-independent image features into scale-dependent, spatially upscaled features. This is achieved by:
- Feature Extraction: First, scale-independent features are extracted from the low-resolution image using a standard backbone network (like RDN, EDSR, or RCAN).
- Scale and Positional Information: The module explicitly incorporates scale factors and relative positional information of pixels. This information is processed through dedicated convolutional blocks to extract "scale and relative positional features."
- Feature Transformation: These extracted features, along with subsampled image features, are used to estimate transformation matrices (Wa and Wb). These matrices then perform an affine transformation on B-spline interpolated image features, effectively upscaling them to the desired arbitrary scale.
- Flexibility: The FT module is designed to be a plug-and-play component, adaptable to various backbone architectures. It can be applied sequentially for asymmetric scaling (e.g., upscaling width and height separately) and has been extended to handle homographic transformations with minor modifications.
Datasets
- Training and Validation: DIV2K dataset.
- Testing: Standard benchmark datasets for SR: Set5, Set14, B100, Urban100, and Manga109.
- Homographic Transformation: Custom datasets derived from DIV2K validation images with varying degrees of random scaling, sheering, rotation, and projection.
Key Insights and Ideas
- Explicit Scale and Position Encoding: Instead of relying solely on implicit representations, explicitly encoding scale and relative positional information proves highly effective for arbitrary scale SR.
- Fully Convolutional for Efficiency: Adopting a fully convolutional approach for the upscaling module significantly reduces computational complexity and memory footprint compared to methods using pixel-wise MLPs or implicit neural representations.
- Unified Module for Arbitrary and Homographic SR: The FT module's architecture allows it to be adapted for homographic transformation with minimal changes, demonstrating its flexibility and robustness.
- Scale-Independent Feature Reuse: Extracting scale-independent features once and then transforming them for arbitrary scales is more efficient than re-extracting features for each scale.
Key Results
- Arbitrary Scale SR: The proposed FT module, when integrated with various backbones (RDN, RCAN, EDSR), achieves state-of-the-art PSNR results across multiple datasets and arbitrary scale factors, outperforming methods like Meta-SR, LIIF, LTE, and OPE.
- Asymmetric Scale SR: The model demonstrates comparable or better performance than existing methods on asymmetric scale SR tasks.
- Integer Scale SR: The method achieves competitive results for integer scale SR, even for "out-of-distribution" scale factors (scales not seen during training), by employing a multi-stage upscaling strategy.
- Homographic Transformation: The FT module shows strong performance in super-resolution under homographic transformation, outperforming several baselines, particularly in restoring complicated distortions.
- Computational Efficiency: The FT module boasts significantly fewer trainable parameters, lower memory consumption, and faster inference times compared to many existing arbitrary scale SR methods. For instance, FT-EDSR* uses only 0.13 million parameters and consumes 45.541 MB of memory, with a fast inference time.
Limitations and Future Work
- Out-of-Distribution Scaling: For integer scale SR, the paper notes that for "out-of-distribution" scale factors, they adopt a strategy of dividing the upscaling into multiple in-distribution stages. While this improves performance, it's acknowledged that the model cannot learn continuous functions as directly as implicit neural representation-based methods.
- Future Work: The authors suggest exploring innovative strategies for extracting scale features and developing better mechanisms for fusing scale and image features to further enhance arbitrary scale upscaling modules. They also highlight the potential for this module to be integrated into other encoder-decoder-based image reconstruction models.
Summary updated