TL;DR
This paper introduces SCAN, a novel lightweight Convolutional Neural Network (CNN) for image super-resolution (SR). SCAN effectively balances performance and computational cost by employing large-kernel convolutions and a unique spatial and channel aggregation mechanism, outperforming existing lightweight SR methods on benchmark and remote sensing datasets.
What Problem Does This Paper Address?
Single Image Super-Resolution (SISR) aims to enhance the resolution of low-resolution (LR) images to produce high-resolution (HR) images. While deep learning methods have achieved remarkable success, many state-of-the-art SISR models are computationally expensive, limiting their practical deployment on resource-constrained devices like those used in edge computing. The challenge lies in developing SISR methods that are both high-performing and computationally efficient, striking a balance between image quality and speed.
Methods
The proposed Spatial and Channel Aggregation Network (SCAN) is a pure CNN-based architecture designed for lightweight image super-resolution. Its key components include:
- Shallow Feature Extraction Module: A simple 3x3 convolution to extract initial low-level features.
- Deep Feature Extraction Module: This module is the core of SCAN and consists of cascaded Spatial and Channel Aggregation Groups (SCAGs).
- Spatial and Channel Aggregation Block (SCAB): Each SCAB aggregates multi-order spatial and channel information. It comprises:
- Triple-Scale Spatial Aggregation (TSSA) Attention Module: This novel module aggregates contextual information at three different scales. It utilizes a Feature Decomposition (FD) module to remove redundant interactions and a Triple-Scale Gated Aggregation (TSGA) module to combine multi-scale features. A key innovation here is the use of large-kernel depth-wise dilated convolutions (5x5, 7x7, 9x9) with increasing dilation ratios.
- Channel Aggregation (CA) Module: This module aggregates channel information, inspired by MogaNet. It uses a combination of 1x1 convolutions, 3x3 depth-wise convolutions, and a Multi-Scale Feature Reallocation (MSFR) module to refine channel-specific features.
- Large Kernel Convolution Tail: A 9x9 depth-wise dilated convolutional layer with a dilation ratio of 4 is appended to the SCAGs to further expand the receptive field.
- Image Reconstruction Module: Combines the shallow and deep features using a pixel shuffle operation and a 3x3 convolution for the final HR image generation.
SCAN leverages Metaformer-style architectures and incorporates large-kernel convolutions, a strategy that has shown promise in recent vision tasks.
Datasets
The paper uses standard datasets for training and evaluation:
- Training: Flickr2K and DIV2K.
- Evaluation (Benchmark): Set5, Set14, BSD100, Urban100, and Manga109.
- Evaluation (Remote Sensing): DIOR and DOTA.
Key Insights and Ideas
- Large Kernels for Lightweight Models: The paper demonstrates that large-kernel convolutions, often associated with larger models, can be effectively integrated into lightweight architectures to boost performance by increasing the receptive field and capturing richer contextual information.
- Triple-Scale Spatial Aggregation: Aggregating information at multiple scales (triple-scale) is crucial for capturing diverse contextual cues, leading to improved feature representation.
- Feature Decomposition for Focus: The FD module within TSSA helps the network focus on more relevant intermediate-level information by reducing redundant interactions.
- Channel Aggregation for Efficiency: The CA module, inspired by MogaNet, efficiently aggregates channel information, contributing to the model's overall performance and lightweight nature.
- Strategic Placement of Large Kernels: Using a 9x9 large kernel convolution at the end of the deep feature extraction module is a deliberate choice to maximize its impact on capturing global context.
Key Results
- Benchmark Datasets: SCAN achieves a 0.13 dB improvement in PSNR and a 0.0013 increase in SSIM compared to state-of-the-art lightweight SISR methods on benchmark datasets.
- Remote Sensing Datasets: On remote sensing datasets, SCAN shows a more significant improvement of 0.4 dB in PSNR and a 0.0033 increase in SSIM, highlighting its effectiveness on challenging imagery.
- Computational Efficiency: SCAN achieves comparable or superior performance with fewer parameters and lower computational complexity (FLOPs) compared to existing lightweight methods. For example, SCAN-tiny achieves comparable performance to MAN-tiny with similar parameters and FLOPs, and outperforms ShuffleMixer-tiny with only slightly higher complexity.
- Qualitative Results: Visual comparisons show that SCAN reconstructs sharper edges and more accurate details, especially in challenging cases with small targets or complex textures, outperforming other methods that often produce blurry or artifact-ridden results.
Limitations and Future Work
- Limitations: While SCAN demonstrates strong performance, the paper implicitly suggests that further exploration of attention mechanisms and feature aggregation strategies could yield even better results. The ablation studies also indicate that while individual components (FD, TSGA, DWConv, MSFR) contribute positively, their synergistic combination is key.
- Future Work: The authors suggest that the proposed SCAN architecture, with its focus on lightweight design and effective feature aggregation, can be further explored and adapted for other computer vision tasks beyond image super-resolution. The successful integration of large-kernel convolutions in a lightweight setting also opens avenues for future research in this direction.
Summary updated