TL;DR
This paper introduces the Hierarchical Residual Attention Network (HRAN), a novel lightweight architecture for single image super-resolution (SISR). HRAN achieves state-of-the-art performance by efficiently aggregating residual features and employing a parallel, lightweight channel attention mechanism, significantly reducing computational cost and memory footprint compared to existing methods.
What Problem Does This Paper Address?
Single Image Super-Resolution (SISR) aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) input. This is an inherently ill-posed problem, as multiple HR images can map to the same LR input. While deep convolutional neural networks (CNNs), particularly those with residual connections and attention mechanisms, have dramatically improved SISR performance, they often come with a substantial increase in computational complexity and memory requirements. This makes them impractical for real-world applications, especially on resource-constrained devices like mobile phones. A key challenge is that in deep networks, crucial high-frequency details from early layers can be lost or diluted as they propagate through many residual blocks. Existing attention mechanisms, often integrated "in-place" within residual blocks, can also inadvertently discard relevant details needed at later stages.
Methods
The core innovation of this paper lies in the Hierarchical Residual Attention Network (HRAN), which tackles the trade-off between performance and efficiency. HRAN's non-linear mapping module is built around Residual Attention Feature Groups (RAFGs). Each RAFG consists of two parallel computational paths:
- Feature Banks: Residual features from multiple stacked residual blocks within a RAFG are hierarchically aggregated into "feature banks." This allows for a more global exploitation of residual features across the network, preventing degradation and preserving information that might otherwise be lost in local processing.
- Attention Banks: A Lightweight Channel Attention (LCA) block is introduced. Crucially, this attention mechanism operates in a parallel path to the residual blocks, rather than "in-place." This separation ensures that attention can focus on relevant features without discarding potentially useful details from the main residual path. The LCA block uses a single 1x1 convolution for efficiency, followed by average pooling and a sigmoid activation to generate channel importance weights. These weights are then used to re-scale the input features.
The outputs of the feature and attention banks are combined and fed to subsequent RAFGs, creating a hierarchical aggregation of both residual features and attention. A final upscaling module (sub-pixel shuffle) and a convolutional layer produce the SR image. The network is trained using an L₁ loss function.
Datasets
The paper uses the following datasets:
- Training: 800 high-quality images from the DIV2K dataset.
- Testing: Set5, Set14, B100, and Urban100.
The experiments are conducted using two degradation models:
- Bicubic (BI): Standard downscaling.
- Blur-downscale (BD): Includes blurring in addition to downscaling.
Key Insights and Ideas
- Parallel Attention is Key: The most significant insight is that processing channel attention in a parallel path, separate from the main residual computation, is more effective than "in-place" attention. This prevents the loss of valuable details that might be discarded by in-place mechanisms.
- Hierarchical Aggregation for Feature Preservation: Aggregating residual features hierarchically into "banks" across multiple RAFGs allows the network to retain and utilize information from earlier stages more effectively, combating feature degradation in deep networks.
- Lightweight Attention for Efficiency: The proposed LCA block is designed to be computationally efficient, using a single 1x1 convolution, which contributes to the overall lightweight nature of HRAN.
- Trade-off Mastery: HRAN demonstrates that it's possible to achieve state-of-the-art performance in SISR without the prohibitive computational and memory costs typically associated with high-performing models.
Key Results
- State-of-the-Art Performance: HRAN achieves superior PSNR and SSIM scores compared to numerous lightweight state-of-the-art SISR methods across various datasets and scale factors (x2, x3, x4) for both BI and BD degradation models. For example, on Urban100 (x4, BI), HRAN achieves a PSNR of 26.39 dB, outperforming other comparable methods.
- Computational Efficiency: HRAN is significantly more efficient. It uses a fraction of the parameters of heavy models like EDSR and RDN (e.g., 2-4% of parameters) while achieving comparable or better results.
- Speed: HRAN is reported to be the fastest among the compared lightweight networks on the Urban100 dataset (x4), with a running time of just 0.003 seconds per image.
- Ablation Studies:
- Using hierarchical feature banks instead of standard stacked residual blocks (ResNet) improved PSNR by 0.21 dB on Urban100 (x4).
- The proposed parallel LCA mechanism outperformed in-place attention by up to 0.9 dB on Set14 (x4).
- Removing the feature and attention banks at the output reduced performance by approximately 0.06 dB, indicating their importance.
Limitations and Future Work
- Limitations: While HRAN is lightweight, the paper doesn't explicitly discuss its performance on extremely low-resource devices or its real-time capabilities in video super-resolution. The effectiveness of the attention mechanism is primarily evaluated on channel-wise attention; exploring more sophisticated spatial or spatio-temporal attention within the parallel path could be a future direction.
- Future Work: The authors suggest that the proposed architecture is beneficial for preventing information loss and could be further explored. The parallel processing of features and attention opens avenues for more complex interactions and potentially even more efficient architectures. The paper also mentions that the code is publicly available, encouraging further research and development.
Summary updated