TL;DR
This paper introduces Adaptive Cross-Layer Attention (ACLA), a novel attention mechanism for image restoration that allows features from different network layers to interact. ACLA dynamically selects informative features across layers and optimizes its own placement within a network, leading to improved restoration quality with comparable or reduced computational cost.
What Problem Does This Paper Address?
Image restoration tasks, such as denoising, super-resolution, and artifact removal, aim to recover a high-quality image from a degraded input. While deep learning models, particularly those incorporating non-local attention mechanisms, have shown great promise, existing attention modules typically operate within a single layer. This limitation means they might miss crucial correlations and complementary information that exists between features at different hierarchical levels of the network. Simply extending these to cross-layer attention can lead to prohibitively high computational costs. The paper addresses this by proposing an efficient and adaptive cross-layer attention mechanism that intelligently selects relevant features and optimizes its integration into existing network architectures.
Methods
The core of the proposed method is the Adaptive Cross-Layer Attention (ACLA) module. ACLA builds upon the concept of cross-layer attention, where a "query" feature from one layer can attend to "key" features from previous layers.
Key innovations include:
- Adaptive Key Selection: Instead of using all features from previous layers as keys (which is computationally expensive), ACLA selects a small, adaptive number of keys for each query. This selection is guided by learned gating masks, allowing the module to dynamically choose the most relevant features based on the input.
- Neural Architecture Search (NAS) for Placement: To further optimize efficiency and effectiveness, ACLA employs a NAS approach to automatically determine the best locations to insert its modules within a neural network backbone. This search optimizes for both restoration performance and inference cost (FLOPs).
- Cross-Layer Non-Local (CLNL) Attention: As a precursor to ACLA, the paper introduces CLNL, which extends non-local attention to consider features from multiple layers. However, CLNL is computationally demanding.
- Cross-Layer Attention (CLA): A simplified version of ACLA where key selection is adaptive, but insertion positions are fixed.
Datasets
The paper evaluates ACLA on standard datasets for various image restoration tasks:
- Single Image Super-Resolution: Set5, Set14, B100, Urban100, Manga109, DIV2K.
- Image Denoising: KCLDAk24, BSD68, Urban100.
- Image Compression Artifacts Reduction: LIVE1, Classic5.
- Image Demosaicing: McMaster18, Kodak24, BSD68, Urban100.
Key Insights and Ideas
- Inter-layer correlations are vital: The paper strongly argues that features from different layers contain complementary information that is crucial for high-quality image restoration, and that existing single-layer attention mechanisms miss this.
- Adaptive selection is key to efficiency: Simply extending attention to all cross-layer features is computationally infeasible. ACLA's adaptive selection of a small number of keys per query is critical for balancing performance and efficiency.
- Location matters: The placement of attention modules significantly impacts performance. Using NAS to find optimal insertion points is a powerful strategy for maximizing the benefits of ACLA.
- ACLA learns semantically relevant features: Visualizations show that ACLA effectively identifies and attends to key features that are semantically similar to the query, even across different layers, unlike simpler cross-layer methods.
Key Results
ACLA demonstrates significant improvements across multiple image restoration tasks when integrated into the EDSR backbone:
- Single Image Super-Resolution: Achieves state-of-the-art results, outperforming strong baselines like NLSN and HAT, with improvements in PSNR and SSIM. For example, it shows an average improvement of over 0.11 dB in PSNR for 2x, 3x, and 4x upsampling.
- Image Denoising: Achieves remarkable improvements over methods like SCUNet, with average PSNR gains of around 0.11-0.13 dB across different noise levels.
- Image Compression Artifacts Reduction: Outperforms SwinIR with average PSNR gains of around 0.10-0.11 dB.
- Image Demosaicing: Achieves the best reconstruction results, with an average PSNR improvement of 0.135 dB over PANet.
- Efficiency: ACLA often achieves these performance gains with comparable or even fewer parameters and FLOPs compared to other attention-based methods, thanks to its adaptive nature and NAS-driven placement.
Limitations and Future Work
- Computational Cost: While ACLA is designed to be efficient, cross-layer attention inherently adds computational overhead compared to models without it. The NAS search process itself is also computationally intensive.
- Hyperparameter Tuning: The paper mentions the need for cross-validation to select the hyperparameter $\lambda$ that balances performance and cost.
- Future Work: The paper suggests that the NAS framework can be extended to search for more complex ACLA module designs and to optimize other aspects of the network architecture. The effectiveness of ACLA in other computer vision tasks beyond image restoration could also be explored.
Summary updated