Abstract
Single Image Super-Resolution (SISR) relies on exploiting both local and non-local image priors. While non-local attention mechanisms effectively capture long-range dependencies, they suffer from quadratic computational complexity and noise sensitivity. This paper presents Non-Local Sparse Attention (NLSA), which leverages Spherical Locality Sensitive Hashing (LSH) to group similar features into buckets. This approach reduces attention complexity to asymptotic linear time and improves robustness by focusing on relevant patches. The resulting Non-Local Sparse Network (NLSN) achieves state-of-the-art performance on standard benchmarks with significantly higher efficiency.
Introduction
The goal of Single Image Super-Resolution (SISR) is to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. Modern deep learning approaches utilize two primary priors: sparsity and non-local (NL) self-similarity. Sparsity assumes images can be represented by a few key features, aiding in noise reduction. NL self-similarity exploits the fact that textures and patterns repeat throughout an image.
Standard Non-Local Attention (NLA) is powerful but computationally expensive ($O(N^2)$), as it compares every pixel to every other pixel. Furthermore, NLA often attends to irrelevant or noisy regions, which can degrade reconstruction quality.
Methodology: Non-Local Sparse Attention (NLSA)
To address the limitations of standard NLA, we propose Non-Local Sparse Attention (NLSA). The core mechanism involves partitioning the feature space into "buckets" using Spherical Locality Sensitive Hashing (LSH).
Spherical Locality Sensitive Hashing
Instead of exhaustive global search, NLSA uses LSH to assign pixels with similar angular distances to the same bucket. By computing attention only within these buckets, the model focuses its computational budget on the most relevant patches, effectively filtering out non-local noise.
Computational Efficiency
This bucketing strategy transforms the computational complexity from quadratic to linear relative to the image size. A single-round NLSA module is approximately 17x more efficient than standard NLA. To ensure robustness against bucket boundary artifacts, the model employs multi-round hashing, aggregating results across different random rotations.
Experimental Results
The proposed NLSN was trained on the DIV2K dataset and evaluated on five standard benchmarks: Set5, Set14, B100, Urban100, and Manga109.
Quantitative Performance
NLSN outperformed existing state-of-the-art models like RCAN and SAN. Notably, it achieved a gain of over 0.4 dB in PSNR on the Urban100 and Manga109 datasets, which are rich in repetitive structural patterns. Ablation studies confirmed that attending to a small set of LSH-selected locations (e.g., 100) is more effective than attending to thousands of locations in a dense setup.
Qualitative Analysis
Qualitatively, NLSN excels at restoring sharp, straight lines and repetitive grids, such as windows on architectural structures. While other models often produce blurry or wavy artifacts in these regions, NLSA maintains structural integrity.
Conclusion
Non-Local Sparse Attention (NLSA) provides a robust and efficient alternative to standard non-local mechanisms in image super-resolution. By combining the strengths of sparsity and non-local self-similarity through Spherical LSH, the proposed NLSN achieves superior reconstruction quality with linear computational complexity. Future work may explore the integration of NLSA into more modern backbones like Vision Transformers.
Summary updated