Image Super-Resolution With Non-Local Sparse Attention

Yiqun Mei, Yuchen Fan, Yuqian ZhouCVPR 2021 · OpenReview

Abstract

Single Image Super-Resolution (SISR) relies on exploiting both local and non-local image priors. While non-local attention mechanisms effectively capture long-range dependencies, they suffer from quadratic computational complexity and noise sensitivity. This paper presents Non-Local Sparse Attention (NLSA), which leverages Spherical Locality Sensitive Hashing (LSH) to group similar features into buckets. This approach reduces attention complexity to asymptotic linear time and improves robustness by focusing on relevant patches. The resulting Non-Local Sparse Network (NLSN) achieves state-of-the-art performance on standard benchmarks with significantly higher efficiency.

Introduction

The goal of Single Image Super-Resolution (SISR) is to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. Modern deep learning approaches utilize two primary priors: sparsity and non-local (NL) self-similarity. Sparsity assumes images can be represented by a few key features, aiding in noise reduction. NL self-similarity exploits the fact that textures and patterns repeat throughout an image.

Standard Non-Local Attention (NLA) is powerful but computationally expensive ($O(N^2)$), as it compares every pixel to every other pixel. Furthermore, NLA often attends to irrelevant or noisy regions, which can degrade reconstruction quality.

Methodology: Non-Local Sparse Attention (NLSA)

To address the limitations of standard NLA, we propose Non-Local Sparse Attention (NLSA). The core mechanism involves partitioning the feature space into "buckets" using Spherical Locality Sensitive Hashing (LSH).

Spherical Locality Sensitive Hashing

Instead of exhaustive global search, NLSA uses LSH to assign pixels with similar angular distances to the same bucket. By computing attention only within these buckets, the model focuses its computational budget on the most relevant patches, effectively filtering out non-local noise.

Overview of the Non-Local Sparse Attention (NLSA) mechanism, illustrating the use of Spherical LSH for bucketing and the resulting sparse attention matrix.

Computational Efficiency

This bucketing strategy transforms the computational complexity from quadratic to linear relative to the image size. A single-round NLSA module is approximately 17x more efficient than standard NLA. To ensure robustness against bucket boundary artifacts, the model employs multi-round hashing, aggregating results across different random rotations.

Experimental Results

The proposed NLSN was trained on the DIV2K dataset and evaluated on five standard benchmarks: Set5, Set14, B100, Urban100, and Manga109.

Quantitative Performance

NLSN outperformed existing state-of-the-art models like RCAN and SAN. Notably, it achieved a gain of over 0.4 dB in PSNR on the Urban100 and Manga109 datasets, which are rich in repetitive structural patterns. Ablation studies confirmed that attending to a small set of LSH-selected locations (e.g., 100) is more effective than attending to thousands of locations in a dense setup.

Qualitative Analysis

Qualitatively, NLSN excels at restoring sharp, straight lines and repetitive grids, such as windows on architectural structures. While other models often produce blurry or wavy artifacts in these regions, NLSA maintains structural integrity.

Qualitative comparison of NLSN (Ours) against other state-of-the-art super-resolution methods on the Urban100 dataset.

Conclusion

Non-Local Sparse Attention (NLSA) provides a robust and efficient alternative to standard non-local mechanisms in image super-resolution. By combining the strengths of sparsity and non-local self-similarity through Spherical LSH, the proposed NLSN achieves superior reconstruction quality with linear computational complexity. Future work may explore the integration of NLSA into more modern backbones like Vision Transformers.

Summary updated

BibTeX
@inproceedings{mei2021image,
  author = {Yiqun Mei and Yuchen Fan and Yuqian Zhou},
  title = {{Image Super-Resolution With Non-Local Sparse Attention}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=fUQsqh3LKX},
}
Figure