N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolution

Haram Choi, Jeongmin Lee, Jihoon YangCVPR 2023 · DOI · arXiv

Introduction

Single Image Super-Resolution (SR) is the task of reconstructing high-resolution (HR) images from their low-resolution (LR) counterparts. While Swin Transformers have set new benchmarks in this field through Window Self-Attention (WSA), they are often limited by a restricted receptive field and high computational demands. This paper introduces the N-Gram context to the vision domain, enabling Swin Transformers to capture broader spatial patterns beyond local windows, and proposes NGswin, an efficient asymmetric U-Net architecture designed for lightweight SR.

Methodology

The authors propose a dual-strategy approach involving a novel contextual mechanism and a specialized network architecture.

N-Gram Context

In Natural Language Processing, an N-Gram refers to a sequence of $N$ words. This paper redefines the N-Gram as a group of neighboring local windows. To maintain efficiency, the model generates "Uni-Gram" embeddings using channel-reducing group convolutions. These embeddings then interact via a Sliding-WSA mechanism, which effectively expands the receptive field. This allows the model to incorporate information from surrounding windows before the standard window partitioning occurs, bridging the gap between isolated local regions.

NGswin Architecture

The NGswin model utilizes an asymmetric U-Net structure to balance performance and efficiency:

  • Hierarchical Encoder: A three-stage encoder with patch-merging captures features across multiple scales.
  • SCDP Bottleneck: The "Spatial-Channel-Dimensional-Projection" bottleneck aggregates multi-scale outputs from the encoder, providing rich feature representation without the overhead of traditional U-Net bottlenecks.
  • Asymmetric Decoder: The decoder is significantly smaller than the encoder, focusing computational resources on the critical early stages of feature extraction.
The overall architecture of NGswin, featuring the N-Gram Swin Transformer Block (NSTB) and the N-Gram Window Partition mechanism.

Key Insights

The core innovation lies in leveraging spatial redundancy. Since images often contain repetitive patterns, a degraded pixel in one window can often be restored by referencing a similar, clearer pattern in a neighboring window. The N-Gram context facilitates this cross-window information flow. Furthermore, the authors demonstrate that the N-Gram context is a plug-and-play module; when applied to existing models like SwinIR-light, it consistently improves performance.

Results and Evaluation

The models were evaluated on standard benchmarks including DIV2K, Set5, Set14, BSD100, Urban100, and Manga109.

Efficiency and Performance

NGswin achieves state-of-the-art results for lightweight SR while drastically reducing computational costs. For a $2\times$ upscaling task, NGswin requires only 140.4G Mult-Adds, a 42% reduction compared to SwinIR-light's 243.7G, while maintaining competitive PSNR.

Performance comparison of NGswin against other lightweight SR models, showing superior PSNR with significantly lower computational cost (Mult-Adds).

Qualitative Analysis

Visual results indicate that N-Gram-enhanced models are superior at reconstructing fine, repetitive textures—such as building facades or manga line art—that standard window-based Transformers often blur.

Conclusion

By introducing N-Gram context and the NGswin architecture, this work addresses the fundamental limitations of Swin Transformers in image restoration. The approach provides a significant boost in both receptive field and computational efficiency, making high-quality super-resolution more feasible for resource-constrained environments. Future work may explore extending this context mechanism to other low-level vision tasks like denoising and deblurring.

Summary updated

BibTeX
@inproceedings{choi2023n,
  author = {Haram Choi and Jeongmin Lee and Jihoon Yang},
  title = {{N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolution}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00206},
}
Figure