Lightweight Image Super-Resolution with Superpixel Token Interaction

Aiping Zhang, Wenqi Ren, Yi Liu, Xiaochun Cao

Introduction

Single Image Super-Resolution (SISR) is the task of reconstructing a high-quality, high-resolution (HR) image from a single low-resolution (LR) input. While Transformer-based models have recently set new benchmarks in this field due to their ability to capture long-range dependencies, their computational cost remains a significant barrier for deployment on edge devices. To mitigate this, many lightweight models utilize fixed-size square patches. However, this rigid partitioning often disrupts natural image structures and introduces noise into the attention mechanism.

This paper introduces the Superpixel Token Interaction Network (SPIN), a novel approach that replaces traditional square patches with flexible, content-aware "superpixels" to achieve state-of-the-art performance with high efficiency.

The Challenge of Rigid Partitioning

In standard lightweight Transformer architectures, images are divided into small, fixed-size grids (e.g., 8x8). This approach suffers from two primary drawbacks:

  1. Structural Splitting: Square grids frequently bisect continuous objects or edges, making it difficult for the model to maintain structural integrity during reconstruction.
  2. Noisy Attention: Within a fixed square, a model may attempt to correlate unrelated pixels (e.g., a piece of a window and a piece of a brick wall), leading to interference and visual artifacts.

Proposed Method: SPIN

SPIN addresses these issues by clustering similar, adjacent pixels into irregular shapes known as superpixels. The core of the architecture is the SPI Block, which processes these superpixels through three specialized modules.

The SPIN architecture and the internal structure of the SPI Block, featuring SPA, SPCA, and ISPA modules.

SPI Block Components

  • Superpixel Aggregation (SPA): This module employs a soft k-means iterative clustering algorithm to group pixels into superpixels. This ensures that the regions processed by the model correspond to actual visual features.
  • Superpixel Cross Attention (SPCA): To handle global information, SPCA uses superpixels as "surrogates" for entire regions. This allows the model to pass information across the image efficiently without the quadratic cost of pixel-to-pixel attention.
  • Intra-Superpixel Attention (ISPA): This module focuses on local details by performing attention only among pixels within the same superpixel. Since these pixels are already semantically similar, the model avoids the noise typical of heterogeneous square patches.

Experimental Results

SPIN was trained on the DIV2K dataset and evaluated against five standard benchmarks: Set5, Set14, BSDS100, Urban100, and Manga109.

Performance and Efficiency

SPIN demonstrates a superior trade-off between reconstruction quality (PSNR) and model size. For a $\times4$ upscaling task, SPIN utilizes approximately 555K parameters, significantly fewer than competitors like SwinIR-light (897K) and ESRT (751K).

Comparison of PSNR vs. Model Size (K) on the Urban100 (x4) dataset. SPIN achieves the highest PSNR with a very low parameter count.

As shown in the figure above, SPIN occupies the top-left quadrant of the performance-efficiency graph, indicating it achieves higher PSNR scores than larger models. Qualitatively, SPIN excels at restoring sharp, repetitive textures—such as architectural grates—where other models often produce blurry lines or checkerboard artifacts.

Limitations and Future Work

Despite its success, SPIN faces certain challenges. Performance tends to plateau or drop after 8 SPI blocks, suggesting potential overfitting on current training data. Additionally, the ISPA module requires a "Top-N selection" workaround to handle the varying sizes of superpixels in parallel processing. Finally, while superpixels reduce attention costs, the initial clustering (SPA) adds a computational overhead that must be optimized for real-time mobile applications.

Summary updated

Abstract

Transformer-based methods have demonstrated impressive results on single-image super-resolution (SISR) task. However, self-attention mechanism is computationally expensive when applied to the entire image. As a result, current approaches divide low-resolution input images into small patches, which are processed separately and then fused to generate high-resolution images. Nevertheless, this conventional regular patch division is too coarse and lacks interpretability, resulting in artifacts and non-similar structure interference during attention operations. To address these challenges, we propose a novel super token interaction network (SPIN). Our method employs superpixels to cluster local similar pixels to form the explicable local regions and utilizes intra-superpixel attention to enable local information interaction. It is interpretable because only similar regions complement each other and dissimilar regions are excluded. Moreover, we design a super-pixel cross-attention module to facilitate information propagation via the surrogation of superpixels. Extensive experiments demonstrate that the proposed SPIN model performs favorably against the state-of-the-art SR methods in terms of accuracy and lightweight. Code is available at https://github.com/ArcticHare105/SPIN.
BibTeX
@inproceedings{zhang2023lightweight,
  author = {Aiping Zhang and Wenqi Ren and Yi Liu and Xiaochun Cao},
  title = {{Lightweight Image Super-Resolution with Superpixel Token Interaction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01169},
}
Figure