Introduction
Single Image Super-Resolution (SISR) is the task of reconstructing a high-quality, high-resolution (HR) image from a single low-resolution (LR) input. While Transformer-based models have recently set new benchmarks in this field due to their ability to capture long-range dependencies, their computational cost remains a significant barrier for deployment on edge devices. To mitigate this, many lightweight models utilize fixed-size square patches. However, this rigid partitioning often disrupts natural image structures and introduces noise into the attention mechanism.
This paper introduces the Superpixel Token Interaction Network (SPIN), a novel approach that replaces traditional square patches with flexible, content-aware "superpixels" to achieve state-of-the-art performance with high efficiency.
The Challenge of Rigid Partitioning
In standard lightweight Transformer architectures, images are divided into small, fixed-size grids (e.g., 8x8). This approach suffers from two primary drawbacks:
- Structural Splitting: Square grids frequently bisect continuous objects or edges, making it difficult for the model to maintain structural integrity during reconstruction.
- Noisy Attention: Within a fixed square, a model may attempt to correlate unrelated pixels (e.g., a piece of a window and a piece of a brick wall), leading to interference and visual artifacts.
Proposed Method: SPIN
SPIN addresses these issues by clustering similar, adjacent pixels into irregular shapes known as superpixels. The core of the architecture is the SPI Block, which processes these superpixels through three specialized modules.
SPI Block Components
- Superpixel Aggregation (SPA): This module employs a soft k-means iterative clustering algorithm to group pixels into superpixels. This ensures that the regions processed by the model correspond to actual visual features.
- Superpixel Cross Attention (SPCA): To handle global information, SPCA uses superpixels as "surrogates" for entire regions. This allows the model to pass information across the image efficiently without the quadratic cost of pixel-to-pixel attention.
- Intra-Superpixel Attention (ISPA): This module focuses on local details by performing attention only among pixels within the same superpixel. Since these pixels are already semantically similar, the model avoids the noise typical of heterogeneous square patches.
Experimental Results
SPIN was trained on the DIV2K dataset and evaluated against five standard benchmarks: Set5, Set14, BSDS100, Urban100, and Manga109.
Performance and Efficiency
SPIN demonstrates a superior trade-off between reconstruction quality (PSNR) and model size. For a $\times4$ upscaling task, SPIN utilizes approximately 555K parameters, significantly fewer than competitors like SwinIR-light (897K) and ESRT (751K).
As shown in the figure above, SPIN occupies the top-left quadrant of the performance-efficiency graph, indicating it achieves higher PSNR scores than larger models. Qualitatively, SPIN excels at restoring sharp, repetitive textures—such as architectural grates—where other models often produce blurry lines or checkerboard artifacts.
Limitations and Future Work
Despite its success, SPIN faces certain challenges. Performance tends to plateau or drop after 8 SPI blocks, suggesting potential overfitting on current training data. Additionally, the ISPA module requires a "Top-N selection" workaround to handle the varying sizes of superpixels in parallel processing. Finally, while superpixels reduce attention costs, the initial clustering (SPA) adds a computational overhead that must be optimized for real-time mobile applications.
Summary updated