TL;DR
Lightweight Vision Transformers (ViTs) for image super-resolution typically suffer from limited receptive fields due to restrictive window-based attention mechanisms. PromptSR solves this by introducing low-dimensional "anchor prompts" derived from downscaled features, enabling efficient global context aggregation and coarse-to-fine local refinement. The resulting model achieves state-of-the-art performance on standard benchmarks while keeping the parameter count strictly under 1 million.
What Problem Does This Paper Address?
Image super-resolution (SR) reconstructs high-resolution images from low-resolution inputs. Recently, Vision Transformers (ViTs) have dominated this space due to their ability to model long-range pixel dependencies. However, standard self-attention scales quadratically with image size, making it too computationally expensive for resource-constrained mobile or edge devices.
To build lightweight models (typically < 1M parameters), researchers use window-based self-attention, restricting attention to small, localized patches (e.g., $8 \times 8$ or $16 \times 16$). While this saves compute, it severely limits the model's receptive field. Existing workarounds like sliding windows or window-matching strategies still struggle to capture truly global context across the entire image space.
Methods
PromptSR introduces the Cascade Prompting Block (CPB), a hierarchical architecture that replaces standard Transformer blocks to extract features in a global-to-local and coarse-to-fine manner.
The CPB consists of three cascaded layers:
- Global Anchor Prompting Layer (GAPL): This layer exploits the cross-scale similarity of natural images. It downscales the input feature map to form a set of low-dimensional "Anchors." It then uses cross-attention between these anchors and the full-resolution image tokens to generate Anchor Prompts (APs). Because the APs are low-dimensional, they can interact with the entire image space to capture global context at a fraction of the computational cost ($1/d^2$ complexity reduction, where $d$ is the downscale factor).
- Local Prompting Layers (LPLs): The CPB contains two LPLs that refine local details using the global context maps generated by the GAPL.
- Coarse Prompting: Uses the coarse similarity map from the GAPL to group similar, non-local tokens into irregularly shaped windows via Category Self-Attention (CSA), breaking the rigidity of standard square windows. This is combined with standard Window Self-Attention (WSA).
- Fine Prompting: Repeats this process using a fine-grained similarity map to adaptively refine features down to the pixel level.
Additionally, an Anchor Prompt Update mechanism passes and updates APs across sequential blocks within a Residual Group, ensuring consistent global priors are maintained throughout the network.
Datasets
- Training: DIV2K (800 images).
- Evaluation: Five standard public benchmarks: Set5, Set14, BSD100, Urban100, and Manga109.
- No new benchmarks were introduced.
Key Insights and Ideas
- Leveraging Cross-Scale Invariance: The core idea relies on the fact that basic structures (lines, edges, textures) repeat across different scales in natural images. Downscaling features to create anchors is an incredibly efficient way to summarize global structures without processing the full-resolution token matrix.
- Prompting as a Conditional Guide: Instead of forcing the network to learn global dependencies implicitly through stacked local layers, PromptSR explicitly generates global context vectors and feeds them back into local layers as "prompts" to conditionally guide feature reconstruction.
- Dynamic Windowing via Clustering: By using the GAPL's similarity maps to categorize tokens, the model dynamically clusters related pixels across the image into custom-shaped attention windows, bypassing the spatial limitations of fixed square grids.
Key Results
PromptSR and its enhanced variant, PromptSR*, consistently outperform existing state-of-the-art lightweight models across all datasets:
- Efficiency vs. Performance: On the challenging Urban100 ($\times4$) dataset, the baseline PromptSR achieves a PSNR of 27.02 dB using only 779K parameters. This outperforms heavy, classical CNN models like RCAN (15.6M parameters, 26.82 dB) and SAN (15.9M parameters, 26.79 dB).
- Matching Large Transformers: The enhanced PromptSR* (998K parameters) achieves 27.07 dB on Urban100 ($\times4$), matching the performance of the standard SwinIR model, which requires 11.9M parameters—an 11x reduction in model size.
- Visual Fidelity: Qualitative evaluations show that PromptSR successfully restores sharp, consistent structural details (like fine mesh patterns and text) where other lightweight models produce blurry artifacts or geometric distortions.
Limitations and Future Work
- Complex Textures: The authors note that the model can struggle with highly intricate, dense, and complex textures (e.g., finely woven networks). In these regions, the coarse-to-fine prompting module can sometimes aggressively over-reconstruct, leading to unwanted artifacts.
- Loss of Semantic Information: Visualizations using t-SNE revealed that the generated Anchor Prompts do not cluster well by semantic image categories. Because the downscaling process filters out local details to save compute, it inadvertently discards high-level semantic priors that could otherwise guide the super-resolution process. The authors plan to explore semantic-preserving prompting techniques in future work.
Summary updated