C2D-ISR: Optimizing Attention-based Image Super-resolution from Continuous to Discrete Scales

Yuxuan Jiang, Chengxi Zeng, Siyue Teng, Fan Zhang, Xiaoqing Zhu, Joel Sole, David BullarXiv 2025 · OpenReview

Abstract

In recent years, attention mechanisms have been exploited in single image super-resolution (SISR), achieving impressive reconstruction results. However, these advancements are still limited by the reliance on simple training strategies and network architectures designed for discrete up-sampling scales, which hinder the model's ability to effectively capture information across multiple scales. To address these limitations, we propose a novel framework, C2D-ISR, for optimizing attention-based image super-resolution models from both performance and complexity perspectives. Our approach is based on a two-stage training methodology and a hierarchical encoding mechanism. The new training methodology involves continuous-scale training for discrete scale models, enabling the learning of inter-scale correlations and multi-scale feature representation. In addition, we generalize the hierarchical encoding mechanism with existing attention-based network structures, which can achieve improved spatial feature fusion, cross-scale information aggregation, and more importantly, much faster inference. We have evaluated the C2D-ISR framework based on three efficient attention-based backbones, SwinIR-L, SRFormer-L and MambaIRv2-L, and demonstrated significant improvements over the other existing optimization framework, HiT, in terms of super-resolution performance (up to 0.2dB) and computational complexity reduction (up to 11%).

Introduction

Single Image Super-Resolution (SISR) aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) input. While modern Vision Transformers (ViTs) and attention mechanisms have set new records in quality, they face two major hurdles: rigid scaling and high computational complexity. Most models are designed and trained for fixed, discrete scales (e.g., exactly 2x or 4x), which prevents them from learning "inter-scale" correlations—the underlying logic of how features change across different sizes. Furthermore, high-quality attention models are often too heavy for real-time use. The C2D-ISR framework addresses these issues by introducing a novel training pipeline and a specialized hierarchical architecture.

Methodology

The researchers introduce two core innovations: a novel training pipeline and a specialized architecture.

1. The C2D Training Strategy (Continuous-to-Discrete)

Instead of training for a fixed scale from the outset, C2D-ISR uses a two-stage approach. In the first stage (Continuous Pre-training), the model is trained using a lightweight "Implicit Image Function" (HIIF-L). This allows the network to learn how to up-sample to any arbitrary scale, forcing it to capture scale-invariant features and inter-scale relationships. In the second stage (Discrete Fine-tuning), the continuous up-sampler is swapped for a standard, efficient discrete up-sampler (like PixelShuffle), and the model is fine-tuned for specific target scales. This allows the model to inherit rich multi-scale knowledge while maintaining the speed of a discrete model.

Comparison of training strategies: (a) learning from a single scale, (b) learning from another scale, and (c) the proposed C2D learning from continuous scales.

2. The HiET Architecture

The Hierarchical Encoding Transformer (HiET) layer uses local, hierarchical coordinates to encode positional information, making the feature grid more compact and reducing overhead compared to global coordinates. These layers are arranged in a U-Net style structure with adaptive window sizes. By varying the attention window at different levels of the U-Net, the model aggregates features from different scales without the heavy computational cost of traditional resolution-changing convolutions.

The HiET architecture overview, detailing the HiET Block, HiET Layer components, and the training switch between continuous ($u_C$) and discrete ($u_D$) modules.

Experiments and Results

The models were trained on the DIV2K dataset and evaluated on standard benchmarks: Set5, Set14, BSD100, Urban100, and Manga109. The C2D-ISR framework was applied to three state-of-the-art backbones: SwinIR-L, SRFormer-L, and MambaIRv2-L.

Key Findings

C2D-ISR consistently outperformed HiT, the previous leading optimization framework, in both quality and efficiency:

  • Improved Quality: Achieved PSNR (Peak Signal-to-Noise Ratio) gains of up to 0.2dB.
  • Reduced Complexity: Lowered computational complexity (FLOPs) and parameter counts by up to 11%.
  • Performance Benchmark: For example, the optimized SRFormer-C2D at 4x scale on the Urban100 dataset achieved 26.96 dB PSNR using only 57.0G FLOPs, whereas the unoptimized baseline achieved 26.67 dB while requiring 62.8G FLOPs.

Qualitative results show fewer artifacts and better reconstruction of fine textures. Analysis via Local Attribution Maps confirms that the model effectively utilizes a wider range of relevant pixels to make its predictions.

Conclusion

C2D-ISR demonstrates that learning the "harder" task of continuous scaling serves as a powerful regularizer, enabling models to perform better on fixed scaling tasks. By moving from global to hierarchical local encoding, the framework achieves a superior performance-to-complexity trade-off. Future work will focus on optimizing Feed-Forward Networks (FFN) and extending this continuous-to-discrete logic to Video Super-Resolution, where temporal consistency is critical.

Summary updated

Abstract

In recent years, attention mechanisms have been exploited in single image super-resolution (SISR), achieving impressive reconstruction results. However, these advancements are still limited by the reliance on simple training strategies and network architectures designed for discrete up-sampling scales, which hinder the model's ability to effectively capture information across multiple scales. To address these limitations, we propose a novel framework, \textbf{C2D-ISR}, for optimizing attention-based image super-resolution models from both performance and complexity perspectives. Our approach is based on a two-stage training methodology and a hierarchical encoding mechanism. The new training methodology involves continuous-scale training for discrete scale models, enabling the learning of inter-scale correlations and multi-scale feature representation. In addition, we generalize the hierarchical encoding mechanism with existing attention-based network structures, which can achieve improved spatial feature fusion, cross-scale information aggregation, and more importantly, much faster inference. We have evaluated the C2D-ISR framework based on three efficient attention-based backbones, SwinIR-L, SRFormer-L and MambaIRv2-L, and demonstrated significant improvements over the other existing optimization framework, HiT, in terms of super-resolution performance (up to 0.2dB) and computational complexity reduction (up to 11%). The source code will be made publicly available at www.github.com.
BibTeX
@misc{jiang2025c,
  author = {Yuxuan Jiang and Chengxi Zeng and Siyue Teng and Fan Zhang and Xiaoqing Zhu and Joel Sole and David Bull},
  title = {{C2D-ISR: Optimizing Attention-based Image Super-resolution from Continuous to Discrete Scales}},
  year = {2025},
  eprint = {2503.13740},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.13740},
}
Figure