Exploring Frequency-Inspired Optimization in Transformer for Efficient Single Image Super-Resolution

Ao Li, Le Zhang, Yun Liu, Ce Zhu

Abstract

Transformer-based methods have exhibited remarkable potential in single image super-resolution (SISR) by effectively extracting long-range dependencies. However, most current research has prioritized capturing global information while overlooking high-frequency priors. In this study, we demonstrate that transformer structures are adept at capturing low-frequency information but have limited capacity for high-frequency representations compared to convolutional counterparts. We propose the Cross-Refinement Adaptive Feature Modulation Transformer (CRAFT), which integrates convolutional and transformer strengths. CRAFT utilizes a High-Frequency Enhancement Residual Block (HFERB), a Shift Rectangle Window Attention Block (SRWAB), and a Hybrid Fusion Block (HFB). Furthermore, we introduce a frequency-guided post-training quantization (PTQ) method to enhance efficiency. Experimental results show CRAFT's superiority in both full-precision and quantization scenarios.

Introduction: The Frequency Bias of Transformers

Single Image Super-Resolution (SISR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. While Transformer architectures have dominated recent benchmarks due to their ability to model long-range dependencies, they possess a significant inherent weakness: they act primarily as low-pass filters.

Figure 1: Analysis of frequency attention bias and high-frequency reconstruction effectiveness across different architectures.

As shown in Figure 1, empirical analysis reveals that as high-frequency components are removed from the input, Transformer-based models (like SwinIR and HAT) maintain relatively stable performance compared to convolutional models (like EDSR). This suggests that Transformers rely heavily on low-frequency global context, struggling to reconstruct the sharp edges and fine textures that define high-quality HR images. Additionally, these models are computationally intensive, making deployment on edge devices challenging.

Proposed Method: The CRAFT Framework

To address these limitations, we propose the Cross-Refinement Adaptive Feature Modulation Transformer (CRAFT). This architecture hybridizes CNN-like high-frequency extraction with Transformer-based global context modeling.

1. Architecture Components

CRAFT is built upon three primary modules designed to balance global and local information:

  • High-Frequency Enhancement Residual Block (HFERB): A CNN-based module that utilizes max-pooling and convolutions to specifically target and extract high-frequency details.
  • Shift Rectangle Window Attention Block (SRWAB): An efficient Transformer block using rectangular windows to capture long-range dependencies across a wider receptive field than standard square windows.
  • Hybrid Fusion Block (HFB): This module acts as the integration point, using an inter-attention mechanism where high-frequency features from the HFERB serve as a prior to refine the global features from the Transformer.
Figure 3: Detailed architecture of the proposed CRAFT model, including the RCRFG, CRFB, HFERB, SRWAB, and HFB modules.

2. Frequency-Guided Post-Training Quantization (PTQ)

To enable efficient deployment, we developed a specialized PTQ strategy that preserves image sharpness even at low bit-widths (e.g., 4-bit).

  • Adaptive Dual Clipping (ADC): This method adjusts quantization boundaries by measuring Mean Absolute Error (MAE) in the frequency domain, ensuring that high-frequency "sharpness" is not discarded during bit-reduction.
  • Boundary Refinement (BR): Using a small calibration set of 100 unlabeled images, this step fine-tunes quantization boundaries to mitigate the noise introduced by standard quantization methods.

Experimental Results

CRAFT demonstrates state-of-the-art performance across standard benchmarks such as Set5, Set14, and Manga109. For instance, on the Manga109 dataset (x4 scale), CRAFT achieved a PSNR of 31.18 dB, outperforming SwinIR (30.92 dB) while utilizing approximately 15% fewer parameters.

In quantization tests, our frequency-guided PTQ showed a massive breakthrough. While standard MinMax quantization often causes PSNR to drop to unusable levels (~21 dB) at 4-bit precision, our method maintained a PSNR of 29.46 dB on Set5. Furthermore, 8-bit quantization achieved up to a 24% speedup in inference time on consumer-grade CPUs compared to full-precision versions.

Conclusion

CRAFT effectively bridges the gap between global context modeling and high-frequency detail reconstruction in SISR. By combining a hybrid architecture with a frequency-aware quantization strategy, we provide a solution that is both high-performing and computationally efficient for real-world deployment. Future work will explore extending these frequency-guided optimizations to other restoration tasks like deblurring and denoising.

Summary updated

Abstract

Transformer-based methods have exhibited remarkable potential in single image super-resolution (SISR) by effectively extracting long-range dependencies. However, most of the current research in this area has prioritized the design of transformer blocks to capture global information, while overlooking the importance of incorporating high-frequency priors, which we believe could be beneficial. In our study, we conducted a series of experiments and found that transformer structures are more adept at capturing low-frequency information, but have limited capacity in constructing high-frequency representations when compared to their convolutional counterparts. Our proposed solution, the cross-refinement adaptive feature modulation transformer (CRAFT), integrates the strengths of both convolutional and transformer structures. It comprises three key components: the high-frequency enhancement residual block (HFERB) for extracting high-frequency information, the shift rectangle window attention block (SRWAB) for capturing global information, and the hybrid fusion block (HFB) for refining the global representation. To tackle the inherent intricacies of transformer structures, we introduce a frequency-guided post-training quantization (PTQ) method aimed at enhancing CRAFT's efficiency. These strategies incorporate adaptive dual clipping and boundary refinement. To further amplify the versatility of our proposed approach, we extend our PTQ strategy to function as a general quantization method for transformer-based SISR techniques. Our experimental findings showcase CRAFT's superiority over current state-of-the-art methods, both in full-precision and quantization scenarios. These results underscore the efficacy and universality of our PTQ strategy. The source code is available at: https://github.com/AVC2-UESTC/Frequency-Inspired-Optimization-for-EfficientSR.git.
BibTeX
@article{li2023exploring,
  author = {Ao Li and Le Zhang and Yun Liu and Ce Zhu},
  title = {{Exploring Frequency-Inspired Optimization in Transformer for Efficient Single Image Super-Resolution}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
}
Figure