Abstract
Transformer-based methods have exhibited remarkable potential in single image super-resolution (SISR) by effectively extracting long-range dependencies. However, most current research has prioritized capturing global information while overlooking high-frequency priors. In this study, we demonstrate that transformer structures are adept at capturing low-frequency information but have limited capacity for high-frequency representations compared to convolutional counterparts. We propose the Cross-Refinement Adaptive Feature Modulation Transformer (CRAFT), which integrates convolutional and transformer strengths. CRAFT utilizes a High-Frequency Enhancement Residual Block (HFERB), a Shift Rectangle Window Attention Block (SRWAB), and a Hybrid Fusion Block (HFB). Furthermore, we introduce a frequency-guided post-training quantization (PTQ) method to enhance efficiency. Experimental results show CRAFT's superiority in both full-precision and quantization scenarios.
Introduction: The Frequency Bias of Transformers
Single Image Super-Resolution (SISR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. While Transformer architectures have dominated recent benchmarks due to their ability to model long-range dependencies, they possess a significant inherent weakness: they act primarily as low-pass filters.
As shown in Figure 1, empirical analysis reveals that as high-frequency components are removed from the input, Transformer-based models (like SwinIR and HAT) maintain relatively stable performance compared to convolutional models (like EDSR). This suggests that Transformers rely heavily on low-frequency global context, struggling to reconstruct the sharp edges and fine textures that define high-quality HR images. Additionally, these models are computationally intensive, making deployment on edge devices challenging.
Proposed Method: The CRAFT Framework
To address these limitations, we propose the Cross-Refinement Adaptive Feature Modulation Transformer (CRAFT). This architecture hybridizes CNN-like high-frequency extraction with Transformer-based global context modeling.
1. Architecture Components
CRAFT is built upon three primary modules designed to balance global and local information:
- High-Frequency Enhancement Residual Block (HFERB): A CNN-based module that utilizes max-pooling and convolutions to specifically target and extract high-frequency details.
- Shift Rectangle Window Attention Block (SRWAB): An efficient Transformer block using rectangular windows to capture long-range dependencies across a wider receptive field than standard square windows.
- Hybrid Fusion Block (HFB): This module acts as the integration point, using an inter-attention mechanism where high-frequency features from the HFERB serve as a prior to refine the global features from the Transformer.
2. Frequency-Guided Post-Training Quantization (PTQ)
To enable efficient deployment, we developed a specialized PTQ strategy that preserves image sharpness even at low bit-widths (e.g., 4-bit).
- Adaptive Dual Clipping (ADC): This method adjusts quantization boundaries by measuring Mean Absolute Error (MAE) in the frequency domain, ensuring that high-frequency "sharpness" is not discarded during bit-reduction.
- Boundary Refinement (BR): Using a small calibration set of 100 unlabeled images, this step fine-tunes quantization boundaries to mitigate the noise introduced by standard quantization methods.
Experimental Results
CRAFT demonstrates state-of-the-art performance across standard benchmarks such as Set5, Set14, and Manga109. For instance, on the Manga109 dataset (x4 scale), CRAFT achieved a PSNR of 31.18 dB, outperforming SwinIR (30.92 dB) while utilizing approximately 15% fewer parameters.
In quantization tests, our frequency-guided PTQ showed a massive breakthrough. While standard MinMax quantization often causes PSNR to drop to unusable levels (~21 dB) at 4-bit precision, our method maintained a PSNR of 29.46 dB on Set5. Furthermore, 8-bit quantization achieved up to a 24% speedup in inference time on consumer-grade CPUs compared to full-precision versions.
Conclusion
CRAFT effectively bridges the gap between global context modeling and high-frequency detail reconstruction in SISR. By combining a hybrid architecture with a frequency-aware quantization strategy, we provide a solution that is both high-performing and computationally efficient for real-world deployment. Future work will explore extending these frequency-guided optimizations to other restoration tasks like deblurring and denoising.
Summary updated