Abstract
Transformer-based methods have exhibited remarkable potential in single image super-resolution (SISR) by effectively extracting long-range dependencies. However, most of the current research in this area has prioritized the design of transformer blocks to capture global information, while overlooking the importance of incorporating high-frequency priors, which we believe could be beneficial. In our study, we conducted a series of experiments and found that transformer structures are more adept at capturing low-frequency information, but have limited capacity in constructing high-frequency representations when compared to their convolutional counterparts. Our proposed solution, the cross-refinement adaptive feature modulation transformer (CRAFT), integrates the strengths of both convolutional and transformer structures. It comprises three key components: the high-frequency enhancement residual block (HFERB) for extracting high-frequency information, the shift rectangle window attention block (SRWAB) for capturing global information, and the hybrid fusion block (HFB) for refining the global representation. Our experiments on multiple datasets demonstrate that CRAFT outperforms state-of-the-art methods by up to 0.29dB while using fewer parameters. The source code will be made available at: https://github.com/AVC2-UESTC/CRAFT-SR.git.
Introduction
Single Image Super-Resolution (SISR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. While Transformer-based models have dominated recent benchmarks due to their ability to capture long-range dependencies, they often struggle with the reconstruction of sharp edges and fine textures. This research identifies a "frequency gap": Transformers excel at capturing low-frequency information (general shapes and smooth regions) but are less effective than Convolutional Neural Networks (CNNs) at handling high-frequency details.
As illustrated in the frequency analysis above, Transformer-based structures like HAT and SwinIR show a lower sensitivity to the removal of high-frequency components compared to CNN-based models like EDSR. This empirical evidence motivates our hybrid approach, which seeks to explicitly incorporate high-frequency priors into the Transformer framework.
Methodology
The CRAFT Architecture
The proposed Cross-refinement Adaptive Feature modulation Transformer (CRAFT) architecture combines the detail-capturing power of CNNs with the global context of Transformers. It consists of three core modules:
- High-Frequency Enhancement Residual Block (HFERB): A CNN-based module designed to extract fine-grained textures and edges.
- Shift Rectangle Window Attention Block (SRWAB): A Transformer module that utilizes rectangular windows to capture broader global context with reduced computational overhead.
- Hybrid Fusion Block (HFB): The central innovation of CRAFT, which uses high-frequency features as the query in an attention mechanism to refine the global representation.
Frequency-Guided Post-Training Quantization (PTQ)
To enable efficient deployment, we also propose a frequency-guided PTQ strategy. This method avoids the need for full-dataset retraining and employs Adaptive Dual Clipping (ADC) to minimize information loss. Furthermore, Frequency-Guided Optimization (FGO) ensures that the quantization process preserves high-frequency components by applying constraints in the Fourier domain during calibration.
Key Results
CRAFT demonstrates superior performance across standard benchmarks such as Set5, Set14, and Manga109. For example, on the Manga109 dataset (x4 scale), CRAFT achieves a PSNR of 31.18 dB with only 753K parameters, outperforming larger models like SwinIR-light (897K parameters, 30.92 dB).
In terms of efficiency, our PTQ strategy allows for 4-bit quantization (W4A4) with significantly reduced quality loss. While standard MinMax quantization results in a PSNR drop to ~21.14 dB on Set5, our method maintains 29.46 dB. Hardware benchmarks on an AMD Ryzen 5 CPU show that the 8-bit quantized CRAFT model provides up to a 24% speedup in inference latency.
Conclusion
By addressing the frequency bias inherent in Transformers, CRAFT provides a robust and efficient solution for SISR. The integration of high-frequency priors through the Hybrid Fusion Block allows for superior detail reconstruction, while the frequency-guided PTQ strategy makes the model practical for real-world deployment on edge devices.
Summary updated