Abstract
Transformer-based models have achieved remarkable performance in single image super-resolution (SR). However, they often suffer from heavy computational costs and a "low-frequency preference," where high-frequency details like sharp edges and textures are suppressed. We propose FreqFormer, a lightweight Frequency-aware Transformer that employs a divide-and-conquer strategy. By introducing a Frequency Division Module (FDM) and Frequency-aware Transformer Blocks (FTB), FreqFormer effectively recovers high-frequency information while maintaining a small parameter footprint.
The Low-Frequency Bias in Transformers
A significant challenge in applying Transformers to SR is their tendency to act as low-pass filters. The authors observed a distinct "amplitude drop" in high-frequency components as the network depth increases in standard models like SwinIR. This phenomenon confirms that vanilla self-attention mechanisms naturally prioritize smooth areas over fine textures, which is counterproductive for high-quality image reconstruction.
Methodology: The FreqFormer Architecture
To address the frequency bias, FreqFormer treats frequency as a "first-class citizen" through a specialized architecture designed to handle different image frequencies separately.
Key Components
- Frequency Division Module (FDM): Utilizes the Dual-Tree Complex Wavelet Transform (DTCWT) to explicitly split input features into high- and low-frequency paths, allowing for targeted processing from the start.
- Frequency-aware Transformer Block (FTB): The core unit containing two specialized attention mechanisms: Spatial-Frequency Attention (SFA), which integrates high-frequency priors to preserve fine details, and Channel Transposed Attention (CTA), which captures global context efficiently.
- Dual Frequency Aggregation Feed-Forward Network (DFFN): Employs a "frequency gate" to selectively aggregate information for enhanced detail reconstruction.
- High-Frequency Recovery Block (HFRB): A final convolutional stage designed to compensate for any detail loss that may occur during the attention phases.
Experimental Results
FreqFormer was evaluated on standard benchmarks including Set5, Set14, B100, Urban100, and Manga109, using DIV2K and Flickr2K for training.
- Quantitative Performance: FreqFormer achieves state-of-the-art results among lightweight SR methods. On the texture-heavy Manga109 dataset, it outperformed the classic EDSR by 1.11 dB to 1.35 dB in PSNR across various scaling factors.
- Efficiency: With approximately 870K to 890K parameters, FreqFormer is significantly more efficient than heavy Transformer models and outperforms other lightweight competitors like CRAFT and SRFormer-light.
- Visual Quality: Qualitative analysis shows that FreqFormer excels at reconstructing sharp, cohesive lines in complex architectural textures where other models often produce blur or ringing artifacts.
Conclusion and Future Work
FreqFormer demonstrates that explicit frequency awareness is a powerful tool for lightweight image super-resolution. By separating frequency components, the model overcomes the inherent low-frequency bias of Transformers. Future research may explore simplifying the wavelet transform overhead or extending this frequency-aware approach to video super-resolution, where temporal high-frequency consistency is critical.
Summary updated