Feature Modulation Transformer: Cross-Refinement of Global Representation via High-Frequency Prior for Image Super-Resolution

Ao Li, Le Zhang, Yun Liu, Ce Zhu

Abstract

Transformer-based methods have exhibited remarkable potential in single image super-resolution (SISR) by effectively extracting long-range dependencies. However, most of the current research in this area has prioritized the design of transformer blocks to capture global information, while overlooking the importance of incorporating high-frequency priors, which we believe could be beneficial. In our study, we conducted a series of experiments and found that transformer structures are more adept at capturing low-frequency information, but have limited capacity in constructing high-frequency representations when compared to their convolutional counterparts. Our proposed solution, the cross-refinement adaptive feature modulation transformer (CRAFT), integrates the strengths of both convolutional and transformer structures. It comprises three key components: the high-frequency enhancement residual block (HFERB) for extracting high-frequency information, the shift rectangle window attention block (SRWAB) for capturing global information, and the hybrid fusion block (HFB) for refining the global representation. Our experiments on multiple datasets demonstrate that CRAFT outperforms state-of-the-art methods by up to 0.29dB while using fewer parameters. The source code is available at: https://github.com/AVC2-UESTC/CRAFT-SR.git.

Introduction: The High-Frequency Blind Spot

Single Image Super-Resolution (SISR) aims to reconstruct a high-resolution (HR) image from a single low-resolution (LR) input. Recently, Transformers have dominated this field because they excel at capturing long-range dependencies—understanding how distant pixel patches relate to one another. However, the authors identify a critical weakness: Transformers essentially act as low-pass filters.

Through empirical analysis, the study demonstrates that while Transformers are excellent at capturing the "big picture" (low-frequency information), they are less effective than Convolutional Neural Networks (CNNs) at reconstructing sharp, high-frequency details.

Empirical analysis of frequency dependency and the experimental pipeline for evaluating high-frequency reconstruction.

As shown in Figure 1, there is a clear "frequency gap": Transformers are robust to the loss of high-frequency data but struggle to recreate it, whereas CNNs show the opposite trend. The challenge is to build a model that leverages global context without sacrificing these fine details.

The CRAFT Architecture

The proposed Cross-Refinement Adaptive Feature Modulation Transformer (CRAFT) addresses this by integrating three novel components:

  1. High-Frequency Enhancement Residual Block (HFERB): This component uses CNN-like structures, including max-pooling and convolutions, to specifically isolate and extract high-frequency features from the input.
  2. Shift Rectangle Window Attention Block (SRWAB): Instead of standard square windows, CRAFT uses horizontal and vertical rectangular windows. This expands the receptive field more effectively, allowing the model to capture long-range dependencies across the image more efficiently, particularly for natural images with elongated structures.
  3. Hybrid Fusion Block (HFB): This serves as the integration layer. It uses an inter-attention mechanism where the high-frequency information (from the HFERB) acts as a "prior" (the Query) to refine the global representations (the Key and Value) produced by the transformer branch. Crucially, the HFB performs attention along the channel dimension, shifting computational complexity from $O(N^2C)$ to $O(NC^2)$, making the model lighter and faster.

Experimental Results and Analysis

The model was trained on the DIV2K dataset and evaluated on five standard benchmarks: Set5, Set14, BSD100, Urban100, and Manga109, across $\times2, \times3,$ and $\times4$ upscaling factors.

Quantitative Performance

CRAFT outperforms previous state-of-the-art models like SwinIR across all benchmarks. For example, on the Manga109 dataset at $\times4$ scale, it achieved a 0.26dB PSNR improvement. Overall, improvements of up to 0.29dB were observed.

Efficiency and Visual Quality

Despite the performance boost, CRAFT is more efficient. At $\times4$ scale, it uses 753K parameters compared to SwinIR’s 897K, and it is significantly faster (42.8ms vs 72.0ms average inference time). Qualitative results show that CRAFT is much better at recovering "line direction"—it avoids the blurry or jagged edges often seen in other models when dealing with repetitive patterns or fine text.

Conclusion

CRAFT demonstrates that the "blind spot" of Transformers in reconstructing high-frequency details can be mitigated by using CNN-extracted features as a modulation prior. By combining the global context-awareness of Transformers with the high-frequency detail extraction of CNNs, CRAFT sets a new state-of-the-art for efficient and accurate SISR. Future work may extend this cross-refinement strategy to other low-level vision tasks like image denoising or deblurring.

Summary updated

Abstract

Transformer-based methods have exhibited remarkable potential in single image super-resolution (SISR) by effectively extracting long-range dependencies. However, most of the current research in this area has prioritized the design of transformer blocks to capture global information, while overlooking the importance of incorporating high-frequency priors, which we believe could be beneficial. In our study, we conducted a series of experiments and found that transformer structures are more adept at capturing low-frequency information, but have limited capacity in constructing high-frequency representations when compared to their convolutional counterparts. Our proposed solution, the cross-refinement adaptive feature modulation transformer (CRAFT), integrates the strengths of both convolutional and transformer structures. It comprises three key components: the high-frequency enhancement residual block (HFERB) for extracting high-frequency information, the shift rectangle window attention block (SRWAB) for capturing global information, and the hybrid fusion block (HFB) for refining the global representation. Our experiments on multiple datasets demonstrate that CRAFT outperforms state-of-the-art methods by up to 0.29dB while using fewer parameters. The source code will be made available at: https://github.com/AVC2-UESTC/CRAFT-SR.git.
BibTeX
@article{li2023feature,
  author = {Ao Li and Le Zhang and Yun Liu and Ce Zhu},
  title = {{Feature Modulation Transformer: Cross-Refinement of Global Representation via High-Frequency Prior for Image Super-Resolution}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
}
Figure