Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration

Shihao Zhou, Jinshan Pan, Jinglei Shi, Duosheng Chen, Lishen Qu, Jufeng Yang

Introduction

Image restoration tasks such as deraining, deblurring, and demoiréing are fundamental challenges in computer vision. While deep learning models, particularly Transformers, have achieved significant success, most existing architectures focus primarily on spatial relations—how pixels relate to their neighbors. However, different types of image degradation often manifest in distinct frequency bands. For instance, rain streaks might obscure global structures (low frequency), while raindrops create sharp, localized occlusions (high frequency).

The authors of "Seeing the Unseen" argue that neglecting these frequency-domain characteristics leads to subtle artifacts or a failure to recover fine textures. To address this, they propose FPro (Frequency Prompting), a novel framework that uses frequency-domain information as "prompts" to guide a Transformer-based restoration model.

The FPro Architecture

FPro consists of a standard restoration branch (an encoder-decoder Transformer) and a specialized Prompt Branch. The primary goal is to explicitly decouple and modulate low- and high-frequency components to provide better guidance for the restoration process.

The overall architecture of FPro, featuring the main restoration branch and the Dual Prompt Block (DPB).

The restoration branch follows a typical U-shaped structure. The input degraded image $I$ is first processed by a convolution layer to extract shallow features $F_s$. These features then pass through an encoder and a decoder. The innovation lies in the parallel Prompt Branch, which takes $F_s$ and generates frequency-specific prompts that are injected into the decoder.

Frequency-Guided Prompting

The Prompt Branch relies on three key components to extract and utilize frequency information:

1. Gated Dynamic Decoupler (GDD)

The GDD module splits the input features into separate low-frequency ($F_{lo}$) and high-frequency ($F_{hi}$) maps. It employs dynamically learned filters and a gating mechanism to suppress uninformative elements, ensuring the model focuses on the most relevant frequency signals for the specific restoration task.

2. Dual Prompt Block (DPB)

Once decoupled, the signals are processed by two distinct modulators within the DPB: the Low-frequency Prompt Modulator (LPM) and the High-frequency Prompt Modulator (HPM).

Detailed view of the High-frequency Prompt Modulator (HPM) and Low-frequency Prompt Modulator (LPM).
  • Low-frequency Prompt Modulator (LPM): Operating in the Fourier domain using Fast Fourier Transforms (FFT), the LPM captures global structures and long-range interactions efficiently. This approach provides the benefits of a massive spatial convolution kernel or global attention but with significantly lower computational cost.
  • High-frequency Prompt Modulator (HPM): This modulator operates in the spatial domain. It uses depth-wise convolutions (acting as high-pass filters) and window-based self-attention to sharpen edges and recover fine textures.

3. Prompt Injection

The outputs of the LPM and HPM are combined and injected into the decoder of the main restoration branch. This "prompting" mechanism tells the model what to look for, helping it distinguish between actual image content and degradation artifacts.

Experimental Results

FPro was evaluated across five major restoration tasks using benchmark datasets such as SPAD (deraining), AGAN-Data (raindrop removal), TIP-2018 (demoireing), SOTS (dehazing), and GoPro (deblurring).

The model consistently outperformed state-of-the-art (SOTA) methods:

  • Deraining: On the SPAD dataset, FPro achieved a 48.99 dB PSNR, surpassing DRSformer by 0.46 dB.
  • Raindrop Removal: It reached 31.96 dB PSNR on AGAN-Data.
  • Demoireing: On TIP-2018, it achieved 29.25 dB PSNR, a 0.38 dB improvement over previous SOTAs.

Beyond performance, FPro is computationally efficient. With 22.3M parameters and 81.9G FLOPs, it is more lightweight than competitors like DRSformer (33.7M parameters / 242.9G FLOPs) while delivering superior results. Qualitative evaluations show that FPro is particularly effective at preserving structural edges and removing complex moiré patterns.

Conclusion

FPro demonstrates that explicitly incorporating frequency-domain information as prompts can significantly enhance image restoration. By decoupling frequency components and processing them with specialized modulators (LPM and HPM), the model achieves a better balance between global structure preservation and fine texture recovery.

While highly effective, the model faces challenges in nighttime scenes with heavy degradation, where distinguishing between clear content and noise becomes extremely difficult. Future research will focus on improving robustness in these extreme conditions and optimizing the architecture for low-light photography.

Summary updated

BibTeX
@inproceedings{zhou2024seeing,
  author = {Shihao Zhou and Jinshan Pan and Jinglei Shi and Duosheng Chen and Lishen Qu and Jufeng Yang},
  title = {{Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72640-8\_14},
}
Figure