Hybrid Frequency Modulation Network for Image Restoration

Yuning Cui, Mingyu Liu, Wenqi Ren, Alois Knoll, Fei Yuan, Bob Zhang, Jie Wen

Presented at IJCAI-24, CSNet introduces a novel framework for image restoration that leverages frequency-domain modulation to outperform existing state-of-the-art (SOTA) Transformers and CNNs. By processing features across both channel and spatial dimensions using Fourier transforms and multi-scale DC-component refinement, CSNet achieves superior results in tasks like dehazing, deblurring, and desnowing with remarkable computational efficiency.

Addressing the Limitations of Spatial-Only Restoration

Image restoration—recovering clean images from blurry, hazy, or snowy inputs—is a classic ill-posed problem. Traditional CNNs often struggle due to the local nature of convolutional kernels, which makes capturing the global context required for complex degradation patterns difficult.

While Transformers provide the necessary "global receptive fields," they are often computationally prohibitive. Furthermore, existing frequency-based methods typically apply Fourier transforms only to spatial dimensions, neglecting the rich interactions between feature channels. CSNet addresses these gaps by treating both channel and spatial dimensions within the frequency domain.

Methodology: Frequency-Based Modulation

The CSNet architecture is a U-shaped network built around three core innovations designed to maximize information exchange and efficiency:

  1. Frequency-based Channel Feature Modulation (FCFM): Instead of standard channel attention, FCFM applies a Fast Fourier Transform (FFT) across the channel dimension. This allows each pixel to perceive signals and degradation patterns across all channels simultaneously in the spectral space.
  2. Multi-scale Frequency-based Spatial Feature Modulation (MFSFM): The authors observed that the "Direct Current" (DC) component (the lowest frequency) of an image carries the bulk of degradation information. MFSFM uses a coarse-to-fine learning paradigm to refine this DC component across multiple scales using lightweight learnable parameters, avoiding expensive full-image FFTs.
  3. Frequency-inspired Loss Function (FLF): To ensure the model captures fine details, the FLF explicitly compares high-frequency components of predicted and ground-truth images.

Key Insights and Efficiency

A pivotal insight of the paper is the significance of the DC component. The researchers demonstrated that simply replacing the DC component of a hazy image with that of the ground truth significantly improves clarity. This led to the design of the MFSFM module, which focuses on these critical low-frequency elements.

Visual demonstration of the impact of the DC component on image restoration and the resulting clarity improvements.

By applying FFT across the channel stack, CSNet treats the channel dimension as a periodic signal. This allows the model to identify correlations between different learned features that standard spatial convolutions would miss, all while maintaining a competitive parameter count.

Performance and Results

CSNet achieved SOTA performance across nine diverse datasets covering three major tasks:

  • Image Dehazing: On the SOTS dataset, CSNet reached 41.34 dB PSNR, outperforming the previous best Transformer-based model (MB-TaylorFormer-B) by 0.63 dB.
  • Defocus Deblurring: On the DPDD combined dataset, it achieved 26.25 dB PSNR, surpassing both FocalNet and Restormer.
  • Image Desnowing: It set a new record on the CSD dataset with 38.13 dB PSNR, a significant 0.95 dB improvement over previous methods.
Performance comparison of CSNet against SOTA methods, showing its superior PSNR relative to its FLOPs budget.

Qualitatively, CSNet is particularly effective at removing "hard" blurs and preserving fine textures in nighttime dehazing, where other models often produce artifacts. Despite its high performance, it maintains a lower FLOP (Floating Point Operations) budget than many heavy-duty Transformer models, making it a highly efficient solution for high-quality image restoration.

Summary updated

BibTeX
@inproceedings{cui2024hybrid,
  author = {Yuning Cui and Mingyu Liu and Wenqi Ren and Alois Knoll and Fei Yuan and Bob Zhang and Jie Wen},
  title = {{Hybrid Frequency Modulation Network for Image Restoration}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=WsspD1QbZQ},
}
Figure